DEEP DIVE SERIES · #09

AST Skeletonization for Large Codebase Ingestion: Slashing 70% of Context Window Waste

Luca Viscomi

· Published Sep 17, 2026 · Updated Sep 24, 2026 · 4 min read

When an autonomous AI agent starts working on a new repository, its first instinct is to understand the codebase. In naive agent harnesses, this translates into executing recursive cat, view_file, or glob commands that dump thousands of lines of raw source code directly into the context window.

Consider a typical 800-line TypeScript service:

• Type Definitions & Function Signatures: ~120 lines (15%)
• Exported Interface Contracts: ~80 lines (10%)
• Internal Loop Logic, Local Variables, and Math Operations: ~600 lines (75%)

When an agent needs to plan an architectural refactor, choose a method to call, or wire a new dependency, it does not need the internal 600 lines of implementation logic. Ingesting full function bodies wastes thousands of tokens, crowds out instructions, and speeds up context saturation.

The solution is AST Skeletonization: programmatically parsing files into an Abstract Syntax Tree, stripping internal function and method bodies, and serving a lightweight “skeleton” to the agent.

1. The Anatomy of an AST Skeleton

An AST Skeleton retains all structural metadata while reducing tokens to the bare minimum:

Full Source Code (145 Tokens)

export class PaymentProcessor {
  private apiKey: string;

  constructor(key: string) {
    this.apiKey = key;
    if (!key.startsWith('pk_live_')) {
      throw new Error('Invalid live key format detected');
    }
  }

  public async chargeCustomer(customerId: string, amountCents: number): Promise<ChargeReceipt> {
    const payload = {
      customer: customerId,
      amount: amountCents,
      currency: 'usd',
      timestamp: Date.now()
    };
    const response = await fetch('https://api.payments.internal/v1/charges', {
      method: 'POST',
      headers: { 'Authorization': `Bearer ${this.apiKey}` },
      body: JSON.stringify(payload)
    });
    if (!response.ok) {
      throw new Error(`Charge failed with status ${response.status}`);
    }
    return await response.json();
  }
}

AST Skeleton (38 Tokens — 74% Reduction!)

export class PaymentProcessor {
  private apiKey: string;
  constructor(key: string);
  public async chargeCustomer(customerId: string, amountCents: number): Promise<ChargeReceipt>;
}

The agent learns everything it needs to know: class name, constructor signature, method name, input argument types, and return type promise. It can immediately generate valid client calls without ingesting the internal fetch logic or error handling boilerplate.

2. Token Savings Across Real-World Codebases

To demonstrate the quantitative impact, we benchmarked AST skeletonization across four popular open-source repositories using the o200k_base tokenizer:

Instead of saturating a 128k context window by viewing three large modules, an agent equipped with AST skeletonization can comfortably ingest the architectural surface of an entire microservice in a single prompt.

3. Building an AST Skeletonizer with TypeScript Compiler API

Here is a production-ready AST skeletonizer implemented in TypeScript using the official TypeScript compiler API (or cross-language tree parsers like Tree-sitter):

import * as ts from 'typescript';

export function skeletonizeSource(sourceCode: string, fileName: string = 'temp.ts'): string {
  const sourceFile = ts.createSourceFile(
    fileName,
    sourceCode,
    ts.ScriptTarget.Latest,
    true,
    ts.ScriptKind.TS
  );

  const printer = ts.createPrinter({ removeComments: true });

  const transformer: ts.TransformerFactory<ts.SourceFile> = (context) => {
    return (rootNode) => {
      function visit(node: ts.Node): ts.Node {
        // Strip Function Declaration bodies
        if (ts.isFunctionDeclaration(node)) {
          return ts.factory.updateFunctionDeclaration(
            node,
            node.modifiers,
            node.asteriskToken,
            node.name,
            node.typeParameters,
            node.parameters,
            node.type,
            undefined // Strip body!
          );
        }

        // Strip Method Declaration bodies
        if (ts.isMethodDeclaration(node)) {
          return ts.factory.updateMethodDeclaration(
            node,
            node.modifiers,
            node.asteriskToken,
            node.name,
            node.questionToken,
            node.typeParameters,
            node.parameters,
            node.type,
            undefined // Strip body!
          );
        }

        // Strip Constructor bodies
        if (ts.isConstructorDeclaration(node)) {
          return ts.factory.updateConstructorDeclaration(
            node,
            node.modifiers,
            node.parameters,
            undefined // Strip body!
          );
        }

        return ts.visitEachChild(node, visit, context);
      }

      return ts.visitNode(rootNode, visit) as ts.SourceFile;
    };
  }

  const result = ts.transform(sourceFile, [transformer]);
  const transformedSourceFile = result.transformed[0];
  const skeleton = printer.printFile(transformedSourceFile);
  result.dispose();

  return skeleton;
}

4. Architectural Integration: The Two-Tier File Ingestion Strategy

To harness AST skeletonization in production agent loops, implement a Two-Tier File Ingestion Strategy:

                  [ Agent Needs Codebase Context ]
                                 │
                                 ▼
                     [ Tier 1: Skeleton Pass ]
               Reads `inspect_skeleton(path)`
               Slashing 75% tokens. Locates exact 
               functions and interfaces required.
                                 │
                                 ▼
                     [ Tier 2: Surgical View ]
               Calls `view_file(path, startLine, endLine)`
               Reads ONLY the 20 lines that require 
               editing or deep inspection.

By decoupling Architectural Orientation from Surgical Execution, your agent harnesses consume a fraction of the token budget while completely avoiding context amnesia.