perf(mcp): answer-directly steering — ~35% cheaper, ~70% fewer tool calls (#224)
* perf(mcp): steer agents to answer directly instead of delegating to subagents CodeGraph beats native grep/read on cost only when the agent queries it directly. When the agent delegates to file-reading sub-agents, those sub-agents read files regardless of the index, so CodeGraph becomes net overhead on top of the reads. The install templates even told agents to "spawn a subagent for explore-class questions" — the expensive path. Changes: - server-instructions + both install templates: add an "Answer directly — don't delegate exploration" directive; reposition codegraph_explore as the efficient one-call multi-symbol tool (was: "spawn a subagent for it"). - codegraph_explore: hard-cap output to its adaptive budget (it overran, ~30k vs a 28k cap) and tighten the medium tier (28k->13k). - codegraph_node: return a member outline for container kinds instead of the full class body. Rigorous N>=4-per-arm warm-block benchmark (median total_cost_usd): excalidraw (~600 files): WITH $0.54 vs native $1.02 (-47%) vscode (~10k files): WITH $0.41 vs native $0.72 (-42%) ky (~25 files): WITH $0.46 vs native $0.44 (wash) Answers were equal-or-better (correct, file:line-cited) with ~6x fewer tool calls; the directive drove the direct path on 14/14 codegraph runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(readme): rebuild benchmark with real-world repos + cost/token/time/tool savings Replace the "Claude Code (Python+Rust/Java)" rows — which benchmarked the Claude Code CLI repo, not real codebases in those languages — with real open-source projects per language: Django (Python), Tokio (Rust), OkHttp (Java), Gin (Go), plus Alamofire (Swift) and the existing TypeScript repos (VS Code, Excalidraw). The table now reports all four savings the change targets — cost, tokens, time, tool calls — as the median of 4 runs per arm (Claude Opus 4.7, headless claude -p, with vs empty MCP config). Averages across the 7 repos: 35% cheaper, 59% fewer tokens, 49% faster, 70% fewer tool calls. Adds a methodology note and raw WITH->WITHOUT medians. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
a47355780b
commit
f5bbc26c60
@@ -22,6 +22,18 @@ in the workspace. Reads are sub-millisecond; the index lags writes by
|
||||
about a second through the file watcher. Consult it BEFORE writing or
|
||||
editing code, not during.
|
||||
|
||||
## Answer directly — don't delegate exploration
|
||||
|
||||
For "how does X work", architecture, trace, or where-is-X questions,
|
||||
answer DIRECTLY using 2-3 codegraph calls: \`codegraph_context\` first,
|
||||
then ONE \`codegraph_explore\` for the source of the symbols it surfaces.
|
||||
Codegraph IS the pre-built search index — so delegating the lookup to a
|
||||
separate file-reading sub-task/agent, or running your own grep + read
|
||||
loop, repeats work codegraph already did and costs more for the same
|
||||
answer. Reach for raw Read/Grep only to confirm a specific detail
|
||||
codegraph didn't cover. A direct codegraph answer is typically a handful
|
||||
of calls; a grep/read exploration is dozens.
|
||||
|
||||
## Tool selection by intent
|
||||
|
||||
- **"What is the symbol named X?"** → \`codegraph_search\`
|
||||
@@ -30,7 +42,7 @@ editing code, not during.
|
||||
- **"What does this call?"** → \`codegraph_callees\`
|
||||
- **"What would changing this break?"** → \`codegraph_impact\`
|
||||
- **"Show me this symbol's source / signature / docstring."** → \`codegraph_node\`
|
||||
- **"Survey an unfamiliar topic / pattern / module."** → \`codegraph_explore\` (heavier; deep dive)
|
||||
- **"Show me several related symbols' source / survey an area."** → \`codegraph_explore\` (ONE capped call; prefer over many codegraph_node/Read)
|
||||
- **"What's in directory X?"** → \`codegraph_files\`
|
||||
- **"Is the index ready / what's its size?"** → \`codegraph_status\`
|
||||
|
||||
@@ -44,7 +56,7 @@ editing code, not during.
|
||||
|
||||
- **Don't grep first** when looking up a symbol by name — \`codegraph_search\` is faster and returns kind + location + signature.
|
||||
- **Don't chain \`codegraph_search\` + \`codegraph_node\`** when you just want context — \`codegraph_context\` is one round-trip.
|
||||
- **Don't use \`codegraph_explore\` for narrow questions** — it's a multi-call deep dive, expensive in tokens. Save it for genuine "I'm new here" surveys.
|
||||
- **Don't loop \`codegraph_node\` over many symbols** — one \`codegraph_explore\` call returns them all grouped by file, while each separate call re-reads the whole context and costs far more. Use \`codegraph_node\` for a single symbol.
|
||||
- **Don't query the index immediately after editing a file** — the watcher needs ~500ms to debounce + sync. Wait for the next turn.
|
||||
|
||||
## Limitations
|
||||
|
||||
+74
-14
@@ -25,6 +25,16 @@ const MAX_OUTPUT_LENGTH = 15000;
|
||||
*/
|
||||
const RUST_PATH_PREFIXES = new Set(['crate', 'super', 'self']);
|
||||
|
||||
/**
|
||||
* Node kinds that contain other symbols. For these, `codegraph_node` with
|
||||
* `includeCode=true` returns a structural outline (member names + signatures
|
||||
* + line numbers) instead of the full body, which for a large class is a
|
||||
* multi-thousand-character wall of source that bloats the agent's context.
|
||||
*/
|
||||
const CONTAINER_NODE_KINDS = new Set<NodeKind>([
|
||||
'class', 'struct', 'interface', 'trait', 'protocol', 'enum', 'namespace', 'module',
|
||||
]);
|
||||
|
||||
/** Last `::` / `.` / `/`-separated segment of a qualified symbol. */
|
||||
function lastQualifierPart(symbol: string): string {
|
||||
const parts = symbol.split(/::|[./]/).filter((p) => p.length > 0);
|
||||
@@ -102,12 +112,12 @@ export function getExploreOutputBudget(fileCount: number): ExploreOutputBudget {
|
||||
}
|
||||
if (fileCount < 5000) {
|
||||
return {
|
||||
maxOutputChars: 28000,
|
||||
defaultMaxFiles: 9,
|
||||
maxCharsPerFile: 5000,
|
||||
gapThreshold: 12,
|
||||
maxSymbolsInFileHeader: 10,
|
||||
maxEdgesPerRelationshipKind: 10,
|
||||
maxOutputChars: 13000,
|
||||
defaultMaxFiles: 6,
|
||||
maxCharsPerFile: 2500,
|
||||
gapThreshold: 10,
|
||||
maxSymbolsInFileHeader: 8,
|
||||
maxEdgesPerRelationshipKind: 8,
|
||||
includeRelationships: true,
|
||||
includeAdditionalFiles: true,
|
||||
includeCompletenessSignal: true,
|
||||
@@ -263,7 +273,7 @@ export const tools: ToolDefinition[] = [
|
||||
},
|
||||
{
|
||||
name: 'codegraph_context',
|
||||
description: 'PRIMARY TOOL: Build comprehensive context for a task. Returns entry points, related symbols, and key code - often enough to understand the codebase without additional tool calls. NOTE: This provides CODE context, not product requirements. For new features, still clarify UX/behavior questions with the user before implementing.',
|
||||
description: 'PRIMARY TOOL — call this FIRST for any "how does X work", architecture, feature, or bug-context question. Composes search + node + callers + callees and returns entry points, related symbols, and key code in ONE call — usually enough to answer with no further search/Read/Grep. Prefer this over chaining codegraph_search + codegraph_node, and over codegraph_explore. NOTE: provides CODE context, not product requirements; for new features still clarify UX/edge cases with the user.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
@@ -348,7 +358,7 @@ export const tools: ToolDefinition[] = [
|
||||
},
|
||||
{
|
||||
name: 'codegraph_node',
|
||||
description: 'Get detailed information about a specific code symbol. Use includeCode=true only when you need the full source code - otherwise just get location and signature to minimize context usage.',
|
||||
description: 'Get detailed info about ONE symbol (location, signature, docstring). Pass includeCode=true for source: a function/method returns its body; a class/interface/struct/enum returns a compact member OUTLINE (fields + method signatures + line numbers), not every method body — Read or codegraph_node a specific member for its body. Keep includeCode=false to minimize context. For SEVERAL related symbols, make ONE codegraph_explore (or codegraph_context) call instead of many node calls — repeated node calls each re-read the whole context and cost far more.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
@@ -368,7 +378,7 @@ export const tools: ToolDefinition[] = [
|
||||
},
|
||||
{
|
||||
name: 'codegraph_explore',
|
||||
description: 'Deep exploration tool — returns comprehensive context for a topic in a SINGLE call. Groups all relevant source code by file (contiguous sections, not snippets), includes a relationship map, and uses deeper graph traversal. Designed to replace multiple codegraph_node + file Read calls. Use this instead of codegraph_context when you need thorough understanding. IMPORTANT: Use specific symbol names, file names, or short code terms in your query — NOT natural language sentences. Before calling this, use codegraph_search to discover relevant symbol names, then include those names in your query. Bad: "how are agent prompts loaded and passed to the CLI". Good: "readAgentsFromDirectory createClaudeSession chat-manager agents.ts".',
|
||||
description: 'Returns source for SEVERAL related symbols grouped by file, plus a relationship map, in ONE capped call. This is the efficient way to inspect many related symbols at once — strongly prefer it over a series of codegraph_node or Read calls (each separate call re-reads the whole context, so 8 node calls cost far more than 1 explore). Use it after codegraph_context when you need to see the actual source of several symbols. Query with specific symbol/file/code terms, NOT natural-language sentences — run codegraph_search first to find names. Bad: "how are agent prompts loaded and passed to the CLI". Good: "renderStaticScene drawElementOnCanvas ShapeCache renderElement.ts".',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
@@ -1241,7 +1251,20 @@ export class ToolHandler {
|
||||
}
|
||||
}
|
||||
|
||||
return this.textResult(lines.join('\n'));
|
||||
// Hard-cap to the adaptive budget. The per-file loop bounds the source
|
||||
// sections, but the relationship map, additional-files list, and
|
||||
// completeness/budget notes can still push the assembled output past
|
||||
// maxOutputChars (observed 30k against a 28k tier cap). A fat explore
|
||||
// payload persists in the agent's context and is re-read as cache-input
|
||||
// on every subsequent turn, so the overrun is paid many times over.
|
||||
const output = lines.join('\n');
|
||||
if (output.length > budget.maxOutputChars) {
|
||||
const cut = output.slice(0, budget.maxOutputChars);
|
||||
const lastNewline = cut.lastIndexOf('\n');
|
||||
const safe = lastNewline > budget.maxOutputChars * 0.8 ? cut.slice(0, lastNewline) : cut;
|
||||
return this.textResult(safe + '\n\n... (explore output truncated to budget — use codegraph_node or Read for more)');
|
||||
}
|
||||
return this.textResult(output);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1261,12 +1284,24 @@ export class ToolHandler {
|
||||
}
|
||||
|
||||
let code: string | null = null;
|
||||
let outline: string | null = null;
|
||||
|
||||
if (includeCode) {
|
||||
code = await cg.getCode(match.node.id);
|
||||
// For container symbols (class/interface/struct/…), the full body is the
|
||||
// sum of every method body — a wall of source (e.g. a 10k-char class)
|
||||
// that bloats context and is rarely needed in full. Return a structural
|
||||
// outline (members + signatures + line numbers) instead; the agent can
|
||||
// Read or codegraph_node a specific method for its body. Leaf symbols
|
||||
// (function/method/etc.) return their full body as before.
|
||||
if (CONTAINER_NODE_KINDS.has(match.node.kind)) {
|
||||
outline = this.buildContainerOutline(cg, match.node);
|
||||
}
|
||||
if (!outline) {
|
||||
code = await cg.getCode(match.node.id);
|
||||
}
|
||||
}
|
||||
|
||||
const formatted = this.formatNodeDetails(match.node, code) + match.note;
|
||||
const formatted = this.formatNodeDetails(match.node, code, outline) + match.note;
|
||||
return this.textResult(this.truncateOutput(formatted));
|
||||
}
|
||||
|
||||
@@ -1716,7 +1751,29 @@ export class ToolHandler {
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
private formatNodeDetails(node: Node, code: string | null): string {
|
||||
/**
|
||||
* Build a compact structural outline of a container symbol from its
|
||||
* indexed children (methods, fields, properties, …) — name, kind,
|
||||
* line number, and signature — so the agent gets the shape of a class
|
||||
* without the full source of every method. Returns '' when the container
|
||||
* has no indexed children, so the caller can fall back to full source.
|
||||
*/
|
||||
private buildContainerOutline(cg: CodeGraph, node: Node): string {
|
||||
const children = cg.getChildren(node.id)
|
||||
.filter(c => c.kind !== 'import' && c.kind !== 'export')
|
||||
.sort((a, b) => (a.startLine ?? 0) - (b.startLine ?? 0));
|
||||
if (children.length === 0) return '';
|
||||
|
||||
const lines = [`**Members (${children.length}):**`, ''];
|
||||
for (const c of children) {
|
||||
const loc = c.startLine ? `:${c.startLine}` : '';
|
||||
const sig = c.signature ? ` — \`${c.signature}\`` : '';
|
||||
lines.push(`- ${c.name} (${c.kind})${loc}${sig}`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
private formatNodeDetails(node: Node, code: string | null, outline?: string | null): string {
|
||||
const location = node.startLine ? `:${node.startLine}` : '';
|
||||
const lines: string[] = [
|
||||
`## ${node.name} (${node.kind})`,
|
||||
@@ -1733,7 +1790,10 @@ export class ToolHandler {
|
||||
lines.push('', node.docstring);
|
||||
}
|
||||
|
||||
if (code) {
|
||||
if (outline) {
|
||||
lines.push('', outline, '',
|
||||
`> Structural outline only. Read \`${node.filePath}\` or call codegraph_node on a specific member for its body.`);
|
||||
} else if (code) {
|
||||
lines.push('', '```' + node.language, code, '```');
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user