From a7db24d0c08cbedcd1b0aff100bc0079f666ee14 Mon Sep 17 00:00:00 2001 From: Colby McHenry Date: Wed, 5 Aug 2026 14:30:34 -0500 Subject: [PATCH] docs(readme): disclose the context-footprint side of the benchmark The benchmark table measures throughput -- tokens processed, tools called, cost to reach one answer. It has never measured what is still resident in the window afterward, and on that axis codegraph costs more: ~80% more retrieval context left behind than a file-reading agent, on all seven repos. That is the axis issue #1500 reported, and it is structural rather than a defect -- one dense payload that answers the question and stays, versus grep-and-read churn that evicts. Worth stating plainly next to the cost note rather than leaving a user to discover it in a long session. The Opus 4.8 single-question figures in the table are deliberately untouched: the occupancy campaign ran sonnet / 3-turn, a different regime, and nothing measured there licenses restating them. Co-Authored-By: Claude Opus 5 --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index dccebd3..0bc4c03 100644 --- a/README.md +++ b/README.md @@ -194,6 +194,8 @@ When an AI agent needs to understand code — to answer a question or make a cha > **A note on cost:** CodeGraph's win on *every* codebase is precision — the agent stops crawling files and answers from the graph. On current models that precision is also a large direct saving: the 2026-07 re-validation measured **60% lower cost and 69% fewer tokens on average** across the seven benchmark repos, because a strong model *without* the graph burns millions of tokens re-deriving structure. The savings scale with repo size and tangle — dramatic on VS-Code-class trees, modest on a 100-file project — and compound across a team's daily agent usage. +> **A note on context:** the numbers above measure *throughput* — tokens processed, tools called, dollars spent to reach one answer. They don't measure what is still sitting in your context window afterward, and on that axis CodeGraph costs **more**, not less. Across the same seven repos in multi-turn sessions, CodeGraph's responses leave about **80% more retrieval context resident** at the end of a session than a file-reading agent's do — on VS Code, 67k tokens against 18k. The mechanism is the same one that makes it fast: CodeGraph returns one dense, verbatim payload that answers the question and then stays in the window, where a grep-and-read agent churns through many small results that get evicted. Fewer tokens *processed* and a larger persistent *footprint* are both real at once. If you run long sessions in a small window, budget for it. Measured per-repo: [`docs/benchmarks/residual-context-occupancy.md`](docs/benchmarks/residual-context-occupancy.md). + ### Benchmark Results Tested across **7 real-world open-source codebases** spanning 7 languages, comparing an agent (Claude Code, headless) answering one architecture question **with** and **without** CodeGraph, at the **median of 4 runs per arm**. _Re-validated 2026-07-21 on **Claude Opus 4.8** against the current build — the Rust kernel plus this cycle's resolution overhaul._