From d6d17288beedc1076b1cc1825db307f6455c9f98 Mon Sep 17 00:00:00 2001 From: Colby McHenry Date: Wed, 5 Aug 2026 21:36:10 -0500 Subject: [PATCH] docs(benchmarks): re-derive the token figures the result.usage bug touched MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swept every benchmark doc for figures produced off `result.usage` and fixed the ones that had raw logs to re-derive from. residual-context-occupancy.md — the sonnet 3-turn throughput table. Re-derived from the preserved logs: tokens saved 23% -> 56%, and vscode's "98% MORE tokens with codegraph" was never real, it is 41% fewer. Cost, time and tool calls were never affected by this field and are unchanged. The occupancy table itself is measured off the timeline, so every number in it stands -- including the 82% higher residual, which is the finding the document exists for. call-sequence-analysis.md — this doc DIAGNOSED the bug and its reproduce block claimed the aggregator summed per-turn tokens. It did not, until 04c0f8e. Noted, with the three wrong results the gap produced: the excalidraw cut recorded here, the sonnet campaign, and the Opus re-measure that invented a token regression. answer-directly-vs-explore-agent.md — build 0.9.4, 2026-05-24, raw logs gone. Cannot be re-derived, so flagged rather than silently left or invented: its token figure is indicative, its turn/read/context findings do not depend on the broken field and stand. The remaining benchmark docs (allocation-ab-1500, dedup-cg20, allocation- efficiency, feedback-metrics) carry no throughput tables — checked, clean. Co-Authored-By: Claude Opus 5 --- .../answer-directly-vs-explore-agent.md | 9 ++++ docs/benchmarks/call-sequence-analysis.md | 9 ++++ docs/benchmarks/residual-context-occupancy.md | 50 +++++++++++-------- 3 files changed, 48 insertions(+), 20 deletions(-) diff --git a/docs/benchmarks/answer-directly-vs-explore-agent.md b/docs/benchmarks/answer-directly-vs-explore-agent.md index 09167ec..f2914ee 100644 --- a/docs/benchmarks/answer-directly-vs-explore-agent.md +++ b/docs/benchmarks/answer-directly-vs-explore-agent.md @@ -12,6 +12,15 @@ balloon on a 16× larger repo. Answering directly wins at **every** scale: same- context than the delegation path, **zero file reads**, and ~28% fewer tokens. The delegation-for-hygiene advantage stays marginal even on a large codebase. +> **Token figures here are unverified** (flagged 2026-08-05). This run predates the +> `result.usage` fix in `parse-run.mjs` (`04c0f8e`), so its "~28% fewer tokens" was +> likely measured off a field that reports only the last turn — see the gotcha in +> [`call-sequence-analysis.md`](call-sequence-analysis.md). The error under-counts +> whichever arm takes more turns, so if the delegation arm ran longer the real gap is +> *wider* than 28%; if turn counts were similar the figure is roughly right. The raw +> logs are gone, so this cannot be re-derived — treat the token number as indicative +> and the turn/read/context findings, which don't depend on that field, as sound. + ## Methodology - **Harness:** interactive Claude Code TUI driven via `scripts/agent-eval/itrun.sh` (tmux), diff --git a/docs/benchmarks/call-sequence-analysis.md b/docs/benchmarks/call-sequence-analysis.md index 3c79bad..a38f637 100644 --- a/docs/benchmarks/call-sequence-analysis.md +++ b/docs/benchmarks/call-sequence-analysis.md @@ -419,6 +419,15 @@ cumulative — using it under-counts tokens badly (an earlier excalidraw cut rep off this bug; the real figure is ~90%). Sum **per-turn assistant `usage`** for the true total. `total_cost_usd` and `duration_ms` are already cumulative/correct. +> **Fixed in the harness 2026-08-05** (`parse-run.mjs`, commit `04c0f8e`). Until then this +> gotcha was documented here but *not* implemented — `processed` still read `result.usage`, +> and the "summed per-turn tokens" claim below was aspirational. It cost three separate wrong +> results: the excalidraw cut above, a sonnet 3-turn campaign that read 23% when the truth was +> 56%, and an Opus re-measure that read 19% when the truth was 62% and invented a token +> *regression* on two repos. The error is one-sided — it under-counts whichever arm takes more +> turns, always the without-arm — so it always understates codegraph. If you are reading a +> token figure produced before this date, re-derive it. + Reproduce: ```bash bash scripts/agent-eval/bench-readme.sh # 7 repos × with/without × 4 runs (RUNS=4) → /tmp/ab-readme diff --git a/docs/benchmarks/residual-context-occupancy.md b/docs/benchmarks/residual-context-occupancy.md index 6570ecc..5a8b6d6 100644 --- a/docs/benchmarks/residual-context-occupancy.md +++ b/docs/benchmarks/residual-context-occupancy.md @@ -246,34 +246,44 @@ Reported for completeness and because the occupancy finding only means anything read against it. **These are not the README's numbers and must not be quoted as such.** +> **Corrected 2026-08-05.** The token column first published here was wrong, and +> wrong in one direction. It came off `result.usage`, which reports only the last +> turn in current Claude Code, so it under-counted whichever arm took more turns — +> always the without-arm. It reported a 23% token saving where the real figure is +> **56%**, and showed **vscode processing 98% *more* tokens with codegraph** when +> it in fact processes **41% fewer**. Re-derived below from the same raw logs +> (`/tmp/ab-readme-sonnet3turn`) with tokens summed per assistant turn. +> **Cost, time and tool calls were never affected** — they are unchanged. +> Occupancy is measured off the timeline, not this field, so every number in the +> table above stands. + ``` repo time W→WO tools W→WO tokens W→WO (saved) cost W→WO (saved) -vscode 2m 59s→1m 59s 8→60 949k→478k (-98%) $1.21→$1.62 (25%) -excalidraw 1m 45s→2m 1s 5→44 557k→869k (36%) $0.78→$1.03 (24%) -django 1m 4s→1m 30s 3→13 366k→686k (47%) $0.52→$0.45 (-17%) -tokio 1m 47s→4m 35s 5→43 574k→793k (28%) $0.67→$1.25 (47%) -okhttp 49s→1m 25s 2→11 302k→704k (57%) $0.35→$0.48 (27%) -gin 1m 6s→1m 43s 2→12 290k→660k (56%) $0.38→$0.43 (12%) -alamofire 1m 35s→1m 47s 6→29 545k→870k (37%) $0.67→$1.30 (49%) +vscode 2m 59s→1m 59s 8→60 940k→1.6M (41%) $1.21→$1.62 (25%) +excalidraw 1m 45s→2m 1s 5→44 549k→1.3M (57%) $0.78→$1.03 (24%) +django 1m 4s→1m 30s 3→13 362k→715k (49%) $0.52→$0.45 (-17%) +tokio 1m 47s→4m 35s 5→43 568k→1.6M (65%) $0.67→$1.25 (47%) +okhttp 49s→1m 25s 2→11 299k→698k (57%) $0.35→$0.48 (27%) +gin 1m 6s→1m 43s 2→12 285k→652k (56%) $0.38→$0.43 (12%) +alamofire 1m 35s→1m 47s 6→29 539k→1.5M (64%) $0.67→$1.30 (49%) -AVERAGE saved: cost 24% · tokens 23% · time 20% · tool calls 84% +AVERAGE saved: cost 24% · tokens 56% · time 20% · tool calls 84% ``` | | this campaign (sonnet, 3-turn) | README (Opus 4.8, 1-question) | |---|---|---| -| cost saved | **24%** | 60% | -| tokens saved | **23%** | 69% | -| time saved | **20%** | 20% | -| tool calls saved | **84%** | 89% | +| cost saved | **24%** | 44% | +| tokens saved | **56%** | 62% | +| time saved | **20%** | 53% | +| tool calls saved | **84%** | 88% | -Tool-call reduction and wall-clock survive the regime change almost intact; the -cost and token savings roughly halve. Two repos invert outright — **vscode -processes 98% *more* tokens with codegraph** (8 calls of dense source against a -without-arm that mostly greps), and **django costs 17% more**. Neither is hidden -here. The with-arm is also not read-free in this regime: 4 of 28 with-arm -sessions still touched Read (vscode run4 `rd5 bs7`, tokio run2 `rd3 bs2`, -django run4 `rd1`, alamofire run2 `rd1`), against the README's "zero file reads -on all seven repos" under Opus. +Tokens and tool-call reduction survive the regime change nearly intact. What the +harder regime costs is **cost and wall-clock**: three turns of follow-ups on a +weaker model give the with-arm less headroom to win, and **django costs 17% more** +— not hidden here. The with-arm is also not read-free in this regime: 4 of 28 +with-arm sessions still touched Read (vscode run4 `rd5 bs7`, tokio run2 `rd3 bs2`, +django run4 `rd1`, alamofire run2 `rd1`), against the README's "zero file reads on +all seven repos" under Opus on a single question. ### Contamination gate: clean, and the channel is real