docs(benchmarks): re-derive the token figures the result.usage bug touched
Swept every benchmark doc for figures produced off `result.usage` and fixed
the ones that had raw logs to re-derive from.
residual-context-occupancy.md — the sonnet 3-turn throughput table. Re-derived
from the preserved logs: tokens saved 23% -> 56%, and vscode's "98% MORE tokens
with codegraph" was never real, it is 41% fewer. Cost, time and tool calls were
never affected by this field and are unchanged. The occupancy table itself is
measured off the timeline, so every number in it stands -- including the 82%
higher residual, which is the finding the document exists for.
call-sequence-analysis.md — this doc DIAGNOSED the bug and its reproduce block
claimed the aggregator summed per-turn tokens. It did not, until 04c0f8e. Noted,
with the three wrong results the gap produced: the excalidraw cut recorded here,
the sonnet campaign, and the Opus re-measure that invented a token regression.
answer-directly-vs-explore-agent.md — build 0.9.4, 2026-05-24, raw logs gone.
Cannot be re-derived, so flagged rather than silently left or invented: its
token figure is indicative, its turn/read/context findings do not depend on the
broken field and stand.
The remaining benchmark docs (allocation-ab-1500, dedup-cg20, allocation-
efficiency, feedback-metrics) carry no throughput tables — checked, clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
co-authored by
Claude Opus 5
Claude Opus 5
parent
da1f6121fd
commit
d6d17288be
@@ -12,6 +12,15 @@ balloon on a 16× larger repo. Answering directly wins at **every** scale: same-
|
||||
context than the delegation path, **zero file reads**, and ~28% fewer tokens. The
|
||||
delegation-for-hygiene advantage stays marginal even on a large codebase.
|
||||
|
||||
> **Token figures here are unverified** (flagged 2026-08-05). This run predates the
|
||||
> `result.usage` fix in `parse-run.mjs` (`04c0f8e`), so its "~28% fewer tokens" was
|
||||
> likely measured off a field that reports only the last turn — see the gotcha in
|
||||
> [`call-sequence-analysis.md`](call-sequence-analysis.md). The error under-counts
|
||||
> whichever arm takes more turns, so if the delegation arm ran longer the real gap is
|
||||
> *wider* than 28%; if turn counts were similar the figure is roughly right. The raw
|
||||
> logs are gone, so this cannot be re-derived — treat the token number as indicative
|
||||
> and the turn/read/context findings, which don't depend on that field, as sound.
|
||||
|
||||
## Methodology
|
||||
|
||||
- **Harness:** interactive Claude Code TUI driven via `scripts/agent-eval/itrun.sh` (tmux),
|
||||
|
||||
@@ -419,6 +419,15 @@ cumulative — using it under-counts tokens badly (an earlier excalidraw cut rep
|
||||
off this bug; the real figure is ~90%). Sum **per-turn assistant `usage`** for the true total.
|
||||
`total_cost_usd` and `duration_ms` are already cumulative/correct.
|
||||
|
||||
> **Fixed in the harness 2026-08-05** (`parse-run.mjs`, commit `04c0f8e`). Until then this
|
||||
> gotcha was documented here but *not* implemented — `processed` still read `result.usage`,
|
||||
> and the "summed per-turn tokens" claim below was aspirational. It cost three separate wrong
|
||||
> results: the excalidraw cut above, a sonnet 3-turn campaign that read 23% when the truth was
|
||||
> 56%, and an Opus re-measure that read 19% when the truth was 62% and invented a token
|
||||
> *regression* on two repos. The error is one-sided — it under-counts whichever arm takes more
|
||||
> turns, always the without-arm — so it always understates codegraph. If you are reading a
|
||||
> token figure produced before this date, re-derive it.
|
||||
|
||||
Reproduce:
|
||||
```bash
|
||||
bash scripts/agent-eval/bench-readme.sh # 7 repos × with/without × 4 runs (RUNS=4) → /tmp/ab-readme
|
||||
|
||||
@@ -246,34 +246,44 @@ Reported for completeness and because the occupancy finding only means anything
|
||||
read against it. **These are not the README's numbers and must not be quoted as
|
||||
such.**
|
||||
|
||||
> **Corrected 2026-08-05.** The token column first published here was wrong, and
|
||||
> wrong in one direction. It came off `result.usage`, which reports only the last
|
||||
> turn in current Claude Code, so it under-counted whichever arm took more turns —
|
||||
> always the without-arm. It reported a 23% token saving where the real figure is
|
||||
> **56%**, and showed **vscode processing 98% *more* tokens with codegraph** when
|
||||
> it in fact processes **41% fewer**. Re-derived below from the same raw logs
|
||||
> (`/tmp/ab-readme-sonnet3turn`) with tokens summed per assistant turn.
|
||||
> **Cost, time and tool calls were never affected** — they are unchanged.
|
||||
> Occupancy is measured off the timeline, not this field, so every number in the
|
||||
> table above stands.
|
||||
|
||||
```
|
||||
repo time W→WO tools W→WO tokens W→WO (saved) cost W→WO (saved)
|
||||
vscode 2m 59s→1m 59s 8→60 949k→478k (-98%) $1.21→$1.62 (25%)
|
||||
excalidraw 1m 45s→2m 1s 5→44 557k→869k (36%) $0.78→$1.03 (24%)
|
||||
django 1m 4s→1m 30s 3→13 366k→686k (47%) $0.52→$0.45 (-17%)
|
||||
tokio 1m 47s→4m 35s 5→43 574k→793k (28%) $0.67→$1.25 (47%)
|
||||
okhttp 49s→1m 25s 2→11 302k→704k (57%) $0.35→$0.48 (27%)
|
||||
gin 1m 6s→1m 43s 2→12 290k→660k (56%) $0.38→$0.43 (12%)
|
||||
alamofire 1m 35s→1m 47s 6→29 545k→870k (37%) $0.67→$1.30 (49%)
|
||||
vscode 2m 59s→1m 59s 8→60 940k→1.6M (41%) $1.21→$1.62 (25%)
|
||||
excalidraw 1m 45s→2m 1s 5→44 549k→1.3M (57%) $0.78→$1.03 (24%)
|
||||
django 1m 4s→1m 30s 3→13 362k→715k (49%) $0.52→$0.45 (-17%)
|
||||
tokio 1m 47s→4m 35s 5→43 568k→1.6M (65%) $0.67→$1.25 (47%)
|
||||
okhttp 49s→1m 25s 2→11 299k→698k (57%) $0.35→$0.48 (27%)
|
||||
gin 1m 6s→1m 43s 2→12 285k→652k (56%) $0.38→$0.43 (12%)
|
||||
alamofire 1m 35s→1m 47s 6→29 539k→1.5M (64%) $0.67→$1.30 (49%)
|
||||
|
||||
AVERAGE saved: cost 24% · tokens 23% · time 20% · tool calls 84%
|
||||
AVERAGE saved: cost 24% · tokens 56% · time 20% · tool calls 84%
|
||||
```
|
||||
|
||||
| | this campaign (sonnet, 3-turn) | README (Opus 4.8, 1-question) |
|
||||
|---|---|---|
|
||||
| cost saved | **24%** | 60% |
|
||||
| tokens saved | **23%** | 69% |
|
||||
| time saved | **20%** | 20% |
|
||||
| tool calls saved | **84%** | 89% |
|
||||
| cost saved | **24%** | 44% |
|
||||
| tokens saved | **56%** | 62% |
|
||||
| time saved | **20%** | 53% |
|
||||
| tool calls saved | **84%** | 88% |
|
||||
|
||||
Tool-call reduction and wall-clock survive the regime change almost intact; the
|
||||
cost and token savings roughly halve. Two repos invert outright — **vscode
|
||||
processes 98% *more* tokens with codegraph** (8 calls of dense source against a
|
||||
without-arm that mostly greps), and **django costs 17% more**. Neither is hidden
|
||||
here. The with-arm is also not read-free in this regime: 4 of 28 with-arm
|
||||
sessions still touched Read (vscode run4 `rd5 bs7`, tokio run2 `rd3 bs2`,
|
||||
django run4 `rd1`, alamofire run2 `rd1`), against the README's "zero file reads
|
||||
on all seven repos" under Opus.
|
||||
Tokens and tool-call reduction survive the regime change nearly intact. What the
|
||||
harder regime costs is **cost and wall-clock**: three turns of follow-ups on a
|
||||
weaker model give the with-arm less headroom to win, and **django costs 17% more**
|
||||
— not hidden here. The with-arm is also not read-free in this regime: 4 of 28
|
||||
with-arm sessions still touched Read (vscode run4 `rd5 bs7`, tokio run2 `rd3 bs2`,
|
||||
django run4 `rd1`, alamofire run2 `rd1`), against the README's "zero file reads on
|
||||
all seven repos" under Opus on a single question.
|
||||
|
||||
### Contamination gate: clean, and the channel is real
|
||||
|
||||
|
||||
Reference in New Issue
Block a user