docs(benchmarks): re-derive the token figures the result.usage bug touched
Swept every benchmark doc for figures produced off `result.usage` and fixed
the ones that had raw logs to re-derive from.
residual-context-occupancy.md — the sonnet 3-turn throughput table. Re-derived
from the preserved logs: tokens saved 23% -> 56%, and vscode's "98% MORE tokens
with codegraph" was never real, it is 41% fewer. Cost, time and tool calls were
never affected by this field and are unchanged. The occupancy table itself is
measured off the timeline, so every number in it stands -- including the 82%
higher residual, which is the finding the document exists for.
call-sequence-analysis.md — this doc DIAGNOSED the bug and its reproduce block
claimed the aggregator summed per-turn tokens. It did not, until 04c0f8e. Noted,
with the three wrong results the gap produced: the excalidraw cut recorded here,
the sonnet campaign, and the Opus re-measure that invented a token regression.
answer-directly-vs-explore-agent.md — build 0.9.4, 2026-05-24, raw logs gone.
Cannot be re-derived, so flagged rather than silently left or invented: its
token figure is indicative, its turn/read/context findings do not depend on the
broken field and stand.
The remaining benchmark docs (allocation-ab-1500, dedup-cg20, allocation-
efficiency, feedback-metrics) carry no throughput tables — checked, clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
da1f6121fd
commit
d6d17288be
@@ -419,6 +419,15 @@ cumulative — using it under-counts tokens badly (an earlier excalidraw cut rep
|
||||
off this bug; the real figure is ~90%). Sum **per-turn assistant `usage`** for the true total.
|
||||
`total_cost_usd` and `duration_ms` are already cumulative/correct.
|
||||
|
||||
> **Fixed in the harness 2026-08-05** (`parse-run.mjs`, commit `04c0f8e`). Until then this
|
||||
> gotcha was documented here but *not* implemented — `processed` still read `result.usage`,
|
||||
> and the "summed per-turn tokens" claim below was aspirational. It cost three separate wrong
|
||||
> results: the excalidraw cut above, a sonnet 3-turn campaign that read 23% when the truth was
|
||||
> 56%, and an Opus re-measure that read 19% when the truth was 62% and invented a token
|
||||
> *regression* on two repos. The error is one-sided — it under-counts whichever arm takes more
|
||||
> turns, always the without-arm — so it always understates codegraph. If you are reading a
|
||||
> token figure produced before this date, re-derive it.
|
||||
|
||||
Reproduce:
|
||||
```bash
|
||||
bash scripts/agent-eval/bench-readme.sh # 7 repos × with/without × 4 runs (RUNS=4) → /tmp/ab-readme
|
||||
|
||||
Reference in New Issue
Block a user