docs(benchmarks): re-derive the token figures the result.usage bug touched

Swept every benchmark doc for figures produced off `result.usage` and fixed
the ones that had raw logs to re-derive from.

residual-context-occupancy.md — the sonnet 3-turn throughput table. Re-derived
from the preserved logs: tokens saved 23% -> 56%, and vscode's "98% MORE tokens
with codegraph" was never real, it is 41% fewer. Cost, time and tool calls were
never affected by this field and are unchanged. The occupancy table itself is
measured off the timeline, so every number in it stands -- including the 82%
higher residual, which is the finding the document exists for.

call-sequence-analysis.md — this doc DIAGNOSED the bug and its reproduce block
claimed the aggregator summed per-turn tokens. It did not, until 04c0f8e. Noted,
with the three wrong results the gap produced: the excalidraw cut recorded here,
the sonnet campaign, and the Opus re-measure that invented a token regression.

answer-directly-vs-explore-agent.md — build 0.9.4, 2026-05-24, raw logs gone.
Cannot be re-derived, so flagged rather than silently left or invented: its
token figure is indicative, its turn/read/context findings do not depend on the
broken field and stand.

The remaining benchmark docs (allocation-ab-1500, dedup-cg20, allocation-
efficiency, feedback-metrics) carry no throughput tables — checked, clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-05 21:36:10 -05:00
co-authored by Claude Opus 5
parent da1f6121fd
commit d6d17288be
3 changed files with 48 additions and 20 deletions
@@ -419,6 +419,15 @@ cumulative — using it under-counts tokens badly (an earlier excalidraw cut rep
off this bug; the real figure is ~90%). Sum **per-turn assistant `usage`** for the true total.
`total_cost_usd` and `duration_ms` are already cumulative/correct.
> **Fixed in the harness 2026-08-05** (`parse-run.mjs`, commit `04c0f8e`). Until then this
> gotcha was documented here but *not* implemented — `processed` still read `result.usage`,
> and the "summed per-turn tokens" claim below was aspirational. It cost three separate wrong
> results: the excalidraw cut above, a sonnet 3-turn campaign that read 23% when the truth was
> 56%, and an Opus re-measure that read 19% when the truth was 62% and invented a token
> *regression* on two repos. The error is one-sided — it under-counts whichever arm takes more
> turns, always the without-arm — so it always understates codegraph. If you are reading a
> token figure produced before this date, re-derive it.
Reproduce:
```bash
bash scripts/agent-eval/bench-readme.sh # 7 repos × with/without × 4 runs (RUNS=4) → /tmp/ab-readme