github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(benchmarks): re-derive the token figures the result.usage bug touched

Browse Source 
Swept every benchmark doc for figures produced off `result.usage` and fixed
the ones that had raw logs to re-derive from.

residual-context-occupancy.md — the sonnet 3-turn throughput table. Re-derived
from the preserved logs: tokens saved 23% -> 56%, and vscode's "98% MORE tokens
with codegraph" was never real, it is 41% fewer. Cost, time and tool calls were
never affected by this field and are unchanged. The occupancy table itself is
measured off the timeline, so every number in it stands -- including the 82%
higher residual, which is the finding the document exists for.

call-sequence-analysis.md — this doc DIAGNOSED the bug and its reproduce block
claimed the aggregator summed per-turn tokens. It did not, until 04c0f8e. Noted,
with the three wrong results the gap produced: the excalidraw cut recorded here,
the sonnet campaign, and the Opus re-measure that invented a token regression.

answer-directly-vs-explore-agent.md — build 0.9.4, 2026-05-24, raw logs gone.
Cannot be re-derived, so flagged rather than silently left or invented: its
token figure is indicative, its turn/read/context findings do not depend on the
broken field and stand.

The remaining benchmark docs (allocation-ab-1500, dedup-cg20, allocation-
efficiency, feedback-metrics) carry no throughput tables — checked, clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-05 21:36:10 -05:00
co-authored by Claude Opus 5
parent da1f6121fd
commit d6d17288be
3 changed files with 48 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/benchmarks/answer-directly-vs-explore-agent.md
+9
View File
@@ -12,6 +12,15 @@ balloon on a 16× larger repo. Answering directly wins at **every** scale: same-
context than the delegation path, **zero file reads**, and ~28% fewer tokens. The
delegation-for-hygiene advantage stays marginal even on a large codebase.
> **Token figures here are unverified** (flagged 2026-08-05). This run predates the
> `result.usage` fix in `parse-run.mjs` (`04c0f8e`), so its "~28% fewer tokens" was
> likely measured off a field that reports only the last turn — see the gotcha in
> [`call-sequence-analysis.md`](call-sequence-analysis.md). The error under-counts
> whichever arm takes more turns, so if the delegation arm ran longer the real gap is
> *wider* than 28%; if turn counts were similar the figure is roughly right. The raw
> logs are gone, so this cannot be re-derived — treat the token number as indicative
> and the turn/read/context findings, which don't depend on that field, as sound.
## Methodology
- **Harness:** interactive Claude Code TUI driven via `scripts/agent-eval/itrun.sh` (tmux),