docs: the occupancy baseline says our residual is higher — write that down (CG-13)
Fills the empty RESULTS placeholder with the 2026-08-05 campaign
(bgjob-6d357cd2: 7 repos x 2 arms x 4 runs x 3 turns, 137 min).
The finding is not the flattering one. Retrieval residual is 82% HIGHER
with codegraph and share-of-context 27% higher, on all seven repos —
vscode 67k resident against 18k. At the same time six of seven
without-arms *process* more total tokens (gin 660k vs 290k) while
leaving less behind. Both are true: one dense verbatim payload stays
resident where many small Read/Grep results evict. This corroborates
issue #1500 on our own harness; the aggregator used to print it as
"-82% lower with codegraph" until the sign bug at 520ed9d.
Also:
- States the regime everywhere. This ran claude-sonnet-5 / 3-turn; the
README's table is Opus 4.8 / single-question. Measured 24/23/20/84
against the published 60/69/20/89 — model and turn count, not
contamination. Records the two inversions honestly (vscode processes
98% more tokens, django costs 17% more) and that 4 of 28 with-arm
sessions still touched Read.
- Corrects the "Settled" section, which claimed a 7-repo baseline
existed before one did, and adds the unclaimed Opus rerun.
- Records the contamination gate: 0 CLI calls returned output in 56
sessions, but 29 attempts were blocked — 26 of 28 without-arm
sessions tried. no-cli-shim.sh is load-bearing, not precautionary.
- Records the secondary readings as absolute, not before/after: 86.7%
allocation efficiency pooled over 110 calls, read-of-a-file-we-
returned 2%, explore-again 73% and ambiguous by construction.
README.md is deliberately untouched — restating its numbers from sonnet
3-turn data would be wrong. A proposed README paragraph is drafted at
the end of the benchmark doc for the maintainer to accept or reject.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
520ed9d933
commit
5dd4db68cd
@@ -54,7 +54,10 @@ table still renders against whichever arm's logs are already in `$AGENT_EVAL_OUT
|
||||
|
||||
**A campaign — `bench-readme.sh`.** The 7 README repos, three turns each,
|
||||
`RUNS` per arm, through `run-all.sh` — so every run in a campaign carries all
|
||||
three metrics. Aggregate with `parse-bench-readme.mjs`.
|
||||
three metrics. Aggregate with `parse-bench-readme.mjs`. One has been run:
|
||||
[the 2026-08-05 baseline](residual-context-occupancy.md#baseline-the-7-readme-repos)
|
||||
(sonnet, 3 turns, 4 runs/arm) — read its regime box before comparing anything to
|
||||
it, and note that it is **not** the regime the README's table was published in.
|
||||
|
||||
**A log you already have.** `parse-run.mjs <run.jsonl> [run.tN.jsonl …]` prints
|
||||
the three blocks for any stream-json log; `--brief` drops the numbered call
|
||||
|
||||
Reference in New Issue
Block a user