Fills the empty RESULTS placeholder with the 2026-08-05 campaign
(bgjob-6d357cd2: 7 repos x 2 arms x 4 runs x 3 turns, 137 min).
The finding is not the flattering one. Retrieval residual is 82% HIGHER
with codegraph and share-of-context 27% higher, on all seven repos —
vscode 67k resident against 18k. At the same time six of seven
without-arms *process* more total tokens (gin 660k vs 290k) while
leaving less behind. Both are true: one dense verbatim payload stays
resident where many small Read/Grep results evict. This corroborates
issue #1500 on our own harness; the aggregator used to print it as
"-82% lower with codegraph" until the sign bug at 520ed9d.
Also:
- States the regime everywhere. This ran claude-sonnet-5 / 3-turn; the
README's table is Opus 4.8 / single-question. Measured 24/23/20/84
against the published 60/69/20/89 — model and turn count, not
contamination. Records the two inversions honestly (vscode processes
98% more tokens, django costs 17% more) and that 4 of 28 with-arm
sessions still touched Read.
- Corrects the "Settled" section, which claimed a 7-repo baseline
existed before one did, and adds the unclaimed Opus rerun.
- Records the contamination gate: 0 CLI calls returned output in 56
sessions, but 29 attempts were blocked — 26 of 28 without-arm
sessions tried. no-cli-shim.sh is load-bearing, not precautionary.
- Records the secondary readings as absolute, not before/after: 86.7%
allocation efficiency pooled over 110 calls, read-of-a-file-we-
returned 2%, explore-again 73% and ambiguous by construction.
README.md is deliberately untouched — restating its numbers from sonnet
3-turn data would be wrong. A proposed README paragraph is drafted at
the end of the benchmark doc for the maintainer to accept or reject.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three per-metric docs told a maintainer what each number means; none said
which one answers which question, which harness produces it, or how to read
the arm table. agent-eval-feedback-metrics.md is that page — the metric →
question map, when to reach for ab-new-vs-baseline.sh (isolates a change,
both arms codegraph-on) versus run-all.sh (with vs without, a different
question) versus bench-readme.sh, the worked CG-22 express table where all
three read together, and the bucket → fix mapping. Not a fourth restatement:
the derivations stay where they are and each doc now points here.
The caveats that change how the summary table is read are carried over rather
than dropped — allocation efficiency is relative (attribution is by citation,
so same-question builds only, and never "codegraph wastes N%"), occupancy
shares are Claude Code / 200k and do not transfer between hosts while the arm
ratio does, sufficient is not correct, small-n throughout. Plus the
contamination row, which means different things in the two harnesses and is
the first thing to look at in both.
Also records that the CG-8 7-repo bucket block no longer re-derives:
bench-readme.sh overwrites /tmp/ab-readme, so the swept logs are gone. The
current logs give a different distribution over the same 62 calls, and the
CG-8-era and current classifiers agree exactly on them — so nothing moved
under the metric, the corpus did. CG-13 re-establishes the baseline.
Records what each bucket means and which fix it points at, the four rules that
keep the classification honest (same-message calls, bookkeeping tools, subagent
threads, earlier-explore files), and the three real transcripts it was
hand-checked against -- including the excalidraw canvasNonce run, where it
independently found the data-flow frontier CLAUDE.md already documents: 0%
sufficient, without being told what to look for.
Also states what it does NOT say: sufficient is not correct, one Read is a vote
rather than a proof, and bucket 1 is ambiguous by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An agent denied `codegraph` on PATH ran `find / -maxdepth 4 -iname "*codegraph*"`,
found the binary, and invoked it by ABSOLUTE PATH — 12 times in one without-arm
run. So block the invocation itself with a PreToolUse hook on Bash, written into
the run's output dir as an artifact alongside the MCP configs rather than as a
repo file.
The pattern matches command positions only, so looking is still allowed and only
using is denied: `grep codegraph src/`, `ls .codegraph` and `which codegraph`
pass through, while `codegraph explore`, `/abs/path/codegraph …`, `cd x &&
codegraph …` and `VAR=1 codegraph …` are refused. run-all.sh proves both
directions at startup and refuses to run if either fails. parse-run.mjs's
detector uses the same rule, so prevention and detection cannot drift — and it no
longer false-positives on the corpus path, which contains the word codegraph.
Verified end-to-end: the without-arm now probes with `ls .codegraph; which
codegraph`, finds nothing usable, and falls back to Read/Bash.
parse-run.mjs --selftest runs the math over synthetic transcripts with known
answers: attribution, message.id dedupe, compact_boundary, FIFO micro-compaction,
and multi-turn stitching. It found a real bug. A gap where the window also SHED
content has a delta far below what was added, which reads as absurdly dense text
and dragged the whole run's ratio with it -- a shed gap in the fixture pushed
2.5 chars/tok to 4.4 and left the wrong result resident. Shedding can only push a
gap's ratio up, so the calibration now takes the lower median as its centre,
drops gaps well above it, and pools the rest. Runs that never shed are unaffected
(gin and vscode re-measure identically).
Also drafts docs/benchmarks/residual-context-occupancy.md -- method, error bar,
and the limitations this metric does not settle. Baseline numbers to follow.
Colby McHenryandClaude Opus 5