Three per-metric docs told a maintainer what each number means; none said
which one answers which question, which harness produces it, or how to read
the arm table. agent-eval-feedback-metrics.md is that page — the metric →
question map, when to reach for ab-new-vs-baseline.sh (isolates a change,
both arms codegraph-on) versus run-all.sh (with vs without, a different
question) versus bench-readme.sh, the worked CG-22 express table where all
three read together, and the bucket → fix mapping. Not a fourth restatement:
the derivations stay where they are and each doc now points here.
The caveats that change how the summary table is read are carried over rather
than dropped — allocation efficiency is relative (attribution is by citation,
so same-question builds only, and never "codegraph wastes N%"), occupancy
shares are Claude Code / 200k and do not transfer between hosts while the arm
ratio does, sufficient is not correct, small-n throughout. Plus the
contamination row, which means different things in the two harnesses and is
the first thing to look at in both.
Also records that the CG-8 7-repo bucket block no longer re-derives:
bench-readme.sh overwrites /tmp/ab-readme, so the swept logs are gone. The
current logs give a different distribution over the same 62 calls, and the
CG-8-era and current classifiers agree exactly on them — so nothing moved
under the metric, the corpus did. CG-13 re-establishes the baseline.
The 14 multi-turn with-arm sessions of the README corpus exercise every bucket
(62 calls), so the sweep is no longer one repo family: 47% explored again, 11%
Read a file we returned, 2% Read a file we did not, 23% Grep/Glob, 18% moved on.
Flagged as a baseline rather than a verdict -- three-turn sessions on hard flow
questions, and "explored again" includes the legitimate second call on a repo
whose budget is 2-3.
The recall bucket's one real instance is worth reading: explore returned
InteractiveCanvas.tsx and named StaticCanvas.tsx without shipping it, and the
agent went and read exactly that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records what each bucket means and which fix it points at, the four rules that
keep the classification honest (same-message calls, bookkeeping tools, subagent
threads, earlier-explore files), and the three real transcripts it was
hand-checked against -- including the excalidraw canvasNonce run, where it
independently found the data-flow frontier CLAUDE.md already documents: 0%
sufficient, without being told what to look for.
Also states what it does NOT say: sufficient is not correct, one Read is a vote
rather than a proof, and bucket 1 is ambiguous by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Colby McHenry
