docs: one entry point for the three feedback metrics, and how to run them (CG-11)

Three per-metric docs told a maintainer what each number means; none said
which one answers which question, which harness produces it, or how to read
the arm table. agent-eval-feedback-metrics.md is that page — the metric →
question map, when to reach for ab-new-vs-baseline.sh (isolates a change,
both arms codegraph-on) versus run-all.sh (with vs without, a different
question) versus bench-readme.sh, the worked CG-22 express table where all
three read together, and the bucket → fix mapping. Not a fourth restatement:
the derivations stay where they are and each doc now points here.

The caveats that change how the summary table is read are carried over rather
than dropped — allocation efficiency is relative (attribution is by citation,
so same-question builds only, and never "codegraph wastes N%"), occupancy
shares are Claude Code / 200k and do not transfer between hosts while the arm
ratio does, sufficient is not correct, small-n throughout. Plus the
contamination row, which means different things in the two harnesses and is
the first thing to look at in both.

Also records that the CG-8 7-repo bucket block no longer re-derives:
bench-readme.sh overwrites /tmp/ab-readme, so the swept logs are gone. The
current logs give a different distribution over the same 62 calls, and the
CG-8-era and current classifiers agree exactly on them — so nothing moved
under the metric, the corpus did. CG-13 re-establishes the baseline.
This commit is contained in:
Colby McHenry
2026-08-05 00:59:58 -05:00
parent 3e8922dfad
commit 382791f11e
6 changed files with 252 additions and 0 deletions
@@ -1,5 +1,10 @@
# Explore allocation efficiency
> One of three feedback metrics the agent-eval harness reports on every run.
> [`agent-eval-feedback-metrics.md`](agent-eval-feedback-metrics.md) is the entry
> point: which metric answers which question, which harness to run, and how to
> read the arm-comparison table.
**What it measures:** of the bytes a `codegraph_explore` response spent, what
share went to files the agent's answer actually drew on.