docs: one entry point for the three feedback metrics, and how to run them (CG-11)

Three per-metric docs told a maintainer what each number means; none said
which one answers which question, which harness produces it, or how to read
the arm table. agent-eval-feedback-metrics.md is that page — the metric →
question map, when to reach for ab-new-vs-baseline.sh (isolates a change,
both arms codegraph-on) versus run-all.sh (with vs without, a different
question) versus bench-readme.sh, the worked CG-22 express table where all
three read together, and the bucket → fix mapping. Not a fourth restatement:
the derivations stay where they are and each doc now points here.

The caveats that change how the summary table is read are carried over rather
than dropped — allocation efficiency is relative (attribution is by citation,
so same-question builds only, and never "codegraph wastes N%"), occupancy
shares are Claude Code / 200k and do not transfer between hosts while the arm
ratio does, sufficient is not correct, small-n throughout. Plus the
contamination row, which means different things in the two harnesses and is
the first thing to look at in both.

Also records that the CG-8 7-repo bucket block no longer re-derives:
bench-readme.sh overwrites /tmp/ab-readme, so the swept logs are gone. The
current logs give a different distribution over the same 62 calls, and the
CG-8-era and current classifiers agree exactly on them — so nothing moved
under the metric, the corpus did. CG-13 re-establishes the baseline.
This commit is contained in:
Colby McHenry
2026-08-05 00:59:58 -05:00
parent 3e8922dfad
commit 382791f11e
6 changed files with 252 additions and 0 deletions
+16
View File
@@ -1,5 +1,10 @@
# Explore sufficiency
> One of three feedback metrics the agent-eval harness reports on every run.
> [`agent-eval-feedback-metrics.md`](agent-eval-feedback-metrics.md) is the entry
> point: which metric answers which question, which harness to run, and how to
> read the arm-comparison table.
**What it measures:** whether a `codegraph_explore` response was *enough* — read
off what the agent did next, which the harness was throwing away.
@@ -113,6 +118,17 @@ Read it as a baseline, not a verdict: these are three-turn sessions on hard
flow questions, and "explored again" includes the legitimate second call on a
repo whose budget is 23 calls.
**That block is a snapshot, and it no longer re-derives.** `bench-readme.sh`
overwrites `/tmp/ab-readme` on every campaign, so the logs sitting there are not
the ones swept above. Pooling the 14 with-arm sessions on disk as of 2026-08-05
gives `explore again 47 (76%) · Read a file we returned 1 (2%) · Read a file we
did not return 1 (2%) · Grep/Glob 0 (0%) · moved on 13 (21%)` over the same 62
calls — checked against both the CG-8-era classifier and the current one, which
agree exactly, so the classifier did not move under it. **CG-13 re-establishes
the 7-repo baseline from a single campaign with all three metrics wired**; treat
that as the number to compare against, and archive a campaign's logs elsewhere
if you want a distribution to stay reproducible.
**`cg22/ab-express/run-baseline-1` — the allocation bucket, by hand.** Sequence:
explore *"res.send Content-Type ETag generation"* → explore *"response.js
res.send function body"* → `Read /…/t-base/lib/response.js`. The second explore