docs: one entry point for the three feedback metrics, and how to run them (CG-11)
Three per-metric docs told a maintainer what each number means; none said which one answers which question, which harness produces it, or how to read the arm table. agent-eval-feedback-metrics.md is that page — the metric → question map, when to reach for ab-new-vs-baseline.sh (isolates a change, both arms codegraph-on) versus run-all.sh (with vs without, a different question) versus bench-readme.sh, the worked CG-22 express table where all three read together, and the bucket → fix mapping. Not a fourth restatement: the derivations stay where they are and each doc now points here. The caveats that change how the summary table is read are carried over rather than dropped — allocation efficiency is relative (attribution is by citation, so same-question builds only, and never "codegraph wastes N%"), occupancy shares are Claude Code / 200k and do not transfer between hosts while the arm ratio does, sufficient is not correct, small-n throughout. Plus the contamination row, which means different things in the two harnesses and is the first thing to look at in both. Also records that the CG-8 7-repo bucket block no longer re-derives: bench-readme.sh overwrites /tmp/ab-readme, so the swept logs are gone. The current logs give a different distribution over the same 62 calls, and the CG-8-era and current classifiers agree exactly on them — so nothing moved under the metric, the corpus did. CG-13 re-establishes the baseline.
This commit is contained in:
@@ -1,5 +1,10 @@
|
||||
# Explore sufficiency
|
||||
|
||||
> One of three feedback metrics the agent-eval harness reports on every run.
|
||||
> [`agent-eval-feedback-metrics.md`](agent-eval-feedback-metrics.md) is the entry
|
||||
> point: which metric answers which question, which harness to run, and how to
|
||||
> read the arm-comparison table.
|
||||
|
||||
**What it measures:** whether a `codegraph_explore` response was *enough* — read
|
||||
off what the agent did next, which the harness was throwing away.
|
||||
|
||||
@@ -113,6 +118,17 @@ Read it as a baseline, not a verdict: these are three-turn sessions on hard
|
||||
flow questions, and "explored again" includes the legitimate second call on a
|
||||
repo whose budget is 2–3 calls.
|
||||
|
||||
**That block is a snapshot, and it no longer re-derives.** `bench-readme.sh`
|
||||
overwrites `/tmp/ab-readme` on every campaign, so the logs sitting there are not
|
||||
the ones swept above. Pooling the 14 with-arm sessions on disk as of 2026-08-05
|
||||
gives `explore again 47 (76%) · Read a file we returned 1 (2%) · Read a file we
|
||||
did not return 1 (2%) · Grep/Glob 0 (0%) · moved on 13 (21%)` over the same 62
|
||||
calls — checked against both the CG-8-era classifier and the current one, which
|
||||
agree exactly, so the classifier did not move under it. **CG-13 re-establishes
|
||||
the 7-repo baseline from a single campaign with all three metrics wired**; treat
|
||||
that as the number to compare against, and archive a campaign's logs elsewhere
|
||||
if you want a distribution to stay reproducible.
|
||||
|
||||
**`cg22/ab-express/run-baseline-1` — the allocation bucket, by hand.** Sequence:
|
||||
explore *"res.send Content-Type ETag generation"* → explore *"response.js
|
||||
res.send function body"* → `Read /…/t-base/lib/response.js`. The second explore
|
||||
|
||||
Reference in New Issue
Block a user