From fa15d1046a11723aa5d2b20d68b99fbc2ee2efde Mon Sep 17 00:00:00 2001 From: Colby McHenry Date: Wed, 5 Aug 2026 00:44:30 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20allocation=20efficiency=20=E2=80=94=20t?= =?UTF-8?q?he=20metric,=20its=20guards,=20and=20the=20103-run=20baseline?= =?UTF-8?q?=20(CG-9)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the sweep over every A/B log on this machine (103 sessions, 297 explore calls, 0 crashes) and, more usefully, the new-vs-baseline arm table the metric exists for: express 82% → 100%, cg21/client-go 67% → 95%, two pairs going the other way. States the caveat in the places it can be misread: the corpus median sits in the eighties because these are flow questions whose answers name most of the chain, the metric is byte-weighted, and an agent can use a file without citing it. It compares two builds on one question; it is not an absolute waste figure. --- docs/benchmarks/explore-allocation-ab-1500.md | 5 + .../explore-allocation-efficiency.md | 195 ++++++++++++++++++ 2 files changed, 200 insertions(+) create mode 100644 docs/benchmarks/explore-allocation-efficiency.md diff --git a/docs/benchmarks/explore-allocation-ab-1500.md b/docs/benchmarks/explore-allocation-ab-1500.md index d354787..db46308 100644 --- a/docs/benchmarks/explore-allocation-ab-1500.md +++ b/docs/benchmarks/explore-allocation-ab-1500.md @@ -51,6 +51,11 @@ rendered markdown of the responses the agent actually received. The CG-4 diagnos only exists on the new build, so it cannot measure the baseline arm; the markdown parse is the only instrument that measures both arms the same way. +That `--answer ` is hand-specified ground truth. Every run now also reports the same +intersection inferred from the agent's own final answer, so it needs no per-question setup — +see [`explore-allocation-efficiency.md`](explore-allocation-efficiency.md), which re-scores +the arms below (express: 82% baseline → 100% new). + | Repo | Lang | Files | Generated | Tier | Role | |---|---|---|---|---|---| | `kubernetes/client-go` | Go | 2,454 | 2,001 | medium (2 calls / 28K) | **the #1500 shape** — generated CRUD beside hand-written machinery | diff --git a/docs/benchmarks/explore-allocation-efficiency.md b/docs/benchmarks/explore-allocation-efficiency.md new file mode 100644 index 0000000..2deab65 --- /dev/null +++ b/docs/benchmarks/explore-allocation-efficiency.md @@ -0,0 +1,195 @@ +# Explore allocation efficiency + +**What it measures:** of the bytes a `codegraph_explore` response spent, what +share went to files the agent's answer actually drew on. + +``` +allocation efficiency = bytes returned for files the final answer cited + ─────────────────────────────────────────────── + all bytes returned +``` + +That is issue #1500 as a number. The envelope view +([`explore-allocation-ab-1500.md`](explore-allocation-ab-1500.md)) already +showed *how* a response was divided across files, but deciding which of those +files mattered took a human writing `--answer 'lib/response.js'` per question. +This reads the same intersection off the agent's own final answer, so every run +reports it for free. + +**Read it as a relative metric.** Attribution is by citation, and an agent can +use a file's source without ever naming it — to rule the file out, or to build a +mental model it then writes up from somewhere else. So this compares two builds +on the *same* question; it is not an absolute claim that 15% of an envelope was +waste. [What it does not say](#what-it-does-not-say) is the full list. + +Harness-only, like the other two feedback metrics: nothing is emitted from the +product, nothing leaves the machine. It is parsed out of transcripts we already +write. + +--- + +## Running it + +Every run prints it — `run-all.sh` and anything else that calls `parse-run.mjs`: + +```bash +node scripts/agent-eval/parse-run.mjs /private/tmp/cg22/ab-express/run-baseline-1.jsonl +``` + +``` + Explore allocation — share of returned bytes the answer used (2 calls, 4 files): + efficiency 81.9% 17,465 of 21,319 chars (3/4 files cited; path-cited alone 64.9%) + call 1: 84.0% 10,121/12,048 chars 2/3 files + call 2: 79.2% 7,344/9,271 chars 2/3 files + * 34.9% 7447 lib/response.js path + * 30.0% 6396 lib/utils.js path + 18.1% 3854 lib/express.js — + * 17.0% 3622 lib/application.js symbol `compileETag` +``` + +Interactive runs get the same block from `parse-session.mjs `. + +**Per call as well as per run.** A run-level number hides the shape that matters +most — call 1 on target, call 3 pure noise. The per-call lines name which call +spent its budget on files nothing came back to. + +`--answer ` and `--envelope` still work and are unchanged: they are the +*hand-specified* ground truth, useful when you want to score against the files +you know answer the question rather than the ones the agent happened to name. + +--- + +## How a file is judged used + +Two citation channels, ranked so the weaker one stays separable — the summary +line always reports **path-cited alone** next to the combined number. + +**1. Path.** The answer names the file: `lib/response.js:126-220`, an absolute +path, or the bare `utils.js:225` agents drop into prose. Bare basenames are +accepted only when the extension is one the envelope actually shipped, which is +what keeps `res.send` and `mime.contentType` — the same token shape — from +reading as file citations. + +**2. Symbol.** The answer cites, inside a code span, a symbol that the file's +section header lists as **defined**. This catches answers written almost +entirely in symbol names. It is guarded three ways, all in the direction that +costs us efficiency rather than inflating it: + +- **Definitions only.** The header renders `name(kind)` for every node in the + shipped clusters, and that includes call sites (`mutateElement(calls)` on a + file that merely calls it). Crediting those marked excalidraw's + `dragElements.ts` used because the answer named `mutateElement`. +- **A definition beats an import alias.** `var compileETag = require('./utils')` + is a `variable` node, so a file that only re-binds a name loses to the file + that defines it, when both are in the envelope. +- **A name on 3+ returned files identifies none of them.** Without this, + `send` or `get` marks half an envelope used. + +Prose mentions do not count — only code spans. A backtick is the agent marking +the token as code, and that is the whole signal. + +**A file returned twice is charged twice**, because it occupied the window +twice. Same accounting as the envelope view. + +**The answer** is the `result` event's text (one per resumed segment of a +multi-turn session — all of them are answers, so all are pooled). An interactive +transcript has no `result` event, so it falls back to the last main-thread +assistant text; intermediate narration is deliberately excluded, or "let me look +at App.tsx" would count as a citation. + +--- + +## Baseline + +Swept over every A/B log on this machine, the same corpus CG-8 used: **103 +sessions** with at least one answered explore (74 single-question, 29 multi-turn +3-question sessions), **297 explore calls**, 815 file sections, 0 crashes. + +| Slice | n | pooled | median | p25 | p75 | min | path-cited only | +|---|---|---|---|---|---|---|---| +| all | 103 | 85.6% | 90.1% | 82.0% | 97.7% | 20.1% | 80.2% | +| single-question | 74 | 85.2% | 88.8% | 82.0% | 95.6% | 20.1% | 82.2% | +| multi-turn (3 turns) | 29 | 86.2% | 92.1% | 85.0% | 100.0% | 57.4% | 77.5% | + +Per call: median 94.7%, p25 79.2%. + +**The absolute level is high, and that is a property of the corpus, not good +news.** These are flow questions whose answers walk the reader through the +chain, naming most files on it. The metric is byte-weighted, so it is dominated +by whether the *largest* allocations landed on cited files — which is exactly +the #1500 question, and also why the typical run scores in the eighties. The +discrimination lives in the p25 and below, not around the median. **Never quote +the median as "codegraph wastes 10% of what it returns."** + +**The tail is where the signal is.** The lowest run in the corpus is 20.1%: +`cg8-val` — excalidraw's `canvasNonce` question, the documented data-flow +frontier. Three explores returned 11 files / 60,694 chars; the answer drew on +two of them (`Scene.ts`, `StaticCanvas.tsx`) and reconstructed the rest by +reading `Renderer.ts` and `App.tsx`, which the envelope never shipped. That +reproduces the 16–30% range CG-1 measured by hand on the self-query, and it +reproduces it without anyone marking an answer set. + +### New-build vs baseline-build — the intended use + +Medians over the existing CG-15/CG-21/CG-22 A/B arms, both arms codegraph-on: + +| Pass / repo | baseline | new | +|---|---|---| +| cg15 / express | 82.0% | **100.0%** | +| cg15 / excalidraw | 90.1% | 92.8% | +| cg15 / client-go | 84.1% | 80.8% | +| cg21 / express | 84.1% | **100.0%** | +| cg21 / excalidraw | 91.3% | 97.5% | +| cg21 / client-go | 67.1% | **95.0%** | +| cg22 / express | 81.9% | **100.0%** | +| cg22 / excalidraw | 89.3% | 79.2% | +| cg22 / client-go | 86.6% | 93.3% | + +Express is the #1500 case and it moves the most: the baseline arm reliably spent +18% of its envelope on `lib/express.js`, a file no answer ever cited, and the +allocation fix cut it — 81.9 / 82.0 / 81.9% across three baseline runs, 100.0 / +92.5 / 100.0% across three new ones. That is the change stated as a number +rather than as "the express arm looks tighter now." + +Two pairs go the other way (cg15/client-go −3pt, cg22/excalidraw −10pt). The +cg22/excalidraw spread is 89.3 / 85.6 / 90.8% baseline against 87.4 / 79.2 / +78.2% new — real, but n=3 on a question with known run-to-run variance. Report +the range, not the median of three. + +--- + +## What it does not say + +- **Uncited is not unused.** An agent reads a file, decides it is not the + answer, and never mentions it. That was useful work and this metric calls it + waste. The error is one-sided and it is why the number is only comparable + between builds on the same question. +- **Cited is not "needed at that size."** The metric asks which *files* earned + their bytes, not whether the right *lines within* a cited file were shipped. + A file cited by the answer scores 100% of its section even if the agent then + had to Read it for the part we clipped out. The sufficiency bucket + `Read a file we returned` is what catches that; the two metrics are + complementary, and CG-13 reports both. +- **A basename match can land on the wrong file.** `src/index.ts` in the + envelope matches an answer citing `packages/element/src/index.ts`. Rare in + practice, but it is a real false-positive channel. +- **Small-n.** Runs make 1–5 explore calls, so one run's percentage is coarse. + Compare over a pass (`RUNS>=2`, and the CG-13 7-repo campaign), never n=1. +- **Efficiency is not value.** A response can be 100% efficient and useless — a + single small file the answer names in passing. Read it alongside sufficiency + ([`explore-sufficiency.md`](explore-sufficiency.md)) and occupancy + ([`residual-context-occupancy.md`](residual-context-occupancy.md)), which is + the point of wiring all three into the same run. + +--- + +## Tests + +`node scripts/agent-eval/parse-run.mjs --selftest` covers the metric over +synthetic transcripts with known answers, alongside the occupancy and +sufficiency checks (68/68). The allocation cases: the one-of-three-files shape, +bare-basename citations and the dotted-expression tokens that must *not* match, +symbol attribution crediting the definer and not a caller, a definition beating +an import alias, the 3-file ambiguity cutoff, prose-vs-code-span, per-call +versus pooled accounting, a file returned twice being charged twice, both +final-answer sources, and the end-to-end path through `parseSession`.