Colby McHenry
fa15d1046a
Records the sweep over every A/B log on this machine (103 sessions, 297 explore calls, 0 crashes) and, more usefully, the new-vs-baseline arm table the metric exists for: express 82% → 100%, cg21/client-go 67% → 95%, two pairs going the other way. States the caveat in the places it can be misread: the corpus median sits in the eighties because these are flow questions whose answers name most of the chain, the metric is byte-weighted, and an agent can use a file without citing it. It compares two builds on one question; it is not an absolute waste figure.
196 lines
9.3 KiB
Markdown
196 lines
9.3 KiB
Markdown
# Explore allocation efficiency
|
||
|
||
**What it measures:** of the bytes a `codegraph_explore` response spent, what
|
||
share went to files the agent's answer actually drew on.
|
||
|
||
```
|
||
allocation efficiency = bytes returned for files the final answer cited
|
||
───────────────────────────────────────────────
|
||
all bytes returned
|
||
```
|
||
|
||
That is issue #1500 as a number. The envelope view
|
||
([`explore-allocation-ab-1500.md`](explore-allocation-ab-1500.md)) already
|
||
showed *how* a response was divided across files, but deciding which of those
|
||
files mattered took a human writing `--answer 'lib/response.js'` per question.
|
||
This reads the same intersection off the agent's own final answer, so every run
|
||
reports it for free.
|
||
|
||
**Read it as a relative metric.** Attribution is by citation, and an agent can
|
||
use a file's source without ever naming it — to rule the file out, or to build a
|
||
mental model it then writes up from somewhere else. So this compares two builds
|
||
on the *same* question; it is not an absolute claim that 15% of an envelope was
|
||
waste. [What it does not say](#what-it-does-not-say) is the full list.
|
||
|
||
Harness-only, like the other two feedback metrics: nothing is emitted from the
|
||
product, nothing leaves the machine. It is parsed out of transcripts we already
|
||
write.
|
||
|
||
---
|
||
|
||
## Running it
|
||
|
||
Every run prints it — `run-all.sh` and anything else that calls `parse-run.mjs`:
|
||
|
||
```bash
|
||
node scripts/agent-eval/parse-run.mjs /private/tmp/cg22/ab-express/run-baseline-1.jsonl
|
||
```
|
||
|
||
```
|
||
Explore allocation — share of returned bytes the answer used (2 calls, 4 files):
|
||
efficiency 81.9% 17,465 of 21,319 chars (3/4 files cited; path-cited alone 64.9%)
|
||
call 1: 84.0% 10,121/12,048 chars 2/3 files
|
||
call 2: 79.2% 7,344/9,271 chars 2/3 files
|
||
* 34.9% 7447 lib/response.js path
|
||
* 30.0% 6396 lib/utils.js path
|
||
18.1% 3854 lib/express.js —
|
||
* 17.0% 3622 lib/application.js symbol `compileETag`
|
||
```
|
||
|
||
Interactive runs get the same block from `parse-session.mjs <project-dir>`.
|
||
|
||
**Per call as well as per run.** A run-level number hides the shape that matters
|
||
most — call 1 on target, call 3 pure noise. The per-call lines name which call
|
||
spent its budget on files nothing came back to.
|
||
|
||
`--answer <glob>` and `--envelope` still work and are unchanged: they are the
|
||
*hand-specified* ground truth, useful when you want to score against the files
|
||
you know answer the question rather than the ones the agent happened to name.
|
||
|
||
---
|
||
|
||
## How a file is judged used
|
||
|
||
Two citation channels, ranked so the weaker one stays separable — the summary
|
||
line always reports **path-cited alone** next to the combined number.
|
||
|
||
**1. Path.** The answer names the file: `lib/response.js:126-220`, an absolute
|
||
path, or the bare `utils.js:225` agents drop into prose. Bare basenames are
|
||
accepted only when the extension is one the envelope actually shipped, which is
|
||
what keeps `res.send` and `mime.contentType` — the same token shape — from
|
||
reading as file citations.
|
||
|
||
**2. Symbol.** The answer cites, inside a code span, a symbol that the file's
|
||
section header lists as **defined**. This catches answers written almost
|
||
entirely in symbol names. It is guarded three ways, all in the direction that
|
||
costs us efficiency rather than inflating it:
|
||
|
||
- **Definitions only.** The header renders `name(kind)` for every node in the
|
||
shipped clusters, and that includes call sites (`mutateElement(calls)` on a
|
||
file that merely calls it). Crediting those marked excalidraw's
|
||
`dragElements.ts` used because the answer named `mutateElement`.
|
||
- **A definition beats an import alias.** `var compileETag = require('./utils')`
|
||
is a `variable` node, so a file that only re-binds a name loses to the file
|
||
that defines it, when both are in the envelope.
|
||
- **A name on 3+ returned files identifies none of them.** Without this,
|
||
`send` or `get` marks half an envelope used.
|
||
|
||
Prose mentions do not count — only code spans. A backtick is the agent marking
|
||
the token as code, and that is the whole signal.
|
||
|
||
**A file returned twice is charged twice**, because it occupied the window
|
||
twice. Same accounting as the envelope view.
|
||
|
||
**The answer** is the `result` event's text (one per resumed segment of a
|
||
multi-turn session — all of them are answers, so all are pooled). An interactive
|
||
transcript has no `result` event, so it falls back to the last main-thread
|
||
assistant text; intermediate narration is deliberately excluded, or "let me look
|
||
at App.tsx" would count as a citation.
|
||
|
||
---
|
||
|
||
## Baseline
|
||
|
||
Swept over every A/B log on this machine, the same corpus CG-8 used: **103
|
||
sessions** with at least one answered explore (74 single-question, 29 multi-turn
|
||
3-question sessions), **297 explore calls**, 815 file sections, 0 crashes.
|
||
|
||
| Slice | n | pooled | median | p25 | p75 | min | path-cited only |
|
||
|---|---|---|---|---|---|---|---|
|
||
| all | 103 | 85.6% | 90.1% | 82.0% | 97.7% | 20.1% | 80.2% |
|
||
| single-question | 74 | 85.2% | 88.8% | 82.0% | 95.6% | 20.1% | 82.2% |
|
||
| multi-turn (3 turns) | 29 | 86.2% | 92.1% | 85.0% | 100.0% | 57.4% | 77.5% |
|
||
|
||
Per call: median 94.7%, p25 79.2%.
|
||
|
||
**The absolute level is high, and that is a property of the corpus, not good
|
||
news.** These are flow questions whose answers walk the reader through the
|
||
chain, naming most files on it. The metric is byte-weighted, so it is dominated
|
||
by whether the *largest* allocations landed on cited files — which is exactly
|
||
the #1500 question, and also why the typical run scores in the eighties. The
|
||
discrimination lives in the p25 and below, not around the median. **Never quote
|
||
the median as "codegraph wastes 10% of what it returns."**
|
||
|
||
**The tail is where the signal is.** The lowest run in the corpus is 20.1%:
|
||
`cg8-val` — excalidraw's `canvasNonce` question, the documented data-flow
|
||
frontier. Three explores returned 11 files / 60,694 chars; the answer drew on
|
||
two of them (`Scene.ts`, `StaticCanvas.tsx`) and reconstructed the rest by
|
||
reading `Renderer.ts` and `App.tsx`, which the envelope never shipped. That
|
||
reproduces the 16–30% range CG-1 measured by hand on the self-query, and it
|
||
reproduces it without anyone marking an answer set.
|
||
|
||
### New-build vs baseline-build — the intended use
|
||
|
||
Medians over the existing CG-15/CG-21/CG-22 A/B arms, both arms codegraph-on:
|
||
|
||
| Pass / repo | baseline | new |
|
||
|---|---|---|
|
||
| cg15 / express | 82.0% | **100.0%** |
|
||
| cg15 / excalidraw | 90.1% | 92.8% |
|
||
| cg15 / client-go | 84.1% | 80.8% |
|
||
| cg21 / express | 84.1% | **100.0%** |
|
||
| cg21 / excalidraw | 91.3% | 97.5% |
|
||
| cg21 / client-go | 67.1% | **95.0%** |
|
||
| cg22 / express | 81.9% | **100.0%** |
|
||
| cg22 / excalidraw | 89.3% | 79.2% |
|
||
| cg22 / client-go | 86.6% | 93.3% |
|
||
|
||
Express is the #1500 case and it moves the most: the baseline arm reliably spent
|
||
18% of its envelope on `lib/express.js`, a file no answer ever cited, and the
|
||
allocation fix cut it — 81.9 / 82.0 / 81.9% across three baseline runs, 100.0 /
|
||
92.5 / 100.0% across three new ones. That is the change stated as a number
|
||
rather than as "the express arm looks tighter now."
|
||
|
||
Two pairs go the other way (cg15/client-go −3pt, cg22/excalidraw −10pt). The
|
||
cg22/excalidraw spread is 89.3 / 85.6 / 90.8% baseline against 87.4 / 79.2 /
|
||
78.2% new — real, but n=3 on a question with known run-to-run variance. Report
|
||
the range, not the median of three.
|
||
|
||
---
|
||
|
||
## What it does not say
|
||
|
||
- **Uncited is not unused.** An agent reads a file, decides it is not the
|
||
answer, and never mentions it. That was useful work and this metric calls it
|
||
waste. The error is one-sided and it is why the number is only comparable
|
||
between builds on the same question.
|
||
- **Cited is not "needed at that size."** The metric asks which *files* earned
|
||
their bytes, not whether the right *lines within* a cited file were shipped.
|
||
A file cited by the answer scores 100% of its section even if the agent then
|
||
had to Read it for the part we clipped out. The sufficiency bucket
|
||
`Read a file we returned` is what catches that; the two metrics are
|
||
complementary, and CG-13 reports both.
|
||
- **A basename match can land on the wrong file.** `src/index.ts` in the
|
||
envelope matches an answer citing `packages/element/src/index.ts`. Rare in
|
||
practice, but it is a real false-positive channel.
|
||
- **Small-n.** Runs make 1–5 explore calls, so one run's percentage is coarse.
|
||
Compare over a pass (`RUNS>=2`, and the CG-13 7-repo campaign), never n=1.
|
||
- **Efficiency is not value.** A response can be 100% efficient and useless — a
|
||
single small file the answer names in passing. Read it alongside sufficiency
|
||
([`explore-sufficiency.md`](explore-sufficiency.md)) and occupancy
|
||
([`residual-context-occupancy.md`](residual-context-occupancy.md)), which is
|
||
the point of wiring all three into the same run.
|
||
|
||
---
|
||
|
||
## Tests
|
||
|
||
`node scripts/agent-eval/parse-run.mjs --selftest` covers the metric over
|
||
synthetic transcripts with known answers, alongside the occupancy and
|
||
sufficiency checks (68/68). The allocation cases: the one-of-three-files shape,
|
||
bare-basename citations and the dotted-expression tokens that must *not* match,
|
||
symbol attribution crediting the definer and not a caller, a definition beating
|
||
an import alias, the 3-file ambiguity cutoff, prose-vs-code-span, per-call
|
||
versus pooled accounting, a file returned twice being charged twice, both
|
||
final-answer sources, and the end-to-end path through `parseSession`.
|