test(agent-eval): report what share of explore's bytes the answer used (CG-9)

The envelope view needed a human to say which files answer the question
(`--answer <glob>`). This reads it off the agent's own final answer and
reports one number per run and per call: bytes returned for files the
answer cited, over all bytes returned. That is the #1500 defect as a
number instead of a hunch.

Attribution has two channels, ranked so the weaker one stays separable:
the answer naming the file (reported alone as the conservative floor),
and the answer citing, in a code span, a symbol only that file DEFINES.
Three guards keep the error from leaning optimistic — the direction a
tuning metric must not lean:

  * Only symbols the file defines. Section headers render `name(kind)`
    for call sites too, and crediting those marked excalidraw's
    dragElements.ts used because the answer named `mutateElement`.
  * A definition beats an import alias of the same name (`variable`),
    or `lib/application.js` gets credit for `require('./utils')`.
  * A name on 3+ returned files identifies none of them.

Bare basenames count as citations (agents write `utils.js:225` in prose)
but only for extensions the envelope shipped, so `res.send` and
`mime.contentType` — the same token shape — do not read as files.

Both the envelope view and this share one parse of the rendered markdown
(parseExploreCall), still not the CG-4 sidecar: the sidecar exists only
on a post-CG-4 build and so cannot measure a baseline arm.
This commit is contained in:
Colby McHenry
2026-08-05 00:42:59 -05:00
parent 254e573f11
commit db3b8d2a1e
2 changed files with 414 additions and 21 deletions
+11 -1
View File
@@ -6,7 +6,10 @@
import { readFileSync, readdirSync, statSync, existsSync, realpathSync } from 'fs';
import { join } from 'path';
import { homedir } from 'os';
import { classifySufficiency, formatSufficiency } from './parse-run.mjs';
import {
classifySufficiency, formatSufficiency,
collectExploreTexts, computeAllocation, finalAnswerText, formatAllocation,
} from './parse-run.mjs';
const projectArg = process.argv[2];
if (!projectArg) { console.error('usage: parse-session.mjs <project-dir>'); process.exit(1); }
@@ -118,3 +121,10 @@ if (existsSync(subDir)) {
}
console.log('');
console.log(formatSufficiency({ sufficiency: classifySufficiency(events) }, ''));
// How much of what explore returned the answer drew on (CG-9). An interactive
// transcript has no `result` event, so the answer is the last main-thread
// assistant text — finalAnswerText already falls back to it.
console.log('');
console.log(formatAllocation(
{ allocation: computeAllocation(collectExploreTexts(events), finalAnswerText(events)) }, ''));