test(agent-eval): self-test the occupancy math, and fix ratio calibration under shedding (CG-7)
parse-run.mjs --selftest runs the math over synthetic transcripts with known answers: attribution, message.id dedupe, compact_boundary, FIFO micro-compaction, and multi-turn stitching. It found a real bug. A gap where the window also SHED content has a delta far below what was added, which reads as absurdly dense text and dragged the whole run's ratio with it -- a shed gap in the fixture pushed 2.5 chars/tok to 4.4 and left the wrong result resident. Shedding can only push a gap's ratio up, so the calibration now takes the lower median as its centre, drops gaps well above it, and pools the rest. Runs that never shed are unaffected (gin and vscode re-measure identically). Also drafts docs/benchmarks/residual-context-occupancy.md -- method, error bar, and the limitations this metric does not settle. Baseline numbers to follow.
This commit is contained in:
Colby McHenry
@@ -0,0 +1,164 @@
|
||||
# Residual context occupancy
|
||||
|
||||
**What it measures:** how many tokens of the context window a tool's responses
|
||||
still occupy once the question has been answered — and therefore how much
|
||||
headroom every following turn has to work in.
|
||||
|
||||
This is the metric issue [#1500](https://github.com/colbymchenry/codegraph/issues/1500)
|
||||
was actually about. The reporter was looking at a live Cursor session: explore's
|
||||
output was still resident after the answer, so it was charged against everything
|
||||
that came next. Our A/B harness ran one headless question to completion and
|
||||
reported cost, tokens, time, and tool calls — none of which can see that. A
|
||||
single-question run reports *throughput*; occupancy is a *stock*, and it only
|
||||
starts costing anything on the turns that follow.
|
||||
|
||||
The harness now measures it, over multi-turn sessions.
|
||||
|
||||
---
|
||||
|
||||
## Running it
|
||||
|
||||
```bash
|
||||
# One repo, one three-turn session, both arms:
|
||||
scripts/agent-eval/run-all.sh /tmp/codegraph-corpus/gin \
|
||||
"How does gin route requests through its middleware chain?||\
|
||||
Where is the 404 / no-route case handled in that same chain?||\
|
||||
What would I change to add a per-route middleware that runs before the global ones?"
|
||||
|
||||
# The 7 README repos (default: 3 turns per session, RUNS=4 per arm):
|
||||
CORPUS=/tmp/codegraph-corpus RUNS=2 scripts/agent-eval/bench-readme.sh
|
||||
node scripts/agent-eval/parse-bench-readme.mjs /tmp/ab-readme
|
||||
```
|
||||
|
||||
`||` separates turns. Turn 1 runs normally; each later turn `--resume`s the same
|
||||
session, so the earlier turns' tool output is still in the window — which is the
|
||||
entire point. Segments land in `run-<label>.jsonl`, `run-<label>.t2.jsonl`, …
|
||||
and `parse-run.mjs` stitches them back into one session (`--resume` does not
|
||||
replay prior messages, so they concatenate cleanly).
|
||||
|
||||
`CG_TURNS=1` restores the original single-question A/B. `CG_WINDOW_TOKENS`
|
||||
overrides the 200k nominal window for the share-of-window column.
|
||||
|
||||
Every arm prints:
|
||||
|
||||
```
|
||||
Residual context occupancy at end of run:
|
||||
final context 54,950 tok 27.5% of 200k window
|
||||
codegraph 13,941 tok 25.4% of ctx 7.0% of 200k win (31,312 chars, 2 results)
|
||||
Read 0 tok 0.0% of ctx 0.0% of 200k win (0 chars, 0 results)
|
||||
Grep/Glob 0 tok 0.0% of ctx 0.0% of 200k win (0 chars, 0 results)
|
||||
Bash 0 tok 0.0% of ctx 0.0% of 200k win (0 chars, 0 results)
|
||||
→ file-access 0 tok 0.0% of ctx 0.0% of 200k win (0 chars, 0 results)
|
||||
other tools 33 tok 0.1% of ctx 0.0% of 200k win (73 chars, 1 result)
|
||||
base (prompt+prose) 40,976 tok 74.6% of ctx 20.5% of 200k win
|
||||
of which fixed 37,726 tok system + tool schemas + question, before any tool answered
|
||||
measure: 2.25 chars/tok measured ±0.9% · turns 6 · compactions 0
|
||||
```
|
||||
|
||||
The comparison is codegraph's residual in the with-arm against **file-access**
|
||||
(Read + Grep/Glob + Bash) in the without-arm — the two ways an agent gets the
|
||||
same bytes into its head. Bash matters: on small repos the without-arm often
|
||||
reaches for `cat`/`grep` through Bash rather than the Read tool, and counting
|
||||
only Read would score those runs as reading nothing.
|
||||
|
||||
---
|
||||
|
||||
## How the tokens are measured
|
||||
|
||||
**Measured, not estimated.** For each assistant request,
|
||||
|
||||
```
|
||||
ctx_k = usage.input_tokens + cache_read_input_tokens + cache_creation_input_tokens
|
||||
```
|
||||
|
||||
is the exact token count of that request's entire prompt. So `ctx_k − ctx_{k−1}`
|
||||
is exactly what was appended since the previous request: the previous assistant
|
||||
output plus the tool results and user text that followed it. Each gap's measured
|
||||
delta is priced against the characters in it.
|
||||
|
||||
The ratio is calibrated on gaps that are **≥80% tool result by characters**, then
|
||||
every result is priced at that ratio. Calibrating on *all* gaps was wrong: when
|
||||
the assistant's own output is under-represented in the transcript — redacted or
|
||||
empty thinking blocks are the common case — a proportional split hands the tool
|
||||
result the whole delta. One 73-character `ToolSearch` result was charged the
|
||||
entire 830-token gap, 5.5 tokens per character.
|
||||
|
||||
Getting this right matters more than it sounds. Explore output measures around
|
||||
**2.2–2.3 chars/token** — it is dense, line-numbered source. The usual bytes/4
|
||||
rule of thumb would under-count it by roughly 40%.
|
||||
|
||||
**Error bar.** On a gap that is ≥95% one tool result, the measured delta *is*
|
||||
that result's token count, so the distance from the run-level ratio is the
|
||||
attribution error for that result. The median over such gaps is printed after
|
||||
the ratio: **±1–2%** on real runs.
|
||||
|
||||
### Residual is not the same as contributed
|
||||
|
||||
Content leaves the window two ways, and both are tracked:
|
||||
|
||||
- a `compact_boundary` system event — everything before it is replaced by a
|
||||
summary, so the resident set is cleared;
|
||||
- **micro-compaction** — the context drops mid-run without a boundary event.
|
||||
Claude Code sheds the oldest tool results first, so eviction is applied FIFO.
|
||||
|
||||
A shortfall only counts as eviction past a tolerance (the larger of 200 tokens or
|
||||
5%); below that it is attribution noise, and real shedding is thousands of tokens.
|
||||
|
||||
### Two transcript traps
|
||||
|
||||
Both were verified against real logs and are worth knowing before writing
|
||||
anything else that reads these files:
|
||||
|
||||
1. **Claude Code emits one `assistant` event per content block**, all carrying
|
||||
the same `message.id` *and the same `usage`*. Summing usage per event
|
||||
double-counts every turn that emits both a thinking block and a tool_use.
|
||||
`parseSession()` dedupes by `message.id`.
|
||||
2. **The streamed `output_tokens` is a partial snapshot** — observed as `out=2`
|
||||
on a turn that really generated ~1,100 tokens. It is unusable; the
|
||||
char-proportional method deliberately does not need it.
|
||||
|
||||
For the record, `result.usage` in Claude Code 2.1.198 is cumulative *within a
|
||||
segment* (its in+cache+out equals the sum of that segment's per-request prompts),
|
||||
not last-turn-only as it was when `CLAUDE.md` was written. `parseSession()` sums
|
||||
per segment either way. That figure is "tokens processed" — every request
|
||||
re-counts the whole prefix — which is exactly why it cannot answer the occupancy
|
||||
question.
|
||||
|
||||
---
|
||||
|
||||
## Baseline: the 7 README repos
|
||||
|
||||
<!-- RESULTS -->
|
||||
|
||||
---
|
||||
|
||||
## What this settles, and what it does not
|
||||
|
||||
**Settled.** The metric exists, it is measured rather than estimated, it runs over
|
||||
multi-turn sessions — the regime where occupancy is actually charged — and there
|
||||
is a baseline across the 7 README repos to compare future changes against.
|
||||
|
||||
**Not settled, and deliberately not claimed:**
|
||||
|
||||
- **A different host.** The reporter was in Cursor. We measure Claude Code.
|
||||
Window size, system prompt, and compaction policy all differ, so the *share*
|
||||
numbers do not transfer host to host; the ratio between the arms is the part
|
||||
that travels.
|
||||
- **Three turns is short.** It is long enough for the residual to be charged
|
||||
against something, which single-turn runs could not do at all. It is not long
|
||||
enough to reach compaction on a 200k window, so the compaction and
|
||||
micro-compaction paths are implemented and instrumented but effectively
|
||||
untested by this baseline — no run here triggered either.
|
||||
- **Deferred tool schemas land in `base`.** `codegraph_explore` is a deferred
|
||||
tool: the initial listing carries its name, and `ToolSearch` pulls the full
|
||||
schema in later. That injection is not a tool result, so its tokens are
|
||||
counted as base rather than attributed to codegraph. The fixed-overhead line
|
||||
(with-arm `ctxBase` minus without-arm `ctxBase`) prices the part that is
|
||||
present from the start.
|
||||
- **Subagent contexts are not counted.** A `Task` subagent has its own window;
|
||||
only its summary returns to the parent. Runs that delegate are measured on the
|
||||
parent's window alone.
|
||||
- **Occupancy is not sufficiency.** A small residual is only good if the answer
|
||||
was still right. This metric says nothing about answer quality — that is
|
||||
CG-8's job (sufficiency) and CG-9's (how much of the returned bytes the answer
|
||||
actually used).
|
||||
@@ -177,10 +177,19 @@ export function parseSession(files) {
|
||||
}
|
||||
gaps.push({ delta: cur.ctx - prev.ctx, chars, toolChars, byFamily, compacted });
|
||||
}
|
||||
const clean = gaps.filter((g) => !g.compacted && g.delta > 0 && g.chars > 500 && g.toolChars / g.chars >= 0.8);
|
||||
// A gap where the window also SHED content has a delta far below what was
|
||||
// added, which reads as absurdly dense text and would drag the whole run's
|
||||
// ratio with it. Shedding can only push a gap's chars/token UP, so take the
|
||||
// lower median as the honest centre and drop anything well above it, then
|
||||
// pool the survivors. (On runs that never shed, every ratio is within a few
|
||||
// percent of the others and this changes nothing.)
|
||||
const ratios = clean.map((g) => g.toolChars / g.delta).sort((a, b) => a - b);
|
||||
const lowerMedian = ratios.length ? ratios[Math.floor((ratios.length - 1) / 2)] : 0;
|
||||
let sumD = 0, sumC = 0;
|
||||
for (const g of gaps) {
|
||||
if (g.compacted || g.delta <= 0) continue;
|
||||
if (g.chars > 500 && g.toolChars / g.chars >= 0.8) { sumD += g.delta; sumC += g.toolChars; }
|
||||
for (const g of clean) {
|
||||
if (lowerMedian > 0 && g.toolChars / g.delta > lowerMedian * 1.5) continue; // shed
|
||||
sumD += g.delta; sumC += g.toolChars;
|
||||
}
|
||||
if (sumD === 0) { // no clean gap — fall back to every growing gap, all chars
|
||||
for (const g of gaps) if (!g.compacted && g.delta > 0 && g.chars > 0) { sumD += g.delta; sumC += g.chars; }
|
||||
@@ -319,10 +328,124 @@ export function formatOccupancy(s, indent = ' ') {
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// `--selftest`: the occupancy math over synthetic transcripts with known
|
||||
// answers. It lives here rather than in a test file on purpose — a new
|
||||
// scripts/agent-eval/*.mjs scores into the self-query eval fixture's corpus.
|
||||
function selftest() {
|
||||
const { writeFileSync, mkdtempSync } = require0('fs');
|
||||
const { join } = require0('path');
|
||||
const { tmpdir } = require0('os');
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cg-occ-'));
|
||||
let n = 0, failures = 0;
|
||||
const check = (name, got, want, tol) => {
|
||||
n++;
|
||||
const ok = Math.abs(got - want) <= tol;
|
||||
if (!ok) failures++;
|
||||
console.log(`${ok ? ' ok ' : ' FAIL'} ${name}: got ${Math.round(got)}, want ${want} ±${tol}`);
|
||||
};
|
||||
// Builders for the event shapes Claude Code actually emits.
|
||||
const req = (ctx, id, blocks) => blocks.map((b) => JSON.stringify({
|
||||
type: 'assistant',
|
||||
message: { id, content: [b], usage: { input_tokens: ctx, cache_read_input_tokens: 0, cache_creation_input_tokens: 0, output_tokens: 2 } },
|
||||
}));
|
||||
const use = (id, name) => ({ type: 'tool_use', id, name, input: {} });
|
||||
const res = (id, chars) => JSON.stringify({
|
||||
type: 'user', message: { content: [{ type: 'tool_result', tool_use_id: id, content: [{ type: 'text', text: 'x'.repeat(chars) }] }] },
|
||||
});
|
||||
const done = () => JSON.stringify({ type: 'result', subtype: 'success', duration_ms: 1000, total_cost_usd: 0.1, usage: {} });
|
||||
const write = (name, lines) => { const f = join(dir, name); writeFileSync(f, lines.join('\n') + '\n'); return f; };
|
||||
|
||||
// 1. Attribution: ratio 2.5 chars/tok, two families, no shedding.
|
||||
// 10,000 explore chars over a 4,000-tok gap; 5,000 Read chars over 2,000.
|
||||
let f = write('basic.jsonl', [
|
||||
...req(10000, 'm1', [use('t1', 'mcp__codegraph__codegraph_explore')]),
|
||||
res('t1', 10000),
|
||||
...req(14000, 'm2', [use('t2', 'Read')]),
|
||||
res('t2', 5000),
|
||||
...req(16000, 'm3', [{ type: 'text', text: 'done' }]),
|
||||
done(),
|
||||
]);
|
||||
let o = parseSession([f]).occupancy;
|
||||
check('chars/token', o.charsPerToken * 1000, 2500, 30);
|
||||
check('codegraph residual', o.residual.codegraph, 4000, 60);
|
||||
check('Read residual', o.residual.read, 2000, 40);
|
||||
check('file-access residual', o.residualFileAccess, 2000, 40);
|
||||
check('final context', o.ctxFinal, 16000, 0);
|
||||
check('fixed base', o.ctxBase, 10000, 0);
|
||||
check('nothing evicted', o.evicted, 0, 1);
|
||||
|
||||
// 2. Dedupe: thinking + tool_use are two events sharing one id and one usage.
|
||||
// Counting usage per event would report 5 requests instead of 3.
|
||||
f = write('dupe.jsonl', [
|
||||
...req(10000, 'm1', [{ type: 'thinking', thinking: '' }, use('t1', 'mcp__codegraph__codegraph_explore')]),
|
||||
res('t1', 10000),
|
||||
...req(14000, 'm2', [{ type: 'thinking', thinking: '' }, use('t2', 'Read')]),
|
||||
res('t2', 5000),
|
||||
...req(16000, 'm3', [{ type: 'text', text: 'done' }]),
|
||||
done(),
|
||||
]);
|
||||
let s = parseSession([f]);
|
||||
check('turns deduped by message.id', s.turns, 3, 0);
|
||||
check('codegraph residual (deduped)', s.occupancy.residual.codegraph, 4000, 60);
|
||||
|
||||
// 3. Compaction: the boundary clears everything resident before it.
|
||||
f = write('compact.jsonl', [
|
||||
...req(10000, 'm1', [use('t1', 'mcp__codegraph__codegraph_explore')]),
|
||||
res('t1', 10000),
|
||||
...req(14000, 'm2', [use('t2', 'mcp__codegraph__codegraph_explore')]),
|
||||
JSON.stringify({ type: 'system', subtype: 'compact_boundary' }),
|
||||
res('t2', 5000),
|
||||
...req(8000, 'm3', [{ type: 'text', text: 'done' }]),
|
||||
done(),
|
||||
]);
|
||||
o = parseSession([f]).occupancy;
|
||||
check('post-compaction residual = last result only', o.residual.codegraph, 2000, 40);
|
||||
check('contributed still counts both', o.contributed.codegraph, 6000, 80);
|
||||
|
||||
// 4. Micro-compaction: context grows less than the results added, so the
|
||||
// oldest result is shed first (FIFO) — here explore, leaving Read.
|
||||
f = write('micro.jsonl', [
|
||||
...req(10000, 'm1', [use('t1', 'mcp__codegraph__codegraph_explore')]),
|
||||
res('t1', 10000),
|
||||
...req(14000, 'm2', [use('t2', 'Read')]),
|
||||
res('t2', 10000),
|
||||
...req(14500, 'm3', [{ type: 'text', text: 'done' }]), // +500 for 4,000 tok of Read
|
||||
done(),
|
||||
]);
|
||||
o = parseSession([f]).occupancy;
|
||||
check('FIFO evicted the older codegraph result', o.residual.codegraph, 500, 60);
|
||||
check('newer Read result survives', o.residual.read, 4000, 60);
|
||||
check('eviction recorded', o.evicted, 3500, 60);
|
||||
|
||||
// 5. Multi-turn stitching: a resumed segment continues the same context, and
|
||||
// a turn that calls no tool leaves the earlier residual in place.
|
||||
const a = write('seg1.jsonl', [
|
||||
...req(10000, 'm1', [use('t1', 'mcp__codegraph__codegraph_explore')]),
|
||||
res('t1', 10000),
|
||||
...req(14000, 'm2', [{ type: 'text', text: 'answer one' }]),
|
||||
done(),
|
||||
]);
|
||||
const b = write('seg2.jsonl', [
|
||||
...req(14600, 'm3', [{ type: 'text', text: 'answer two, from what is already here' }]),
|
||||
done(),
|
||||
]);
|
||||
s = parseSession([a, b]);
|
||||
check('stitched turns', s.turns, 3, 0);
|
||||
check('residual carries into turn 2', s.occupancy.residual.codegraph, 4000, 60);
|
||||
check('stitched final context', s.occupancy.ctxFinal, 14600, 0);
|
||||
check('stitched cost sums segments', s.cost * 100, 20, 0.1);
|
||||
|
||||
console.log(`\n${n - failures}/${n} checks passed`);
|
||||
return failures;
|
||||
}
|
||||
// `--selftest` needs sync fs helpers the module path doesn't import at top level.
|
||||
function require0(m) { return process.getBuiltinModule(m); }
|
||||
|
||||
const isMain = process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href;
|
||||
if (isMain && process.argv.includes('--selftest')) process.exit(selftest() ? 1 : 0);
|
||||
if (isMain) {
|
||||
const files = process.argv.slice(2).filter((a) => !a.startsWith('--'));
|
||||
if (!files.length) { console.error('usage: parse-run.mjs <run.jsonl> [run.t2.jsonl ...]'); process.exit(1); }
|
||||
if (!files.length) { console.error('usage: parse-run.mjs <run.jsonl> [run.t2.jsonl ...] | --selftest'); process.exit(1); }
|
||||
const s = parseSession(files);
|
||||
|
||||
console.log(`\n=== ${files.map((f) => f.split('/').pop()).join(' + ')} ===`);
|
||||
|
||||
Reference in New Issue
Block a user