Carry-forward slack let a file spend what the files ABOVE it left on the
table. Nothing held back what was promised BELOW it. The whole-file BUY arm
has always refused that trade (`owedBelow`); the cluster path read `headroom`
— what is left before the hard ceiling — instead of what is still owed, so
`fileBudget` and `SPINE_CEILING` could pay a 1.5x overshoot out of another
file's reservation.
`fundedHeadroom` is the same inequality in the units the cluster path spends
in: source PLUS the per-section overhead each unreached file will charge.
Floored at the file's own reservation — a kept promise is not a displacement —
and it is <= `headroom` by construction, so it is the only bound the three
render sites need. The skeleton path's `bodyCap` takes it too.
Measured on `__tests__/fixtures/displacement-ts` (a 4-stage pipeline padded
past 500 files, where the 24K envelope genuinely saturates the 24.4K render
ceiling):
before ingest.ts emitted 9,301 on a 6,289 spendable, then lost the whole
section to the final ceiling — 0 delivered. types.ts and sink.ts
skipped `budget-whole-file`. 3 of 6 admitted files delivered.
after ingest.ts bounded to the 4,913 actually free. 6 of 6 delivered,
envelope 14,908 -> 22,066.
The self-query allocation fixture flips back to PASS with it, on a clean full
rebuild of this repo's index (CG-33). Its `afterCG30` verdict blamed an
over-RESERVED incidental file; the reservation was identical in both arms —
the file was over-SPENDING. Recorded honestly in `afterCG31`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Primary evidence is deterministic: on django, query.py rendered 2.12x its budget
on main and 1.49x with the bound, and the freed bytes reach the files below it
(+2,104 chars of source in the same five files). gin is a true control — the two
builds emit byte-identical explore output there, which is what makes its agent-run
deltas variance by construction.
Also records the harness trap that voided the first two batches: ab-new-vs-baseline
swaps src/ to the baseline ref mid-run, so a commit made while it runs captures
baseline sources. Check the "changed:" line before believing any run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d652c14 was committed while an ab-new-vs-baseline run had the engine checked out
at the BASELINE ref — that harness swaps src/ files mid-run and restores them on
exit — so it captured main's tools.ts and explore-diagnostics.ts and silently
undid 765c06a. Restored from 765c06a, with d652c14's doc tweak re-applied.
The A/B runs started after that commit are void with it (their "changed:" line
lists only explore-diagnostics.ts, i.e. both arms ran the same retrieval code)
and are re-run rather than reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two kernel port checklists record the extractor configs as surveyed
at porting time; their structTypes lines are marked superseded rather
than rewritten, so the surveys stay readable as history.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four cases in extraction.test.ts: a Rust union carrying an
`impl Trait for` edge and owning the impl's method; a named C union
alongside a forward declaration that must NOT mint a node; a
`typedef union` taking the typedef name with no `<anonymous>` twin; a
C++ union with a member function.
Verified they fail without the fix on BOTH extraction paths — the wasm
walker via CODEGRAPH_KERNEL=0 and the kernel with a staged build.
torture.c / torture.rs gain the same shapes. The parity gate compares
the two walkers rather than a snapshot, so the fixtures do not detect
the bug on their own — they pin that the fix stays SYMMETRIC. The
regression tests above are what pin that it is present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The self-query allocation probe fixture's delivered-share gates now fail. The
cause is not the new bound: allocation is unchanged between arms (parse-run.mjs
32.3% vs 33.9% on main) and tools.ts delivers the same 8,282 chars in both. What
changed is that the incidental file now DELIVERS — on main its whole section was
cut by the hard-ceiling truncation, so the fixture passed on truncation luck.
Every file on this repo obeys the new bound (max 1.40x of spendable).
Recorded as `afterCG30` with that reasoning rather than tuning the bound to
restore the pass. The over-reservation it exposes is epic CG-24's subject.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A `union` declaration produced no symbol at all in any of the four
languages that have one. The type never entered the graph, and neither
did anything attached to it — in Rust every `impl Trait for MyUnion`
lost its edge, and the impl's methods were left with a qualifiedName
pointing at a type the graph did not contain.
`union_specifier` / `union_item` were absent from the extraction layer
entirely: no `<x>Types` list on the TS side, no dispatch branch in
either kernel walker.
They join `structTypes` (kind `struct` — NodeKind has no `union`), which
is the extension point the table-driven extractors already provide. The
body guard in extractStruct is untouched, so a bodiless `union U;` stays
a forward declaration and is still skipped, exactly like `struct U;`.
`resolveTypeAliasKind` accepts `union_specifier` too, so
`typedef union { … } N;` takes the typedef's name the way
`typedef struct { … } N;` already did. Without it the anonymous union
body would mint a second `<anonymous>` node beside the alias.
Both walkers change together so kernel<->wasm parity holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
shrinkCluster keeps an oversize cluster's highest-importance member WHOLE on
purpose — an empty file section sends the agent to Read, the outcome explore
exists to prevent. What it lacked was a bound, and "never empty" quietly meant
"never bounded": on the reporting repo one file emitted 22,376 chars against a
9,181-char reservation (2.44x), past both the per-file budget and the spine
ceiling. That overshoot is what collapses `headroom` for every file below it.
The same rule has a second face. When the top member is bigger than the whole
response ceiling, the file does not overshoot — it is dropped entirely at the
renderCeiling check, so the agent gets nothing for a file it named.
renderCluster now takes a ceiling (1.5x what the file may spend — the same
multiple SPINE_CEILING already draws, and never below the cap, so a cluster
that fits is untouched). Past it the member is WINDOWED on whole lines rather
than emitted whole or dropped: leading window plus, on a flow cluster, a window
on the spine's call site. A partial window shorter than 12 lines is dropped
instead — a sliver in the session record forces the next call's dedup to shred
the block around it or re-send it — unless nothing else was emitted, where the
never-empty floor wins.
Measured on the new fixture, pre-fix vs post-fix:
monthly.ts 12,391 chars on a 3,334 budget (3.7x) → 4,941 (1.48x)
quarterly.ts dropped, no headroom left → 4,004 delivered
Also: the diagnostic now reports `spendable` (reservation + inherited slack)
alongside `reserved`. Every render bound reads the former, so reporting only
the latter makes an ordinary carry-forward read as a file spending over budget
— and it made the overshoot this issue is about unmeasurable. A windowed file
is now flagged `clipped` too, instead of presenting a window as the whole file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A live, auto-sync-maintained index does not converge to a clean full
rebuild of the identical tree. On codegraph's own repo, 4.3% of distinct
edges are wrong in both directions (751 missing, 476 stale), dominated by
`calls` — the edges flow queries traverse and that feed the RWR mass
explore ranks files by.
Raw edge rows differ by only +0.7%, because the divergence is
bidirectional and nets out; any drift check must compare edge SETS.
Rebuild-vs-rebuild is 0, so the indexer is deterministic and this is not
noise. Node sets are identical and every integrity check is 0 on both
indexes, so this is stale cross-file resolution, not accumulated residue.
`diff-index-drift.mjs` is read-only and takes two index paths — rebuilding
is the caller's job, so the tool can never clobber the artifact it is
measuring. It also refuses a missing path, since node:sqlite creates an
empty database rather than failing and an empty schema reads exactly like
a stale pre-migration index.
Diagnostic captures from the originating incident are deliberately NOT
committed: they contain verbatim source from a private repo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Swept every benchmark doc for figures produced off `result.usage` and fixed
the ones that had raw logs to re-derive from.
residual-context-occupancy.md — the sonnet 3-turn throughput table. Re-derived
from the preserved logs: tokens saved 23% -> 56%, and vscode's "98% MORE tokens
with codegraph" was never real, it is 41% fewer. Cost, time and tool calls were
never affected by this field and are unchanged. The occupancy table itself is
measured off the timeline, so every number in it stands -- including the 82%
higher residual, which is the finding the document exists for.
call-sequence-analysis.md — this doc DIAGNOSED the bug and its reproduce block
claimed the aggregator summed per-turn tokens. It did not, until 04c0f8e. Noted,
with the three wrong results the gap produced: the excalidraw cut recorded here,
the sonnet campaign, and the Opus re-measure that invented a token regression.
answer-directly-vs-explore-agent.md — build 0.9.4, 2026-05-24, raw logs gone.
Cannot be re-derived, so flagged rather than silently left or invented: its
token figure is indicative, its turn/read/context findings do not depend on the
broken field and stand.
The remaining benchmark docs (allocation-ab-1500, dedup-cg20, allocation-
efficiency, feedback-metrics) carry no throughput tables — checked, clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-measured on the CLI-blocked harness at the README's own stated methodology
(claude-opus-4-8, single question, median of 4, same 7 repos), with tokens
summed per turn per 04c0f8e.
What moved, and what did not:
tool calls 89% -> 88% fewer holds
file reads 0 on all seven holds
tokens 69% -> 62% fewer holds
time 20% -> 53% faster was badly understated
cost 60% -> 44% cheaper genuinely lower
Time was the most wrong figure in the old table, in our favour: every repo is
faster with the graph, including the two that previously carried a footnote
excusing them for being slower. That footnote is gone. Cost is the one real
downgrade, and it varies with how much discovery a question demands rather
than with repo size -- 57-78% where the file-reading arm needed 28-43 tool
calls, near-even on Gin at 7. Said plainly instead of averaged away.
Methodology now records that the CLI is blocked in BOTH arms and why: on an
unblocked harness the control reached codegraph through Bash in 26 of 28 runs,
so it was not a control. All 28 attempted it here; all 28 were blocked.
The context-footprint note from a7db24d is unaffected -- occupancy is measured
off the timeline, not the token field that was wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Tokens processed" was read off result.usage. That was correct when the README
figures were measured; in current Claude Code the field reports the LAST turn
only. Nothing here changed — the host did, silently — and the harness kept
reporting the smaller number.
The error is one-sided, which makes it worse than noise: it under-counts
whichever arm takes more turns, and that is always the WITHOUT arm. On the
2026-08-05 campaign it turned a real 62% token saving into 19% and invented a
token REGRESSION on tokio (-41%) and alamofire (-25%) that does not exist. Those
numbers were one push away from the README.
Now summed per assistant request and deduped by message.id, the same rule the
occupancy timeline already used — Claude Code emits one event per content block
carrying identical usage, so summing per event double-counts (~1.7x measured).
CLAUDE.md already warned about this field. The code did not follow; it does now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The benchmark table measures throughput -- tokens processed, tools called,
cost to reach one answer. It has never measured what is still resident in the
window afterward, and on that axis codegraph costs more: ~80% more retrieval
context left behind than a file-reading agent, on all seven repos.
That is the axis issue #1500 reported, and it is structural rather than a
defect -- one dense payload that answers the question and stays, versus
grep-and-read churn that evicts. Worth stating plainly next to the cost note
rather than leaving a user to discover it in a long session.
The Opus 4.8 single-question figures in the table are deliberately untouched:
the occupancy campaign ran sonnet / 3-turn, a different regime, and nothing
measured there licenses restating them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Never re-serve source this session already sent: session-scoped state (CG-17)
plus a precise back-reference in place of the bytes (CG-18). Duplicated source
drops 6.4% -> 0.74% of the response, at flat cost per call, with more unique
source in its place.
Bar 4 of CG-20 -- "residual occupancy must drop" -- is NOT met, and the gate
records why: CG-18's accepted rule spends reclaimed bytes on files not yet
shown rather than banking them, so a design that re-spends every byte cannot
lower the byte count. The two requirements were mutually unsatisfiable as
written. Bars 1-3 (no extra Reads, no abandonment, no bucket shift) pass across
24 runs on both arms.
Kept on that basis, and cheap to reverse: CODEGRAPH_EXPLORE_DEDUP=0 disables it
at runtime. Full gate: docs/benchmarks/explore-dedup-ab-cg20.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Read is 0 in all 24 runs of both arms across client-go and excalidraw, no
isError, codegraph is the last tool in every run, and both "Read a file we
returned" / "did not return" buckets are empty — with back-references
demonstrably reaching the agent in 8 of the 9 multi-call runs.
Residual occupancy is flat, and the measurement shows it could not have been
anything else: CG-18 was accepted on the rule that reclaimed bytes get spent on
unseen files rather than banked, so the byte count cannot fall. What moves is
the duplicate fraction of that residual — 87% less across the agent runs, 86%
and 94% on two matched deterministic replays.
Also recorded: dedup.savedChars is a pre-clip figure (11,450 reported against
1,042 chars actually re-served), so tuning off it inflates the win ~7x.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extends the session-state design doc with the layer built on it: what gates a
withheld span (session, content fingerprint, two size floors, kill switch),
why the fingerprint and not #1474's drift flag, the two channels the
reclaimed bytes leave by, and why "no duplicate ranges across calls" holds
for every call that had something new to say rather than universally.
A later explore call re-served whatever it re-ranked, so on the #1500 report
the 4th call spent its envelope on the spine the 1st call had already
delivered. CG-17 recorded what was served; this acts on it.
What a withheld span becomes is the whole design: a POINTER, never a silence.
An insufficient-feeling response is what sends an agent to Read, and one or
two of those early in a session teach it to abandon codegraph — so the
replacement names the file, the symbols and the line spans, and says both
that the source came from THIS conversation and that the file has not changed
since.
- Content fingerprint, not the drift flag, gates it. They answer different
questions: two calls inside one drift window served the same current bytes,
while a file edited AND re-synced between calls is never "stale" and yet
the agent's copy is now wrong. An edited file re-emits in full.
- Only a covered run of >= 8 lines is replaced, and a remainder under 160
chars folds into the pointer. Below those the pointer costs more than the
source and the block reads as shredded — a fence holding `228\t` is a
broken-looking response, which is the expensive failure.
- The reclaimed bytes go to files the agent has NOT seen, two ways: a smaller
`sourceSpent` hands slack down CG-21's carry-forward pool, and a fully
back-referenced file gives up its maxFiles slot the way a cliffed one does.
Within a file, the cluster shrink now reads the DEDUPED length, so it never
drops new symbols to make room for source it isn't sending.
- If dedup suppresses everything and nothing new takes its place, the top
suppressed file is spliced back in whole. An all-pointer response is the
shape that reads as "codegraph found nothing"; one re-served file is the
cheaper mistake.
Kill switch: CODEGRAPH_EXPLORE_DEDUP=0.
Explore answers every call as if it were the first: no record of the files
and line ranges it already sent, so a 4th call re-serves the 1st call's
spine and the tier call budget can only be asked for, never enforced.
Track it per MCP session, per resolved project root — files, coalesced line
ranges, bytes, and the call's index in the session. Nothing reads it yet:
the response is byte-identical, which the suite pins against an untracked
call of the same query.
The daemon shares ONE ToolHandler and a pool of worker threads across every
connected client, so the state can live neither on the handler nor in a
worker. It lives on MCPSession and is handed to execute() per call; the
session's view rides DOWN on the args and the call's emission rides BACK on
the ToolResult, both as plain properties so they survive the structured
clone to and from a worker. execute() records the emission on the main
thread and deletes it unconditionally — including for callers that track
nothing, like the CLI — so it can never reach the wire. A view a client
spells itself is discarded rather than trusted.
Ranges are reported by the render loop itself (buildSection now returns the
spans it slices alongside the text), and only files that survive the final
hard-ceiling truncation are recorded. Where a bound forces a choice the
record keeps FEWER ranges than were emitted: under-reporting re-serves
something the agent has, over-reporting withholds source it never saw and
costs a Read.
Every bound caps detail only — callCount keeps counting past eviction, so
CG-19's decay can't reset itself every 8 calls.
Fills the empty RESULTS placeholder with the 2026-08-05 campaign
(bgjob-6d357cd2: 7 repos x 2 arms x 4 runs x 3 turns, 137 min).
The finding is not the flattering one. Retrieval residual is 82% HIGHER
with codegraph and share-of-context 27% higher, on all seven repos —
vscode 67k resident against 18k. At the same time six of seven
without-arms *process* more total tokens (gin 660k vs 290k) while
leaving less behind. Both are true: one dense verbatim payload stays
resident where many small Read/Grep results evict. This corroborates
issue #1500 on our own harness; the aggregator used to print it as
"-82% lower with codegraph" until the sign bug at 520ed9d.
Also:
- States the regime everywhere. This ran claude-sonnet-5 / 3-turn; the
README's table is Opus 4.8 / single-question. Measured 24/23/20/84
against the published 60/69/20/89 — model and turn count, not
contamination. Records the two inversions honestly (vscode processes
98% more tokens, django costs 17% more) and that 4 of 28 with-arm
sessions still touched Read.
- Corrects the "Settled" section, which claimed a 7-repo baseline
existed before one did, and adds the unclaimed Opus rerun.
- Records the contamination gate: 0 CLI calls returned output in 56
sessions, but 29 attempts were blocked — 26 of 28 without-arm
sessions tried. no-cli-shim.sh is load-bearing, not precautionary.
- Records the secondary readings as absolute, not before/after: 86.7%
allocation efficiency pooled over 110 calls, read-of-a-file-we-
returned 2%, explore-again 73% and ambiguous by construction.
README.md is deliberately untouched — restating its numbers from sonnet
3-turn data would be wrong. A proposed README paragraph is drafted at
the end of the benchmark doc for the maintainer to accept or reject.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The occupancy summary hardcoded "% lower with codegraph". pct(w, wo) is the
reduction going with->without, so a negative value means the with-arm's
residual is LARGER -- and the line printed "-82% lower with codegraph" for the
case where codegraph in fact occupies 82% MORE. A double negative that reads as
a win and inverts the headline of the whole metric.
Direction now follows the sign in words, and the negative case says what the
shape actually is: codegraph front-loads one large verbatim payload that stays
resident, where Read/Grep churn many small results that evict. Fewer total
tokens processed and a larger persistent footprint are both true at once --
that pair is the axis issue #1500 reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three per-metric docs told a maintainer what each number means; none said
which one answers which question, which harness produces it, or how to read
the arm table. agent-eval-feedback-metrics.md is that page — the metric →
question map, when to reach for ab-new-vs-baseline.sh (isolates a change,
both arms codegraph-on) versus run-all.sh (with vs without, a different
question) versus bench-readme.sh, the worked CG-22 express table where all
three read together, and the bucket → fix mapping. Not a fourth restatement:
the derivations stay where they are and each doc now points here.
The caveats that change how the summary table is read are carried over rather
than dropped — allocation efficiency is relative (attribution is by citation,
so same-question builds only, and never "codegraph wastes N%"), occupancy
shares are Claude Code / 200k and do not transfer between hosts while the arm
ratio does, sufficient is not correct, small-n throughout. Plus the
contamination row, which means different things in the two harnesses and is
the first thing to look at in both.
Also records that the CG-8 7-repo bucket block no longer re-derives:
bench-readme.sh overwrites /tmp/ab-readme, so the swept logs are gone. The
current logs give a different distribution over the same 62 calls, and the
CG-8-era and current classifiers agree exactly on them — so nothing moved
under the metric, the corpus did. CG-13 re-establishes the baseline.
The three metrics existed but only run-all.sh printed them, one block per
run. ab-new-vs-baseline.sh — the harness that actually isolates a retrieval
change, both arms codegraph-on — grepped its parse output down to `by type`
and `Result`, so occupancy, sufficiency and allocation never reached the
maintainer running the A/B they were built for.
Both harnesses now print the three blocks under every run and end with one
compare-arms.mjs table: median [min–max] per arm across RUNS, sufficiency
pooled (it is per-CALL, so median-of-run-percentages would weight a 1-call
run like a 5-call one), allocation pooled by bytes and per run. The table is
"did it move?"; the per-run blocks stay the "why?" — only they name the query
that fell short and the file nothing cited. It reproduces the recorded CG-22
express result off logs already on disk: baseline 3/6 calls in the
`Read a file we returned` bucket at 82.0%, new 0/5 at 96.9%.
parse-bench-readme.mjs gets the same two metrics as a with-arm table, so the
CG-13 campaign aggregates all three rather than occupancy alone.
Also folds the CLI-block shim into no-cli-shim.sh and gives it to
ab-new-vs-baseline.sh. There it is not a with/without leak but an attribution
one, and it breaks all three metrics at once: output arriving through Bash is
charged to Bash in the occupancy table, and an explore issued through the CLI
is not a tool call at all, so it never reaches the sufficiency classifier or
the allocation parse. The run silently drops calls from every number.
The daemon pre-warm and the model policy are untouched.
Validated on one live gin arm (2 explores, 0 Read, all three blocks + table)
and against the cg22/cg15 and ab-readme logs. Selftest 68/68.
Records the sweep over every A/B log on this machine (103 sessions, 297
explore calls, 0 crashes) and, more usefully, the new-vs-baseline arm
table the metric exists for: express 82% → 100%, cg21/client-go 67% →
95%, two pairs going the other way.
States the caveat in the places it can be misread: the corpus median sits
in the eighties because these are flow questions whose answers name most
of the chain, the metric is byte-weighted, and an agent can use a file
without citing it. It compares two builds on one question; it is not an
absolute waste figure.
The envelope view needed a human to say which files answer the question
(`--answer <glob>`). This reads it off the agent's own final answer and
reports one number per run and per call: bytes returned for files the
answer cited, over all bytes returned. That is the #1500 defect as a
number instead of a hunch.
Attribution has two channels, ranked so the weaker one stays separable:
the answer naming the file (reported alone as the conservative floor),
and the answer citing, in a code span, a symbol only that file DEFINES.
Three guards keep the error from leaning optimistic — the direction a
tuning metric must not lean:
* Only symbols the file defines. Section headers render `name(kind)`
for call sites too, and crediting those marked excalidraw's
dragElements.ts used because the answer named `mutateElement`.
* A definition beats an import alias of the same name (`variable`),
or `lib/application.js` gets credit for `require('./utils')`.
* A name on 3+ returned files identifies none of them.
Bare basenames count as citations (agents write `utils.js:225` in prose)
but only for extensions the envelope shipped, so `res.send` and
`mime.contentType` — the same token shape — do not read as files.
Both the envelope view and this share one parse of the rendered markdown
(parseExploreCall), still not the CG-4 sidecar: the sidecar exists only
on a post-CG-4 build and so cannot measure a baseline arm.
The 14 multi-turn with-arm sessions of the README corpus exercise every bucket
(62 calls), so the sweep is no longer one repo family: 47% explored again, 11%
Read a file we returned, 2% Read a file we did not, 23% Grep/Glob, 18% moved on.
Flagged as a baseline rather than a verdict -- three-turn sessions on hard flow
questions, and "explored again" includes the legitimate second call on a repo
whose budget is 2-3.
The recall bucket's one real instance is worth reading: explore returned
InteractiveCanvas.tsx and named StaticCanvas.tsx without shipping it, and the
agent went and read exactly that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records what each bucket means and which fix it points at, the four rules that
keep the classification honest (same-message calls, bookkeeping tools, subagent
threads, earlier-explore files), and the three real transcripts it was
hand-checked against -- including the excalidraw canvasNonce run, where it
independently found the data-flow frontier CLAUDE.md already documents: 0%
sufficient, without being told what to look for.
Also states what it does NOT say: sufficient is not correct, one Read is a vote
rather than a proof, and bucket 1 is ambiguous by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The agent's next action after a codegraph_explore is free ground truth about
whether the response was enough, and the harness was discarding it. Every run
now bucketed: explored again (insufficient), Read a file we returned
(allocation -- right file, wrong bytes), Read a file we did not return (recall),
Grep/Glob (recall, weak), or moved on (sufficient). The buckets are chosen so
each one names a distinct fix.
The classifier lives in parse-run.mjs next to the occupancy math and takes raw
events, so parse-session.mjs reuses it for interactive runs -- no new
scripts/agent-eval/*.mjs, which would score into the self-query eval fixture's
corpus.
Four rules, three of them found by validating against real transcripts rather
than reasoned up front:
* A call in the SAME assistant message as the explore predates its response,
so it is not a verdict on it. Stepped over, counted as `concurrent`.
* ToolSearch/TodoWrite carry no signal; the call behind them is the verdict.
* SUBAGENTS ARE A SEPARATE THREAD. Claude Code interleaves a subagent's calls
into the same stream under parent_tool_use_id -- verified on a live
excalidraw run where a delegated search's greps landed between the parent's
own calls. Matching reactions across threads scored the subagent's grep as
the parent's verdict on an explore it never saw. A delegation is judged by
what the subagent did FIRST: before that, the same run reported 33%
sufficient while the subagent was off grepping for the file, which is the
one direction of error a tuning metric must not have. In interactive
sessions the subagent is a separate FILE instead, so parse-session.mjs
stitches the threads back with the toolUseId in agent-*.meta.json.
* A re-read of a file an EARLIER explore shipped is still allocation, not
recall -- filing it as recall aims the fix at the wrong end of the pipeline.
Shell file access counts too (`sed -n 100,200p f` reads, `grep`/`find` search),
since both arms have Bash and counting only the Read tool would score those
explores as sufficient. A heredoc or redirect is writing, not reading.
Validated by hand on cg22/ab-express/run-baseline-1 (explore, explore, Read of
lib/response.js which explore #2 returned -- the #1500 allocation bug as a
bucket instead of a hunch; the new-build arm is 100% sufficient) and on
cg15/ab-express/run-new-2 (four explores, the last returning lib/utils.js which
the agent then read at offset 195). Swept over all 76 A/B logs on this machine:
0 crashes, 176 calls bucketed. --selftest covers every bucket, both thread
rules, delegation, shell reads/searches and errored calls: 46/46.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CG-3 branched from main before CG-1 landed and rewrote parse-run.mjs wholesale
into an exported parseSession(), which dropped CG-1's --envelope/--answer
reporting entirely. That view is the instrument the CG-1/CG-22 allocation gate
measures bar 2 with, and it is in that benchmark's documented reproduce steps,
so it cannot be lost to the merge.
Resolution takes CG-3's rewrite as the structure and ports the envelope feature
into it: parseSession now collects codegraph_explore response text in call
order, formatEnvelope renders the per-file share, and the CLI parses
--envelope/--answer ahead of the positional filter so a glob is never mistaken
for a log path.
The glob sentinel stays written as a \u0000 escape, never a literal NUL byte --
a raw one makes git treat the whole script as binary, exactly as the comment
there warns.
Verified: --selftest 18/18, and a synthetic explore transcript reports the
expected per-file shares and answer-set total.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hook denies the invocation, so a denied attempt puts no codegraph output in
the window and must not disqualify the run -- only a call that actually returned
content does. Attempts are still reported, since an agent hunting for the CLI is
worth seeing.
An agent denied `codegraph` on PATH ran `find / -maxdepth 4 -iname "*codegraph*"`,
found the binary, and invoked it by ABSOLUTE PATH — 12 times in one without-arm
run. So block the invocation itself with a PreToolUse hook on Bash, written into
the run's output dir as an artifact alongside the MCP configs rather than as a
repo file.
The pattern matches command positions only, so looking is still allowed and only
using is denied: `grep codegraph src/`, `ls .codegraph` and `which codegraph`
pass through, while `codegraph explore`, `/abs/path/codegraph …`, `cd x &&
codegraph …` and `VAR=1 codegraph …` are refused. run-all.sh proves both
directions at startup and refuses to run if either fails. parse-run.mjs's
detector uses the same rule, so prevention and detection cannot drift — and it no
longer false-positives on the corpus path, which contains the word codegraph.
Verified end-to-end: the without-arm now probes with `ls .codegraph; which
codegraph`, finds nothing usable, and falls back to Read/Bash.
The without-arm had no MCP server but still had Bash, and the target repo carries
the .codegraph/ index the with-arm needs. Agents found it: 14 of 15 without-arm
runs in a 7-repo pass ran `codegraph explore` through Bash, one of them via
`ls .codegraph && codegraph explore ...`. That arm was measuring
codegraph-over-CLI, not codegraph-absent, so every without-arm number it produced
was wrong. It bit the with-arm too -- output arriving through Bash is attributed
to Bash, understating what codegraph itself occupies (1 of 15 runs).
Both arms now run on a PATH where the CLI is hidden, so the MCP server is the
only way to reach codegraph and stays the single variable. The binary shares a
directory with tools the run needs -- claude itself sits next to it -- so the
directory is substituted in place by one of symlinks to every entry except
codegraph, preserving PATH order and precedence. The run aborts if claude or node
did not survive the substitution.
Prevention alone would fail silently the next time the CLI lands somewhere new,
so parse-run.mjs flags any Bash command naming codegraph and parse-bench-readme
drops contaminated without-arm runs from the aggregate (CG_INCLUDE_CONTAMINATED=1
keeps them). CG_ARMS re-runs one arm without redoing the other.
A median over 2-3 runs hides swings big enough to flip a repo's sign. On vscode
the without-arm ranged 40k to 67k and the with-arm 59k to 65k across two runs;
the deciding variable is the tool mix, since a with-arm run that reads files ON
TOP of calling explore pays for both.
parse-run.mjs --selftest runs the math over synthetic transcripts with known
answers: attribution, message.id dedupe, compact_boundary, FIFO micro-compaction,
and multi-turn stitching. It found a real bug. A gap where the window also SHED
content has a delta far below what was added, which reads as absurdly dense text
and dragged the whole run's ratio with it -- a shed gap in the fixture pushed
2.5 chars/tok to 4.4 and left the wrong result resident. Shedding can only push a
gap's ratio up, so the calibration now takes the lower median as its centre,
drops gaps well above it, and pools the rest. Runs that never shed are unaffected
(gin and vscode re-measure identically).
Also drafts docs/benchmarks/residual-context-occupancy.md -- method, error bar,
and the limitations this metric does not settle. Baseline numbers to follow.
On a gap that is >=95% one tool result, the measured context delta IS that
result's token count, so the spread between it and the run-level ratio is the
attribution error. Median over such gaps: +/-1-2% on real runs.
The first request's prompt is system + tool schemas + the question, before any
tool has answered, so differencing the arms' ctxBase prices what codegraph
occupies whether or not the agent ever calls it. Measured on gin: +775 tokens,
small because the tool is deferred -- only its name is in the initial listing.
The A/B arms reported cost, tokens, time and tool counts for one headless
question. They could not report what issue #1500 actually measured: how much of
the context window a tool's responses still occupy once the question is
answered, which every later turn is then charged for.
parse-run.mjs now measures that. Tokens are measured, not estimated: for each
assistant request, input + cache_read + cache_creation is the exact token count
of its whole prompt, so consecutive requests differ by exactly what was appended
between them. That delta is priced against the characters in the gap, calibrated
on gaps that are >=80% tool result. Explore output lands near 2.3 chars/token, so
the usual bytes/4 estimate would have under-counted it by ~40%.
Content also leaves the window, so residual is tracked apart from contributed:
a compact_boundary clears the resident set, and a mid-run context drop is
micro-compaction, which sheds the oldest tool results first and is applied FIFO.
run-all.sh takes "Q1||Q2||Q3" and runs them as one resumed session, one segment
file per turn; parse-run.mjs stitches the segments back together. bench-readme.sh
now runs each README repo as a three-turn session (CG_TURNS=1 restores the
single-question form). parse-bench-readme.mjs reports the arms' retrieval
residual side by side -- codegraph's responses against the without-arm's
Read/Grep/Bash -- in absolute tokens, share of context, and share of window, and
says so explicitly when the rows it aggregated were single-turn.
Two transcript traps are handled and documented at the call site: Claude Code
emits one assistant event per content block, all carrying the same usage (summing
per event double-counts every turn with both thinking and a tool_use), and the
streamed output_tokens is a partial snapshot.
Occupancy lives in parse-run.mjs and is imported by the aggregator rather than
extracted to a module -- a new scripts/agent-eval/*.mjs scores into the
self-query fixture's own corpus and moves its numbers.
CG-21 fixed the unspent-reservation defect and re-ran the A/B itself. CG-22 is
the gate proper: CG-15's setup, unchanged, measured independently of the task
that wrote the fix. RUNS=3, both arms codegraph-on, sonnet/high,
CODEGRAPH_NO_PROMPT_HOOK=1 on both, baseline pinned to 49c11fc by SHA, fresh
clones of the same three repos and the same three questions.
All four bars pass. Read = 0 in all 12 new-arm runs (express 3, excalidraw 3,
client-go 6) while the baseline reads in 3 of 3 express runs and 1 of 6
client-go runs; the express run that failed CG-15 with 4 Reads of lib/utils.js
now reads nothing and receives the file whole. Answer share >= 66.6% in every
new run. Medians: express 26s -> 24s, excalidraw 26s -> 26s, client-go
35s -> 36.5s at n=6 with fully overlapping ranges.
Deterministic core re-measured on BOTH builds in one session rather than
quoted: lib/utils.js renders whole at 6,380 B on baseline and on HEAD (583 B
stub under CG-12), and the source envelope goes 13,849 -> 14,913 against an
unchanged 13,000 budget, so the reservation is spent and the envelope stops
shrinking. client-go's +1.5s median is attributed away from the build: explore
latency 669 vs 668 ms (n=5) and the new build's deterministic response is both
smaller (15.8K vs 18.9K) and more concentrated (top file 50.2% vs 35.7%).
Two counter-points recorded as measured, not smoothed: excalidraw's new arm
runs below its baseline on answer share (66.6-81.9 vs 75.5-92.7, all well over
the bar), and this session's client-go baseline sampled well (85.6-100%), so
the #1500 gap is smaller here than in CG-21's session.
CHANGELOG: the two #1500 bullets were multi-sentence paragraphs carrying
implementation detail. Rewritten to house style as four bullets that lead with
the symptom, with the mechanism, the banner catalogue and the old-behaviour
contrast dropped; the @LeDuyViet credit and the re-index note stay.
Suite green on the measured build: 171 files, 2,868 passed, 6 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-runs CG-15's agent A/B on the fixed build: same harness, same three
prompts, same baseline ref, n=6 per arm on express and excalidraw.
Read = 0 in all 15 new-arm runs. The express regression that routed the
defect to CG-21 does not reproduce in 6 attempts, and the baseline now
reads in 4 of 6 while the new arm reads in none (median 24.5s -> 21.5s),
so the control beats the arm it previously lost to. client-go holds
92.7-96.2% answer share against a baseline run at 53.8%.
Excalidraw's new arm is ~8s slower at the median and that is recorded as
NOT attributable to the build rather than waved through: explore's own
latency is 374ms vs 372ms on the same query and index, the deterministic
responses differ by +2% with one byte-identical, and the unchanged main
build's own median moved 34s -> 26.5s between the two sessions — the same
magnitude as the gap.
Bars were not re-baselined; they are CG-15's four, applied to a larger
sample. The CG-15 section is kept intact and marked superseded, because
its root-cause analysis is the record of why the fix looks like it does.
Found reviewing the CG-21 fix rather than by a failing test, and it is the
same defect inverted. A whole render that overruns `renderCeiling` is
skipped ENTIRELY (the branch refuses to slice a file mid-method), so a buy
that is approved by the funding pool but refused by the ceiling trades a
clustered section for NO section.
Only reachable on the 24K tiers. The funding line is
`reservedTotal + 0.15 * envelope` — ~27.2K when a medium repo saturates —
while `renderCeiling` is `min(1.5 * envelope, 25000) - 600` = 24.4K, so
funding can approve ~2.8K the ceiling then refuses. At 13K the line is
~14.4K against an 18.9K ceiling and the two cannot cross, which is why the
small-tier fixtures cannot see it.
Failing the test in `buysWhole` drops through to the cluster path, which is
bounded by `headroom` and always renders something. The GRACE arm is left
alone deliberately: a file within a sliver of its reservation that still
does not fit is genuinely at the end of a full response, and that predates
this epic.
Verified inert on the three A/B repos, so the agent A/B measured the same
behaviour: excalidraw byte-identical on 3 queries, client-go byte-identical
on 3 queries, express reproducer unchanged at 15,984. Full suite green
(2,867 passed; one unrelated fs.watch timing flake that passes 30/30 in
isolation).
Records both levers, the funding-pool design (and the per-file version that
dropped payslip_builder.go), the resolved memory-budget.ts exception, and the
two hermetic fixtures with their mutation matrix.
The CHANGELOG clause 'no longer trimmed while a smaller, weakly-related one is
included whole' was imprecise after CG-21: the smaller file often IS still
included whole now, when its share nearly covers it. Reworded to say what the
fix actually guarantees.
A file whose proportional reservation lands below its own size stopped
rendering whole, and the fallback cluster render could leave most of that
reservation unspent — the bytes were neither delivered nor redistributed.
Found by CG-15's agent A/B on the express control: `lib/utils.js`, the
top-ranked file, was reserved 3,870 chars and spent 583. The whole-file
grace bound (reservation + a sliver) sat just under the file's 5,293
bytes, so the whole render was declined and three matched symbols became a
stub. The source envelope fell 13,849 -> 9,241 against an UNCHANGED
budget, and the agent Read the file back four times in 1 run of 3.
Two levers, per the task's candidate fixes:
- WHOLE_FILE_BUY_FRACTION: a reservation that already covers 60% of a
file buys the whole file. Funded from ONE shared overshoot pool sized
at 15% of the envelope, spent in rank order. Per-file funding is the
version that fails, and it fails the same way the bug does — the merit
test is a ratio, so several files qualify at once and N independent
overshoots push the last section past the render ceiling. Measured on
the payroll fixture: three files bought whole and `payslip_builder.go`
was dropped entirely. A dropped section is strictly worse than a
clustered one.
- Reservation carry-forward: what a file cannot spend goes to the next
file down, bounded by MAX_SHARE. Tracked as two running totals rather
than a `spent` variable threaded through the render loop's dozen exit
paths, so no path can forget to account, and symmetric — a buy that
overshoots suppresses slack until a later under-spend covers it.
Express reproducer: `lib/utils.js` 583 -> 6,268 whole, envelope 9,241 ->
14,505 on the same 13,000 budget. The `memory-budget.ts` exception CG-14
documented is RESOLVED rather than re-justified: it ships whole again at
5,672 (27.3%) while `src/mcp/tools.ts` rises to 52.6% — so the answer
file wins the envelope AND no previously-unclipped file is clipped, which
is CG-12's own acceptance criterion finally holding.
Two hermetic fixtures added, one per lever, because nothing in the suite
had this shape — which is how it shipped. Both mutation-tested: removing
the buy arm reddens 3, removing the carry-forward reddens 2, and removing
the funding guard reddens 4 (including payroll's dropped
`payslip_builder.go`). Their `fixture shape` blocks are load-bearing: the
gates pass vacuously if a target ever drifts inside the grace bound, so
the window is asserted directly.
Full suite green (2,868 passed); both #1500 regression fixtures pass.
Three separate engineer-shaped entries (CG-5 generated detection, CG-10
scoring, CG-12 allocation) become two user-facing bullets under Fixes, in
the shape #1500 actually reported: explore concentrates on the code that
answers the question, and a generated CRUD/protobuf layer no longer crowds
out the hand-written code beside it.
Per the CHANGELOG rules: strips the benchmark counts and percentages (the
client-go 2,001-file count, the quarter-to-four-fifths envelope shift) and
the internal symbol names, keeps the re-index note, and credits the
reporter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three repos, both arms codegraph-on, sonnet/high, 3 runs per arm.
PASS on the two medium repos: client-go (the reporter's Go shape, 2,001 of
2,454 files generated) and excalidraw hold Read 0 in every run of both arms,
excalidraw goes 34s -> 24s at the median with one fewer explore call, and the
generated clientsets/informers that took 10.5%% of a baseline envelope appear
in no new run.
FAIL on express, the small control, in 1 run of 3: 4 Reads and 52s against a
baseline that read once. Not agent variance — replaying that run's query
deterministically, lib/utils.js goes from 6,380 bytes whole to a 583-byte
cluster stub and the envelope shrinks 13.8K -> 9.2K against an unchanged
13,000 budget. The diagnostic shows the allocator was right and the render
loop was not: utils.js is the top-ranked file, was reserved 3,870 chars, and
spent 583. The whole-file bound (allowance + grace = 4,450) lands just under
the file's 5,293 bytes, so the whole-file render is declined and the unspent
reservation is dropped rather than redistributed.
Bar 1 is the hard gate, so per CG-15's acceptance rule the design goes back to
CG-12 — the budget is not to be widened to compensate. Two candidate fixes are
written up in the design doc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Killing ab-new-vs-baseline.sh mid-baseline-arm left the engine checked out at
the baseline ref with the post-baseline files deleted, so every later build in
the working tree was silently the OLD code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ab-new-vs-baseline.sh now builds and indexes once per arm and runs the task
RUNS times (default 1), so the >=2-runs-per-arm rule costs one build instead
of N. Both arms run with CODEGRAPH_NO_PROMPT_HOOK=1 — the machine's ambient
front-load hook resolves to whatever is in dist/, a second uncontrolled
channel that confounds the tool-call counts — and point explore's CG-4
diagnostic at a per-arm sidecar.
parse-run.mjs gains --envelope/--answer: the per-file share of the explore
source envelope, parsed from the rendered markdown so it works on ANY build.
The CG-4 sidecar only exists post-CG-4, so it cannot measure the baseline arm;
this is the only view that measures both arms the same way. Folded into
parse-run.mjs rather than added as a new script on purpose: a new file named
after explore's budget scores into the self-query fixture's own corpus and
moved its answer share 59.9%% -> 47.9%%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Coverage for the CG-12 allocator, built around "would this go red if the
lever were removed" rather than line coverage — every way this regresses
is silent, ending in an agent falling back to Read.
Unit (`explore-proportional-allocation.test.ts`, 18 -> 38): calibration
pins, envelope safety across every tier and 30 candidate shapes, the
cliff boundary, spine weighting/trim survival, the diffuse control, and
the degenerate inputs — identical scores, a lone file, a runaway top
scorer, zero results, maxFiles 0, a non-finite score.
End-to-end (`explore-allocation-e2e.test.ts`, new): CG-6's second
regression fixture as a deterministic synthetic mirror — a large relevant
file, a small helper that used to win by shipping whole, and an
incidental `explore`/`BUDGET` collision — asserting per-file budget
share, not file presence. Plus degenerate result sets and a survey-style
diffuse control through the real render loop. The live self-query arm
stays in probe-allocation.mjs, where drift is a number to re-baseline
rather than a red suite.
Reverting the render loop to the pre-CG-12 rules reproduces #1500 on the
mirror exactly and takes 5 e2e + 2 payroll gates red:
file score pre-CG-12 CG-12
src/mcp/allocator.ts 77.5 4,843 (39.7%) 9,335 (80.1%)
src/util/budget-math.ts 36.0 6,079 (49.8%) 1,037 ( 8.9%)
Two defects the invariants surfaced, both fixed in tools.ts:
- rounded shares could sum past `pool`, so "reservations fit the
envelope" was approximate rather than exact; both terms now floor
- a non-finite score made every share Infinity/Infinity, handing the
render loop a NaN allowance; `weightOf` now fails safe to 0
Also adds a hard-ceiling gate to the payroll fixture — at 19.3K against
a 19.5K ceiling it is the only fixture that stresses the ~25K inline cap
— and exports EXPLORE_ALLOCATION so invariant tests read the constants
while one test pins the literals.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The explore envelope used to follow FILE SIZE, not relevance. Every admitted
file was capped at the same flat `maxCharsPerFile`, while the whole-file rule
handed anything under `maxCharsPerFile * 3` its entire contents — a 3x swing
decided by how big a file happened to be:
- self-query: `memory-budget.ts` (score 18) shipped whole and took 51.2% of
the response; `src/mcp/tools.ts` (score 41, 4x the graph mass, 3x the term
hits — it holds the allocator itself) was clipped at 3,800 and got 32.9%.
- #1500 Go fixture: two generated CRUD files shipped whole at ~4.5K each AND
consumed two of the tier's four file slots, so `BuildPayslip` — the
hand-written "calculate" half of the question — ranked #6 and never
rendered at all.
`allocateExploreBudget` now reserves each ranked file a share of the envelope
before anything renders, so the render loop spends a reservation instead of
racing for whatever the files above it left:
- weight = score x worth x (spine ? 2 : 1), where `worth` is `rankPenalty`
applied a SECOND time — ranking answers "is this file about the query",
allocation answers "will these bytes teach the agent anything", and
generated CRUD can legitimately rank while its bytes stay boilerplate;
- a relative cliff at 15% of the top weight (capped at SCORE_FLOOR_MAX, so a
god-file can't silence peers the score floor just admitted) gives a file
ZERO source — path, symbols and line numbers only — and crucially frees its
`maxFiles` slot for a file that earns its bytes;
- every admitted file gets MIN_CHARS, then the remainder splits by weight:
the floor keeps a diffuse survey question returning a spread, the remainder
concentrates a precise one;
- the flat per-file cap is retired as the primary guard, leaving a 70%-of-
envelope safety valve.
Two changes were needed to make the reservation bite: an oversize cluster now
shrinks by whole MEMBER symbol ranges (a single-cluster god-file previously
took ~40% more than allotted, and the file below it was dropped for lack of
room), and the arrival-order budget stops are gone — they cut files by the
order they were reached rather than by merit.
Measured: payroll-go answer group 25.6% -> 78.7%, generated 57.4% -> 0%, and
`func (s *Service) BuildPayslip` now delivered; self-query `tools.ts` 18.5% ->
60.6%, past the epic's >50% bar. Controls hold: cobra/gin diffuse survey
queries keep their file spread (3->3, 3->4), express's middleware query is
byte-identical, and gin's flow query moves its top file from the thin `ginS`
singleton wrapper to `routergroup.go`.
One documented exception to "no previously-unclipped file becomes clipped":
`memory-budget.ts` was unclipped-whole at 5,672 and now clusters within its
3.1K reservation. That is the epic's own diagnosis of the bug — it scored 18
against 58 and was taking the larger slice purely for being small.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Colby McHenryandClaude Opus 5
ctype_labandClaude Opus 5