Two permanent fixtures pinning the failure mode from issue #1500 — explore
spending its byte envelope on files that merely name-collide with the query.
BOTH FAIL TODAY, by design: they document the bug and become the pass gate
for CG-10 (scoring) + CG-12 (proportional allocation).
__tests__/fixtures/payroll-go/ — a synthetic Go service mirroring the
reporter's shape: generated FKIT CRUD beside a hand-written payroll use-case,
entered from an HTTP route. Half the generated tree carries ORDINARY names
detectable only by their `// Code generated ... DO NOT EDIT.` header (the
#1500 case, and end-to-end cover for CG-5); `payrollpb/*.pb.go` covers the
path-detectable channel. BuildPayslip, Upsert and Store each exist twice,
generated and hand-written. cycle.go sits above the whole-file window so it
clips; the generated files sit below it so they ship whole.
Asking "how does payroll cycle create and calculate payslips?" — naming none
of the answering symbols — the generated CRUD delivers 57.4% of the envelope
against the hand-written layer's 25.6%, all of the latter domain types.
cycle.go is allocated the single largest slice (30.6%) and delivers ZERO: the
hard ceiling drops its whole section. runPayrollCycleAll, the hand-written
BuildPayslip and the real Upsert never reach the agent.
The second fixture is this repo, "how does explore allocate its output budget
across files", where scripts/agent-eval/*.mjs take 71.8% against tools.ts's
18.5% despite scoring 4.6x lower. It reads the live index, so its assertions
are relative rather than fixed percentages.
- scripts/agent-eval/probe-allocation.mjs — per-file budget-share probe,
driving the CG-4 diagnostic through a JSONL sidecar so it measures the
shipping allocator. Fixture entries are hermetic (copy + re-index per run,
verified byte-identical across runs); exits 1 while any assertion fails.
- scripts/agent-eval/allocation-fixtures.json — both fixtures declared, with
the 2026-08-03 baselines.
- __tests__/explore-allocation-1500.test.ts — fixture-shape assertions green
today; the allocation assertions held as `it.fails` so the suite stays green
while the bug is open and goes RED the moment it is fixed.
Also documented and deliberately left unfixed: runPayrollCycleAll's
`s.store.Upsert` edge resolves to the GENERATED Store.Upsert, not the
hand-written one — same-name method resolution across two packages picks the
wrong receiver. It is upstream of the allocation bug, so it belongs with
CG-10's scoring work.
Refs #1500
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
How codegraph_explore divides its byte envelope among files was
unobservable — you could read a response and guess, but not say "this
file took 16% and that one took 20%." Nothing else in the budget-
allocation epic is measurable without that.
CODEGRAPH_EXPLORE_DEBUG now emits one report per explore call (stderr
table, stderr JSON, or a JSONL sidecar path). Per file: relevance score,
graph mass, term hits, ranking flags, render mode, bytes allocated vs
delivered, both shares, and whether it was clipped — plus why a ranked
candidate never rendered. Totals cover envelope vs maxOutputChars vs the
hard ceiling, the source/meta split, the selection funnel, and the score
floor and relevance-gate thresholds applied.
Allocated and delivered are reported separately on purpose: they diverge
exactly when the 25K ceiling truncates, and conflating them is how a
dropped trailing file goes unnoticed.
Off by default and byte-identical when off — it ships in the product
binary, and a diagnostic that perturbs the response by one byte would
invalidate every A/B taken with it on. ExploreDiagnostics.start() returns
null unless the env var is set, so every call site is a `diag?.` no-op.
Baseline recorded in docs/design/explore-budget-allocation.md: on this
repo, src/mcp/tools.ts gets 15.8% of the envelope while three weakly-
relevant agent-eval scripts take 61% between them — despite tools.ts
carrying 5.4x the score and 2.6x the graph mass of any of them. Small
files ship whole; the large answer file is clipped at maxCharsPerFile.
Rank ordering is correct and buys nothing. The loop also allocated 23,193
chars against an 18,000 budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Colby McHenryandClaude Opus 5