test(explore): #1500 regression fixtures for budget allocation (CG-6)

Two permanent fixtures pinning the failure mode from issue #1500 — explore
spending its byte envelope on files that merely name-collide with the query.
BOTH FAIL TODAY, by design: they document the bug and become the pass gate
for CG-10 (scoring) + CG-12 (proportional allocation).

__tests__/fixtures/payroll-go/ — a synthetic Go service mirroring the
reporter's shape: generated FKIT CRUD beside a hand-written payroll use-case,
entered from an HTTP route. Half the generated tree carries ORDINARY names
detectable only by their `// Code generated ... DO NOT EDIT.` header (the
#1500 case, and end-to-end cover for CG-5); `payrollpb/*.pb.go` covers the
path-detectable channel. BuildPayslip, Upsert and Store each exist twice,
generated and hand-written. cycle.go sits above the whole-file window so it
clips; the generated files sit below it so they ship whole.

Asking "how does payroll cycle create and calculate payslips?" — naming none
of the answering symbols — the generated CRUD delivers 57.4% of the envelope
against the hand-written layer's 25.6%, all of the latter domain types.
cycle.go is allocated the single largest slice (30.6%) and delivers ZERO: the
hard ceiling drops its whole section. runPayrollCycleAll, the hand-written
BuildPayslip and the real Upsert never reach the agent.

The second fixture is this repo, "how does explore allocate its output budget
across files", where scripts/agent-eval/*.mjs take 71.8% against tools.ts's
18.5% despite scoring 4.6x lower. It reads the live index, so its assertions
are relative rather than fixed percentages.

- scripts/agent-eval/probe-allocation.mjs — per-file budget-share probe,
  driving the CG-4 diagnostic through a JSONL sidecar so it measures the
  shipping allocator. Fixture entries are hermetic (copy + re-index per run,
  verified byte-identical across runs); exits 1 while any assertion fails.
- scripts/agent-eval/allocation-fixtures.json — both fixtures declared, with
  the 2026-08-03 baselines.
- __tests__/explore-allocation-1500.test.ts — fixture-shape assertions green
  today; the allocation assertions held as `it.fails` so the suite stays green
  while the bug is open and goes RED the moment it is fixed.

Also documented and deliberately left unfixed: runPayrollCycleAll's
`s.store.Upsert` edge resolves to the GENERATED Store.Upsert, not the
hand-written one — same-name method resolution across two packages picks the
wrong receiver. It is upstream of the allocation bug, so it belongs with
CG-10's scoring work.

Refs #1500

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-03 23:30:17 -05:00
co-authored by Claude Opus 5
parent 16e17495f4
commit bd86ad2061
25 changed files with 2687 additions and 0 deletions
+78
View File
@@ -92,6 +92,84 @@ This is the gap the rest of the epic closes: relevance-proportional allocation w
relative cliff (CG-12), on top of scoring that stops rewarding incidental name collisions
(CG-10).
## The regression fixtures (CG-6)
Two fixtures pin the failure mode so it can never silently return. Both **fail today**
that is what they are for. They become the pass gate for CG-10 + CG-12.
They are declared in `scripts/agent-eval/allocation-fixtures.json` and run by
`scripts/agent-eval/probe-allocation.mjs`, which drives the CG-4 diagnostic through a JSONL
sidecar (so it measures the shipping allocator, not a re-derivation), groups the rendered
files into `answer` vs `incidental`, and checks declared share thresholds. Needs a current
`npm run build`; exits 1 while any assertion fails.
```bash
node scripts/agent-eval/probe-allocation.mjs # both
node scripts/agent-eval/probe-allocation.mjs payroll-go # one
node scripts/agent-eval/probe-allocation.mjs --json # machine-readable
```
### 1. `payroll-go` — the reporter's shape
`__tests__/fixtures/payroll-go/` is a synthetic Go service: generated FKIT CRUD beside a
hand-written payroll use-case, entered from an HTTP route. Full description in that
directory's README. The essentials:
- Generated files with **ordinary names** carrying `// Code generated ... DO NOT EDIT.`
invisible to path-only detection, which is what makes this a #1500 fixture rather than a
`.pb.go` one — beside `payrollpb/*.pb.go` covering the path-detectable channel.
- Deliberate collisions: `BuildPayslip`, `Upsert` and `Store` each exist twice, generated
and hand-written, and the generated layer name-collides on every query term.
- `cycle.go` (227 lines) sits above the whole-file window so it clips; the generated files
sit below it so they ship whole.
Query — an architecture question naming none of the answering symbols: *"how does payroll
cycle create and calculate payslips?"*
| | allocated | delivered |
|---|---|---|
| hand-written | 48.4% | **25.6%** (all of it domain types) |
| generated CRUD | 39.9% | **57.4%** |
`cycle.go` is allocated the single largest slice (7,052 chars, 30.6%) and delivers **zero**
— the 19,500 hard ceiling drops its whole section. `payslip_builder.go` (rank #8) never
renders. So `runPayrollCycleAll`, the hand-written `BuildPayslip` and the real `Upsert`
never reach the agent, and every byte that did arrive describes either CRUD or types.
This fixture is hermetic: the probe copies the tree to a temp dir and re-indexes per run,
so two runs on one build are byte-identical (verified). `__tests__/explore-allocation-1500.test.ts`
runs the same assertions in vitest — the fixture-shape half green, the allocation half as
`it.fails` so the suite stays green while the bug is open and goes **red when it is fixed**.
**Finding, deliberately left unfixed:** `runPayrollCycleAll` calls `s.store.Upsert` on a
`*payslipstore.Store`, but the graph resolves that edge to the **generated**
`internal/gen/fkit/payroll/store.go` `Store.Upsert`. Same-name method resolution across two
packages that both define `Store.Upsert` picks the wrong receiver. It is upstream of the
allocation bug — a wrong edge pulls the generated store into the subgraph and inflates its
score — so it belongs with CG-10's scoring work, not with the fixture.
### 2. `self-query` — the same bug with no generated code in sight
The baseline above, promoted to a fixture: this repo, *"how does explore allocate its output
budget across files"*. `scripts/agent-eval/*.mjs` mention `explore` and `BUDGET`
incidentally — they are eval harnesses, not the allocator — and they are small enough to
ship whole, while `src/mcp/tools.ts` is large enough to be clipped.
At 493 indexed files (small tier): the script corpus takes **71.8%** of the delivered
envelope (79.4% allocated) against `tools.ts`'s **18.5%**, despite `tools.ts` scoring 46 vs
10, carrying 2.3× the graph mass and 3× the distinct term hits.
This fixture reads the **live** index of this repo, so unlike `payroll-go` its exact numbers
move as the repo changes. Its assertions are relative for that reason (answer group vs
incidental group, largest delivered file), never fixed percentages. Two things to know:
- The `<500`-file tier boundary is close. This repo indexes 493 files including the new
fixture; crossing 500 flips `maxOutputChars` 18,000 → 24,000, `maxFiles` 5 → 8 and
`maxCharsPerFile` 3,800 → 6,500, which moves every number in the table above. Re-baseline
after the crossing rather than treating the drift as a regression.
- Adding the `payroll-go` fixture itself moved the count 472 → 493. Its Go files match none
of this query's terms, so they change the tier arithmetic and nothing else.
### Reproducing
The query explores this repo, so **uncommitted edits to `src/mcp/tools.ts` change the