test(explore): #1500 regression fixtures for budget allocation (CG-6)
Two permanent fixtures pinning the failure mode from issue #1500 — explore spending its byte envelope on files that merely name-collide with the query. BOTH FAIL TODAY, by design: they document the bug and become the pass gate for CG-10 (scoring) + CG-12 (proportional allocation). __tests__/fixtures/payroll-go/ — a synthetic Go service mirroring the reporter's shape: generated FKIT CRUD beside a hand-written payroll use-case, entered from an HTTP route. Half the generated tree carries ORDINARY names detectable only by their `// Code generated ... DO NOT EDIT.` header (the #1500 case, and end-to-end cover for CG-5); `payrollpb/*.pb.go` covers the path-detectable channel. BuildPayslip, Upsert and Store each exist twice, generated and hand-written. cycle.go sits above the whole-file window so it clips; the generated files sit below it so they ship whole. Asking "how does payroll cycle create and calculate payslips?" — naming none of the answering symbols — the generated CRUD delivers 57.4% of the envelope against the hand-written layer's 25.6%, all of the latter domain types. cycle.go is allocated the single largest slice (30.6%) and delivers ZERO: the hard ceiling drops its whole section. runPayrollCycleAll, the hand-written BuildPayslip and the real Upsert never reach the agent. The second fixture is this repo, "how does explore allocate its output budget across files", where scripts/agent-eval/*.mjs take 71.8% against tools.ts's 18.5% despite scoring 4.6x lower. It reads the live index, so its assertions are relative rather than fixed percentages. - scripts/agent-eval/probe-allocation.mjs — per-file budget-share probe, driving the CG-4 diagnostic through a JSONL sidecar so it measures the shipping allocator. Fixture entries are hermetic (copy + re-index per run, verified byte-identical across runs); exits 1 while any assertion fails. - scripts/agent-eval/allocation-fixtures.json — both fixtures declared, with the 2026-08-03 baselines. - __tests__/explore-allocation-1500.test.ts — fixture-shape assertions green today; the allocation assertions held as `it.fails` so the suite stays green while the bug is open and goes RED the moment it is fixed. Also documented and deliberately left unfixed: runPayrollCycleAll's `s.store.Upsert` edge resolves to the GENERATED Store.Upsert, not the hand-written one — same-name method resolution across two packages picks the wrong receiver. It is upstream of the allocation bug, so it belongs with CG-10's scoring work. Refs #1500 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
16e17495f4
commit
bd86ad2061
@@ -92,6 +92,84 @@ This is the gap the rest of the epic closes: relevance-proportional allocation w
|
||||
relative cliff (CG-12), on top of scoring that stops rewarding incidental name collisions
|
||||
(CG-10).
|
||||
|
||||
## The regression fixtures (CG-6)
|
||||
|
||||
Two fixtures pin the failure mode so it can never silently return. Both **fail today** —
|
||||
that is what they are for. They become the pass gate for CG-10 + CG-12.
|
||||
|
||||
They are declared in `scripts/agent-eval/allocation-fixtures.json` and run by
|
||||
`scripts/agent-eval/probe-allocation.mjs`, which drives the CG-4 diagnostic through a JSONL
|
||||
sidecar (so it measures the shipping allocator, not a re-derivation), groups the rendered
|
||||
files into `answer` vs `incidental`, and checks declared share thresholds. Needs a current
|
||||
`npm run build`; exits 1 while any assertion fails.
|
||||
|
||||
```bash
|
||||
node scripts/agent-eval/probe-allocation.mjs # both
|
||||
node scripts/agent-eval/probe-allocation.mjs payroll-go # one
|
||||
node scripts/agent-eval/probe-allocation.mjs --json # machine-readable
|
||||
```
|
||||
|
||||
### 1. `payroll-go` — the reporter's shape
|
||||
|
||||
`__tests__/fixtures/payroll-go/` is a synthetic Go service: generated FKIT CRUD beside a
|
||||
hand-written payroll use-case, entered from an HTTP route. Full description in that
|
||||
directory's README. The essentials:
|
||||
|
||||
- Generated files with **ordinary names** carrying `// Code generated ... DO NOT EDIT.` —
|
||||
invisible to path-only detection, which is what makes this a #1500 fixture rather than a
|
||||
`.pb.go` one — beside `payrollpb/*.pb.go` covering the path-detectable channel.
|
||||
- Deliberate collisions: `BuildPayslip`, `Upsert` and `Store` each exist twice, generated
|
||||
and hand-written, and the generated layer name-collides on every query term.
|
||||
- `cycle.go` (227 lines) sits above the whole-file window so it clips; the generated files
|
||||
sit below it so they ship whole.
|
||||
|
||||
Query — an architecture question naming none of the answering symbols: *"how does payroll
|
||||
cycle create and calculate payslips?"*
|
||||
|
||||
| | allocated | delivered |
|
||||
|---|---|---|
|
||||
| hand-written | 48.4% | **25.6%** (all of it domain types) |
|
||||
| generated CRUD | 39.9% | **57.4%** |
|
||||
|
||||
`cycle.go` is allocated the single largest slice (7,052 chars, 30.6%) and delivers **zero**
|
||||
— the 19,500 hard ceiling drops its whole section. `payslip_builder.go` (rank #8) never
|
||||
renders. So `runPayrollCycleAll`, the hand-written `BuildPayslip` and the real `Upsert`
|
||||
never reach the agent, and every byte that did arrive describes either CRUD or types.
|
||||
|
||||
This fixture is hermetic: the probe copies the tree to a temp dir and re-indexes per run,
|
||||
so two runs on one build are byte-identical (verified). `__tests__/explore-allocation-1500.test.ts`
|
||||
runs the same assertions in vitest — the fixture-shape half green, the allocation half as
|
||||
`it.fails` so the suite stays green while the bug is open and goes **red when it is fixed**.
|
||||
|
||||
**Finding, deliberately left unfixed:** `runPayrollCycleAll` calls `s.store.Upsert` on a
|
||||
`*payslipstore.Store`, but the graph resolves that edge to the **generated**
|
||||
`internal/gen/fkit/payroll/store.go` `Store.Upsert`. Same-name method resolution across two
|
||||
packages that both define `Store.Upsert` picks the wrong receiver. It is upstream of the
|
||||
allocation bug — a wrong edge pulls the generated store into the subgraph and inflates its
|
||||
score — so it belongs with CG-10's scoring work, not with the fixture.
|
||||
|
||||
### 2. `self-query` — the same bug with no generated code in sight
|
||||
|
||||
The baseline above, promoted to a fixture: this repo, *"how does explore allocate its output
|
||||
budget across files"*. `scripts/agent-eval/*.mjs` mention `explore` and `BUDGET`
|
||||
incidentally — they are eval harnesses, not the allocator — and they are small enough to
|
||||
ship whole, while `src/mcp/tools.ts` is large enough to be clipped.
|
||||
|
||||
At 493 indexed files (small tier): the script corpus takes **71.8%** of the delivered
|
||||
envelope (79.4% allocated) against `tools.ts`'s **18.5%**, despite `tools.ts` scoring 46 vs
|
||||
10, carrying 2.3× the graph mass and 3× the distinct term hits.
|
||||
|
||||
This fixture reads the **live** index of this repo, so unlike `payroll-go` its exact numbers
|
||||
move as the repo changes. Its assertions are relative for that reason (answer group vs
|
||||
incidental group, largest delivered file), never fixed percentages. Two things to know:
|
||||
|
||||
- The `<500`-file tier boundary is close. This repo indexes 493 files including the new
|
||||
fixture; crossing 500 flips `maxOutputChars` 18,000 → 24,000, `maxFiles` 5 → 8 and
|
||||
`maxCharsPerFile` 3,800 → 6,500, which moves every number in the table above. Re-baseline
|
||||
after the crossing rather than treating the drift as a regression.
|
||||
- Adding the `payroll-go` fixture itself moved the count 472 → 493. Its Go files match none
|
||||
of this query's terms, so they change the tier arithmetic and nothing else.
|
||||
|
||||
### Reproducing
|
||||
|
||||
The query explores this repo, so **uncommitted edits to `src/mcp/tools.ts` change the
|
||||
|
||||
Reference in New Issue
Block a user