test(explore): #1500 regression fixtures for budget allocation (CG-6)
Two permanent fixtures pinning the failure mode from issue #1500 — explore spending its byte envelope on files that merely name-collide with the query. BOTH FAIL TODAY, by design: they document the bug and become the pass gate for CG-10 (scoring) + CG-12 (proportional allocation). __tests__/fixtures/payroll-go/ — a synthetic Go service mirroring the reporter's shape: generated FKIT CRUD beside a hand-written payroll use-case, entered from an HTTP route. Half the generated tree carries ORDINARY names detectable only by their `// Code generated ... DO NOT EDIT.` header (the #1500 case, and end-to-end cover for CG-5); `payrollpb/*.pb.go` covers the path-detectable channel. BuildPayslip, Upsert and Store each exist twice, generated and hand-written. cycle.go sits above the whole-file window so it clips; the generated files sit below it so they ship whole. Asking "how does payroll cycle create and calculate payslips?" — naming none of the answering symbols — the generated CRUD delivers 57.4% of the envelope against the hand-written layer's 25.6%, all of the latter domain types. cycle.go is allocated the single largest slice (30.6%) and delivers ZERO: the hard ceiling drops its whole section. runPayrollCycleAll, the hand-written BuildPayslip and the real Upsert never reach the agent. The second fixture is this repo, "how does explore allocate its output budget across files", where scripts/agent-eval/*.mjs take 71.8% against tools.ts's 18.5% despite scoring 4.6x lower. It reads the live index, so its assertions are relative rather than fixed percentages. - scripts/agent-eval/probe-allocation.mjs — per-file budget-share probe, driving the CG-4 diagnostic through a JSONL sidecar so it measures the shipping allocator. Fixture entries are hermetic (copy + re-index per run, verified byte-identical across runs); exits 1 while any assertion fails. - scripts/agent-eval/allocation-fixtures.json — both fixtures declared, with the 2026-08-03 baselines. - __tests__/explore-allocation-1500.test.ts — fixture-shape assertions green today; the allocation assertions held as `it.fails` so the suite stays green while the bug is open and goes RED the moment it is fixed. Also documented and deliberately left unfixed: runPayrollCycleAll's `s.store.Upsert` edge resolves to the GENERATED Store.Upsert, not the hand-written one — same-name method resolution across two packages picks the wrong receiver. It is upstream of the allocation bug, so it belongs with CG-10's scoring work. Refs #1500 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
co-authored by
Claude Opus 5
Claude Opus 5
parent
16e17495f4
commit
bd86ad2061
@@ -0,0 +1,95 @@
|
||||
# payroll-go — the #1500 regression fixture
|
||||
|
||||
A synthetic Go service reproducing the repo shape from [issue #1500](https://github.com/colbymchenry/codegraph/issues/1500):
|
||||
**generated CRUD sitting beside the hand-written use-case that does the real work.**
|
||||
|
||||
This tree is a fixture, not a program. It never compiles or runs — it exists to be
|
||||
indexed. Keep it valid, idiomatic Go anyway: the extractor's output is the whole point.
|
||||
|
||||
## The shape
|
||||
|
||||
```
|
||||
cmd/payrolld/main.go wires the service
|
||||
internal/transport/httpapi/ HTTP entry point → use-case
|
||||
internal/usecase/payroll/ ← THE ANSWER. Hand-written workflow:
|
||||
cycle.go runPayrollCycleAll (227 lines)
|
||||
payslip_builder.go BuildPayslip — the actual pay calculation
|
||||
prorate.go
|
||||
internal/domain/payroll/payslip.go hand-written domain types
|
||||
internal/store/payslipstore/store.go the real Upsert
|
||||
internal/platform/clock/clock.go
|
||||
|
||||
internal/gen/fkit/payroll/ ← THE NOISE. Generated CRUD, ORDINARY names:
|
||||
payslip.go CreatePayslip, GetPayslip, UpdatePayslip, a second BuildPayslip
|
||||
payroll_cycle.go CreatePayrollCycle, PayrollCycleCreateRequest, …
|
||||
store.go a second Upsert
|
||||
calculate.go CalculatePayrollCycleTotals, CalculatePayslipNet, …
|
||||
dto.go
|
||||
internal/gen/fkit/employee/, timesheet/ more generated CRUD
|
||||
internal/gen/payrollpb/*.pb.go generated, detectable by PATH
|
||||
```
|
||||
|
||||
The chain the fixture is built around is `runPayrollCycleAll` → `BuildPayslip` → `Upsert`,
|
||||
entered from `POST /v1/payroll/cycles/{cycleID}/run`.
|
||||
|
||||
## The three properties that make it a regression fixture
|
||||
|
||||
1. **Generated files that only a CONTENT header betrays.** The `internal/gen/fkit/**`
|
||||
files have ordinary names (`payslip.go`, `store.go`) and carry
|
||||
`// Code generated by fkit v3.11.0. DO NOT EDIT.`. Path-only detection misses every
|
||||
one of them — that is the #1500 case, and why CG-5 added the content check. The
|
||||
`payrollpb/*.pb.go` files cover the path-detectable channel beside them.
|
||||
|
||||
2. **Deliberate name collisions.** `BuildPayslip`, `Upsert` and `Store` each exist twice,
|
||||
once generated and once hand-written. The generated layer also name-collides on every
|
||||
term of the question below — `CreatePayslip`, `PayrollCycleCreateRequest`,
|
||||
`CalculatePayrollCycleTotals` — so a scorer that rewards incidental name matches
|
||||
surfaces the CRUD path.
|
||||
|
||||
3. **A size split that drives the render mode.** `cycle.go` is deliberately over the
|
||||
whole-file window (227 lines) so it falls through to clipped clusters; the generated
|
||||
files are deliberately under it so they ship whole. Allocation follows file size, not
|
||||
relevance. `__tests__/explore-allocation-1500.test.ts` pins both sides of that split —
|
||||
if you edit these files, keep it.
|
||||
|
||||
## The assertion
|
||||
|
||||
Query: **"how does payroll cycle create and calculate payslips?"** — an architecture
|
||||
question that names none of the symbols that answer it. The budget should concentrate on
|
||||
the hand-written workflow. As of 2026-08-03 it does not:
|
||||
|
||||
| | allocated | delivered |
|
||||
|---|---|---|
|
||||
| hand-written | 48.4% | **25.6%** (all of it domain types) |
|
||||
| generated CRUD | 39.9% | **57.4%** |
|
||||
|
||||
`cycle.go` is allocated the single largest slice (7,052 chars, 30.6%) and delivers
|
||||
**zero** — the hard ceiling drops its whole section. `payslip_builder.go` (rank #8) never
|
||||
renders at all. `runPayrollCycleAll`, the hand-written `BuildPayslip` and the real
|
||||
`Upsert` never reach the agent.
|
||||
|
||||
## Running it
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
node scripts/agent-eval/probe-allocation.mjs payroll-go # exits 1 today, by design
|
||||
npx vitest run __tests__/explore-allocation-1500.test.ts # green today, by design
|
||||
```
|
||||
|
||||
The probe reports per-file budget share against `scripts/agent-eval/allocation-fixtures.json`.
|
||||
The vitest suite pins the fixture's shape and holds the allocation assertion as `it.fails` —
|
||||
green while the bug is open, red the moment it is fixed. See
|
||||
`docs/design/explore-budget-allocation.md`.
|
||||
|
||||
## Known finding: the chain's `Upsert` edge resolves to the generated store
|
||||
|
||||
`runPayrollCycleAll` calls `s.store.Upsert(ctx, slip)`, where `s.store` is a
|
||||
`*payslipstore.Store`. The graph resolves that edge to `internal/gen/fkit/payroll/store.go`
|
||||
— the **generated** `Store.Upsert` — not the hand-written one. Same-name method resolution
|
||||
across two packages that both define `Store.Upsert` picks the wrong receiver.
|
||||
|
||||
This is a resolution defect, not a budget one, and it is left unfixed on purpose: it is
|
||||
upstream of the allocation bug (a wrong edge pulls the generated store into the subgraph
|
||||
and inflates its score), so it belongs with the scoring work in CG-10 rather than here.
|
||||
The test asserts only that the workflow reaches *an* `Upsert`, so tightening the resolver
|
||||
later will not break the fixture.
|
||||
Reference in New Issue
Block a user