Coverage for the CG-12 allocator, built around "would this go red if the lever were removed" rather than line coverage — every way this regresses is silent, ending in an agent falling back to Read. Unit (`explore-proportional-allocation.test.ts`, 18 -> 38): calibration pins, envelope safety across every tier and 30 candidate shapes, the cliff boundary, spine weighting/trim survival, the diffuse control, and the degenerate inputs — identical scores, a lone file, a runaway top scorer, zero results, maxFiles 0, a non-finite score. End-to-end (`explore-allocation-e2e.test.ts`, new): CG-6's second regression fixture as a deterministic synthetic mirror — a large relevant file, a small helper that used to win by shipping whole, and an incidental `explore`/`BUDGET` collision — asserting per-file budget share, not file presence. Plus degenerate result sets and a survey-style diffuse control through the real render loop. The live self-query arm stays in probe-allocation.mjs, where drift is a number to re-baseline rather than a red suite. Reverting the render loop to the pre-CG-12 rules reproduces #1500 on the mirror exactly and takes 5 e2e + 2 payroll gates red: file score pre-CG-12 CG-12 src/mcp/allocator.ts 77.5 4,843 (39.7%) 9,335 (80.1%) src/util/budget-math.ts 36.0 6,079 (49.8%) 1,037 ( 8.9%) Two defects the invariants surfaced, both fixed in tools.ts: - rounded shares could sum past `pool`, so "reservations fit the envelope" was approximate rather than exact; both terms now floor - a non-finite score made every share Infinity/Infinity, handing the render loop a NaN allowance; `weightOf` now fails safe to 0 Also adds a hard-ceiling gate to the payroll fixture — at 19.3K against a 19.5K ceiling it is the only fixture that stresses the ~25K inline cap — and exports EXPLORE_ALLOCATION so invariant tests read the constants while one test pins the literals. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
449 lines
26 KiB
Markdown
449 lines
26 KiB
Markdown
# Explore budget allocation — the instrument and the baseline
|
||
|
||
`codegraph_explore` has a fixed byte envelope (`getExploreOutputBudget().maxOutputChars`,
|
||
hard-capped at 25K so the host never externalizes the result). **How that envelope gets
|
||
divided among files** is decided by a long chain of gates, tiers and caps spread across
|
||
`handleExplore` — and until CG-4 that chain was unobservable. You could read an explore
|
||
response and guess; you could not say "this file took 16% and that one took 20%."
|
||
|
||
This document covers the diagnostic that makes it measurable, and the baseline it recorded.
|
||
|
||
## The diagnostic
|
||
|
||
Set `CODEGRAPH_EXPLORE_DEBUG` and every `codegraph_explore` call (MCP tool or
|
||
`codegraph explore` CLI) emits one report:
|
||
|
||
| value | sink |
|
||
|---|---|
|
||
| `1` / `true` / `on` / `yes` / `stderr` | human-readable table on stderr |
|
||
| `json` | one pretty-printed JSON report on stderr |
|
||
| anything else | treated as a path — one JSON report per line, appended (JSONL) |
|
||
| unset / `0` / `false` / `off` / `no` / empty | **off** |
|
||
|
||
Per file it reports: relevance score, graph (RWR) mass, distinct query-term hits, ranking
|
||
flags (named / entry / central / spine / low-value / generated), render mode, bytes of
|
||
source allocated, bytes actually delivered, both shares, and whether it was clipped. For
|
||
files that never rendered it reports why (`max-files`, `budget-90pct`, `budget-whole-file`,
|
||
`budget-clusters`, `unreadable`, `no-ranges`). Totals cover the envelope (delivered vs
|
||
allocated vs `maxOutputChars` vs hard ceiling), the source/meta split, the file-selection
|
||
funnel at each stage, and the thresholds applied (score floor, graph-relevance gate).
|
||
|
||
**It is off by default and produces byte-identical output when off** — it ships in the
|
||
product binary, and a diagnostic that perturbs the response by one byte would invalidate
|
||
every A/B measurement taken with it on. `ExploreDiagnostics.start()` returns `null` unless
|
||
the env var is set, so every call site is a `diag?.` no-op. Pinned by
|
||
`__tests__/explore-diagnostics.test.ts`.
|
||
|
||
Two envelope numbers, deliberately kept separate:
|
||
|
||
- **allocated** — what the render loop chose to emit, before the final hard-ceiling cut.
|
||
This is the allocator's own decision, and the number budget work is about.
|
||
- **delivered** — what the agent actually received.
|
||
|
||
They diverge exactly when the ceiling truncates. Conflating them is how a dropped trailing
|
||
file goes unnoticed.
|
||
|
||
## Baseline (2026-08-03, this repo at `main`)
|
||
|
||
```
|
||
codegraph explore "how does explore allocate its output budget across files" --path .
|
||
```
|
||
|
||
469 files indexed → small tier (`maxOutputChars` 18,000, `maxCharsPerFile` 3,800,
|
||
`defaultMaxFiles` 5). Envelope: **23,196 delivered / 23,193 allocated against an 18,000
|
||
budget — 29% over**, absorbed only because the 25K hard ceiling sits above it.
|
||
|
||
| # | share | bytes | score | graph | hits | flags | render | file |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| 4 | 21.2% | 4,928 | 10 | 0.125 | 1 | entry | whole | `scripts/agent-eval/offload-eval-hook.mjs` |
|
||
| 5 | 20.1% | 4,665 | 10 | 0.125 | 1 | entry | whole | `scripts/agent-eval/offload-eval-metrics.mjs` |
|
||
| 3 | 19.8% | 4,585 | 22 | 0.125 | 1 | entry central | whole | `scripts/agent-eval/parse-session.mjs` |
|
||
| 1 | **15.8%** | 3,659 | **54** | **0.322** | **4** | entry central | clusters* | `src/mcp/tools.ts` |
|
||
| 2 | 15.0% | 3,479 | 34 | 0.082 | 2 | entry | clusters* | `src/index.ts` |
|
||
|
||
\* clipped. Ranked but never rendered: `scripts/agent-eval/offload-eval-cost.mjs` (#6) and
|
||
`src/resolution/lru-cache.ts` (#7), both cut by `maxFiles`.
|
||
|
||
Files: 17 grouped → 10 past the score floor (≥3) → 10 past the low-value filter → 7 past
|
||
the relevance gate (graph ≥ 0.0193, 6% of max 0.3215) → 5 in the output.
|
||
|
||
### What the baseline shows
|
||
|
||
**Score does not drive allocation.** `src/mcp/tools.ts` — the file that actually answers
|
||
the query — carries 5.4× the relevance score, 2.6× the graph mass and 4× the term hits of
|
||
any `.mjs` script, and gets a *smaller* share than each of them. The three agent-eval
|
||
scripts take **61%** of the envelope between them; the answer file takes 16%.
|
||
|
||
The mechanism is that the two allocation paths are decided by **file size, not relevance**:
|
||
a small file clears `WHOLE_FILE_MAX_LINES`/`WHOLE_FILE_MAX_CHARS` and ships entirely, while
|
||
a large file falls through to cluster selection and is clipped at `maxCharsPerFile`. So a
|
||
weakly-relevant 130-line script gets 100% of itself; the strongly-relevant 5,000-line file
|
||
gets 3,800 chars. Rank ordering is correct (tools.ts sorts #1) and buys nothing, because
|
||
rank has no effect on how many bytes a file receives.
|
||
|
||
**The envelope is over-subscribed.** 23,193 allocated against an 18,000 budget means the
|
||
per-file caps do not compose into the total cap; the total is enforced only by the 25K
|
||
ceiling silently dropping whole trailing sections. Under a slightly different index state
|
||
(one more candidate file) the same query allocated 27,518 chars and the ceiling dropped a
|
||
7,678-char section — the single largest allocation in the response — with the only trace
|
||
being the truncation notice at the end.
|
||
|
||
This is the gap the rest of the epic closes: relevance-proportional allocation with a
|
||
relative cliff (CG-12), on top of scoring that stops rewarding incidental name collisions
|
||
(CG-10 — landed; see below).
|
||
|
||
## CG-10 — relevance scoring
|
||
|
||
CG-10 changes **what gets into the response**, ahead of how bytes are split among what's
|
||
in. Four levers, all multiplicative so they compose without ordering surprises.
|
||
|
||
### 1. Kind weighting
|
||
|
||
The tier a symbol reached us by (named seed `+50`, query match `+10`, adjacent to one `+3`,
|
||
peripheral `+1`) says *how it got here*; `RELEVANCE_KIND_WEIGHT` says *whether the match is
|
||
evidence*. Callables and types weigh 1.0, members ~0.5, and `constant`/`variable`/
|
||
`parameter` 0.15–0.35 — a local named `explore` is a name collision until something
|
||
corroborates it.
|
||
|
||
**Isolation.** For a weak-kind symbol in the top two tiers, "is anything using it?" is the
|
||
corroboration: no usage edge anywhere in the graph (`contains` excluded — lexical nesting
|
||
is not usage) drops it to 0.08. Cost is bounded — only weak kinds in the tiers whose weight
|
||
can carry a file pay for the probe, and the subgraph's own edges answer most cases for
|
||
free. Measured: no latency change (210 vs 211 ms/call, n=12 interleaved).
|
||
|
||
**Peripheral cap.** Nodes ≥2 hops from any match now accumulate into a separate bucket
|
||
capped at 5. Uncapped, every such node added a flat `+1`, so a file grew more relevant by
|
||
being bigger — `parse-session.mjs` reached score 22 off one incidental constant plus twelve
|
||
unrelated symbols. Size is not evidence.
|
||
|
||
### 2. Relative score floor
|
||
|
||
`score >= 3` admits noise on any repo where the top file scores 50+. The floor is now
|
||
`clamp(topScore × 0.2, 1, 10)`:
|
||
|
||
- **relative** — on a diffuse question no file dominates, every candidate sits near the top,
|
||
and the whole spread survives; on a precise one it cuts the tail.
|
||
- **capped at 10** — one direct query match on a callable. A single full-strength match is
|
||
never incidental, so no amount of concentration elsewhere may exclude it. Without this
|
||
cap, one named-seed-heavy file pushed the floor to 21 and dropped a file the agent had
|
||
named by *class* name (classes enter at `+10`, not `+50` — named seeds are callables).
|
||
- **backfill** — if fewer than 3 files survive, the best of what the floor cut comes back,
|
||
but only from files with real evidence (≥ the absolute floor). If *nothing* survives, the
|
||
backfill drops that requirement: returning "no relevant code found" when the gather did
|
||
find candidates sends the agent straight back to grep.
|
||
|
||
### 3. Generated status in the score, not the tiebreak
|
||
|
||
`rankPenalty(file)` multiplies both the relevance score and the graph mass by 0.3 for
|
||
generated files (0.5 for low-value ones). Applying it to the score alone would not have
|
||
fixed #1500: the generated CRUD carries **more** graph mass than the hand-written use-case,
|
||
and graph mass outranks score in the comparator. The penalty is self-normalizing — in an
|
||
all-generated repo everything scales together and relative ranking is untouched — and it
|
||
never hard-excludes: ask about the generated API by name and the named-seed tier still puts
|
||
it first.
|
||
|
||
### 4. `excludeLowValueFiles` — the finding
|
||
|
||
The per-tier flag the task asked to reconsider was **dead config**: declared on
|
||
`ExploreOutputBudget` and set per tier, but read nowhere. A later change had already made
|
||
the test/spec/icon/i18n exclusion unconditional at all tiers. The flag is removed.
|
||
|
||
The substantive gap was in the *detector*, not the gating: `isLowValue` matched
|
||
`/\/(tests?|__tests?__|spec)\//`, anchored on a leading slash, so a **repo-root** `test/`
|
||
directory — express, cobra, and most of npm and Go — never matched. Express's "how does
|
||
express route a request to a handler?" spent 59% of its envelope on three test files while
|
||
`lib/application.js` was clipped. Anchored at `^` as well, that query now returns
|
||
`lib/application.js` + `lib/response.js` and no tests.
|
||
|
||
Two related changes: the filter now runs **before** the score floor and judges "are there
|
||
other candidates?" on the whole gather rather than the post-floor set (judging it after was
|
||
how the floor's keep-minimum pulled test files back in as the "spread"); and low-value
|
||
files that survive the filter's `≥2 non-test candidates` escape hatch are down-weighted via
|
||
`rankPenalty` rather than left at full strength.
|
||
|
||
### Measured effect
|
||
|
||
Before/after on the same indexes, deterministic (`CODEGRAPH_EXPLORE_DEBUG` diagnostic, both
|
||
arms same build system, baseline = `bd86ad2`):
|
||
|
||
| repo · query | before | after |
|
||
|---|---|---|
|
||
| this repo · self-query fixture | 72% to eval scripts, `tools.ts` 18.5% | scripts **0%**, `tools.ts` #1 |
|
||
| this repo · `handleExplore buildFlowFromNamedSymbols …` | 82% to eval scripts | `tools.ts` 48% + `index.ts` 32% |
|
||
| this repo · "how is error handling done" | 58% to eval scripts, `tools.ts` delivered 0 | transport/tools/cobol/api |
|
||
| this repo · "what languages does codegraph support" | 63% to `scripts/add-lang/*` | grammars/index/cli |
|
||
| this repo · "main components of the indexing pipeline" | — | **byte-identical** |
|
||
| payroll-go fixture | generated 57.4%, answer 25.6% | answer **61.5%**, generated **23.5%** |
|
||
| express · route a request | 59% to `test/*` | `application.js` + `response.js` |
|
||
| cobra · 3 queries | — | **byte-identical** |
|
||
|
||
The two byte-identical rows are the control: where the answer was already concentrated, the
|
||
new floor prunes the same tail earlier and cheaper and arrives at the same response.
|
||
|
||
**Known thin case.** Express's "how does the app object get created and what does it
|
||
expose" drops from 4 files (top one an `examples/` file at 38%) to `lib/express.js` alone,
|
||
2.6 KB against a 13 KB budget. `lib/application.js` matched on nothing but an unused
|
||
file-scope `var app` — indistinguishable, at the symbol level, from the eval scripts' unused
|
||
`const explore`; express models its API surface as properties assigned to that object,
|
||
which the graph has no edges for. That is extraction coverage, not ranking. Backfilling it
|
||
was tried and rejected: node-count ties handed the slot to `examples/route-middleware`
|
||
instead, at 48% of the envelope. Thin-and-precise beats padded-with-noise — a wrong file
|
||
does not save the agent the follow-up call it would pad against.
|
||
|
||
## CG-12 — score-proportional allocation
|
||
|
||
CG-10 fixed *what* gets into the response. This fixes *how the bytes are split among what
|
||
got in* — which, until now, was not really decided at all. Every admitted file was capped at
|
||
the same flat `maxCharsPerFile`, and the whole-file rule handed anything under
|
||
`maxCharsPerFile × 3` its entire contents. So the envelope followed **file size**:
|
||
|
||
- self-query: `memory-budget.ts` (score 18) shipped whole at 5,672 and took **51.2%**;
|
||
`tools.ts` (score 41, 4× the graph mass, 3× the term hits — it literally holds the
|
||
allocator) was clipped at 3,800 and got **32.9%**.
|
||
- payroll-go: two generated CRUD files shipped whole at ~4.5 KB each *and* consumed two of
|
||
the tier's four file slots, so `BuildPayslip` — the hand-written "calculate" half of the
|
||
question — ranked #6 and never rendered at all.
|
||
|
||
### The model
|
||
|
||
`allocateExploreBudget` (`src/mcp/tools.ts`) runs once, after ranking, before anything
|
||
renders. It reserves each file a share of the envelope; the render loop then spends a
|
||
reservation instead of racing for whatever the files above it left.
|
||
|
||
1. **Weight** = `score × worth × (spine ? 2 : 1)`. `worth` is `rankPenalty` applied a
|
||
*second* time: ranking answers "is this file about the query", allocation answers "will
|
||
these bytes teach the agent anything". Generated CRUD can legitimately rank — it
|
||
name-collides on every domain word, and it is big and densely self-referential, so it
|
||
scores on the structural keys the comparator leads with — while its bytes stay mechanical
|
||
boilerplate. That second penalty is what finally sinks it.
|
||
2. **Relative cliff** at 15% of the top weight, itself capped at `SCORE_FLOOR_MAX`. A file
|
||
under it gets **zero source** — path, symbols and line numbers only. It costs ~100 chars
|
||
instead of ~4,500, and it does **not consume a `maxFiles` slot**, so the slot passes to a
|
||
file that earns its bytes. That slot hand-off is what got `BuildPayslip` into the
|
||
response. The cap matters as much as the fraction: one 500-scoring god-file would
|
||
otherwise put the cliff at 75 and silence every peer the score floor had just admitted.
|
||
3. **Floor then split.** Every admitted file gets `MIN_CHARS` (700 — enough for one complete
|
||
method); the remainder splits by weight. The floor is what keeps a diffuse survey
|
||
question returning a spread; the remainder is what concentrates a precise one.
|
||
4. **Safety valve**, not a per-file cap: no file exceeds 70% of the envelope. The flat
|
||
per-file cap is retired as the primary guard — the proportional split already bounds a
|
||
file by its weight share.
|
||
|
||
The reservation then governs **every** render path — whole-file, clusters, focused/skeleton
|
||
— where before the whole-file branch was 3× more generous than the cluster branch, which is
|
||
the 3× swing that decided the split by file size.
|
||
|
||
Two supporting changes were needed to make the reservation actually bite:
|
||
|
||
- **Oversize clusters shrink by member.** A cluster is a merge of whole symbol ranges, and
|
||
on a densely-packed file every symbol merges into one blob spanning the file (cycle.go's
|
||
209-line `Service`). The old rule took the top-ranked cluster whole however big it was, so
|
||
a single-cluster file simply ignored its budget — it took ~40% more than allotted and the
|
||
file below it was then dropped for lack of room. Shrinking drops whole **members** by
|
||
importance, so a body is still never cut.
|
||
- **The arrival-order stops are gone.** `budget-90pct` and the `!fileNecessary && totalChars
|
||
> maxOutputChars` checks dropped files by the order they were reached: whichever files
|
||
ranked first spent the envelope, and everything after them was cut on a cap it had no say
|
||
in. Only an absolute hard-ceiling stop remains.
|
||
|
||
### Measured effect (CG-12)
|
||
|
||
| repo · query | before (post-CG-10) | after |
|
||
|---|---|---|
|
||
| this repo · self-query fixture | `tools.ts` 32.9%, `memory-budget.ts` 51.2% | **`tools.ts` 60.6%**, memory-budget 17.2% |
|
||
| payroll-go fixture | answer 61.5%, generated 23.5%, `BuildPayslip` absent | answer **78.7%**, generated **0%**, `BuildPayslip` **delivered** |
|
||
| express · route a request | 2 files, top 43.1% | 1 file, top **82.3%** (`response.js` cliffed — 0 term hits) |
|
||
| express · app registers middleware | 1 file, 71.6% | **byte-identical** |
|
||
| cobra · parse flags and execute | 2 files, `command.go` 40.1% | 2 files, `command.go` **77.2%** |
|
||
| cobra · DIFFUSE "main components" | 3 files, top 48.1% | 3 files, top 50.0% — spread preserved |
|
||
| gin · request reaches a handler | 3 files, top `ginS/gins.go` 48.8% | 3 files, top **`routergroup.go`** 53.8% |
|
||
| gin · DIFFUSE "what it provides" | 3 files, top `recovery.go` 43.0% | **4 files**, top `context.go` 33.3% |
|
||
|
||
The two diffuse rows are the over-correction control: file counts hold (3→3, 3→4), so a
|
||
survey question still gets a spread. The two gin rows also moved the top file to a more apt
|
||
one — `routergroup.go` over the thin `ginS` singleton wrapper, `context.go` over
|
||
`recovery.go` — because concentration is decided by weight rather than by which file
|
||
happened to be small.
|
||
|
||
**Exception to "no previously-unclipped file becomes clipped".** `memory-budget.ts` was
|
||
unclipped-whole at 5,672 and now clusters within its 3.1 KB reservation. That is the epic's
|
||
own diagnosis of the bug rather than a regression: it scored 18 against `tools.ts`'s 58 and
|
||
was taking the larger slice purely for being small enough to ship whole. The guarantee holds
|
||
where it was meant to — no file loses bytes to a *tighter cap*; the only files that lose are
|
||
ones the proportional split says were over-served.
|
||
|
||
## The regression fixtures (CG-6)
|
||
|
||
Two fixtures pin the failure mode so it can never silently return. They were written to
|
||
**fail** — that is what they were for. CG-10 closed the ranking half of both and CG-12 the
|
||
byte-split half; **both now pass** and are live regressions. The numbers quoted below are
|
||
the **pre-CG-10 baseline**; see the two "Measured effect" tables above for where they stand.
|
||
|
||
They are declared in `scripts/agent-eval/allocation-fixtures.json` and run by
|
||
`scripts/agent-eval/probe-allocation.mjs`, which drives the CG-4 diagnostic through a JSONL
|
||
sidecar (so it measures the shipping allocator, not a re-derivation), groups the rendered
|
||
files into `answer` vs `incidental`, and checks declared share thresholds. Needs a current
|
||
`npm run build`; exits 1 while any assertion fails.
|
||
|
||
```bash
|
||
node scripts/agent-eval/probe-allocation.mjs # both
|
||
node scripts/agent-eval/probe-allocation.mjs payroll-go # one
|
||
node scripts/agent-eval/probe-allocation.mjs --json # machine-readable
|
||
```
|
||
|
||
### 1. `payroll-go` — the reporter's shape
|
||
|
||
`__tests__/fixtures/payroll-go/` is a synthetic Go service: generated FKIT CRUD beside a
|
||
hand-written payroll use-case, entered from an HTTP route. Full description in that
|
||
directory's README. The essentials:
|
||
|
||
- Generated files with **ordinary names** carrying `// Code generated ... DO NOT EDIT.` —
|
||
invisible to path-only detection, which is what makes this a #1500 fixture rather than a
|
||
`.pb.go` one — beside `payrollpb/*.pb.go` covering the path-detectable channel.
|
||
- Deliberate collisions: `BuildPayslip`, `Upsert` and `Store` each exist twice, generated
|
||
and hand-written, and the generated layer name-collides on every query term.
|
||
- `cycle.go` (227 lines) sits above the whole-file window so it clips; the generated files
|
||
sit below it so they ship whole.
|
||
|
||
Query — an architecture question naming none of the answering symbols: *"how does payroll
|
||
cycle create and calculate payslips?"*
|
||
|
||
| | allocated | delivered |
|
||
|---|---|---|
|
||
| hand-written | 48.4% | **25.6%** (all of it domain types) |
|
||
| generated CRUD | 39.9% | **57.4%** |
|
||
|
||
`cycle.go` is allocated the single largest slice (7,052 chars, 30.6%) and delivers **zero**
|
||
— the 19,500 hard ceiling drops its whole section. `payslip_builder.go` (rank #8) never
|
||
renders. So `runPayrollCycleAll`, the hand-written `BuildPayslip` and the real `Upsert`
|
||
never reach the agent, and every byte that did arrive describes either CRUD or types.
|
||
|
||
This fixture is hermetic: the probe copies the tree to a temp dir and re-indexes per run,
|
||
so two runs on one build are byte-identical (verified). `__tests__/explore-allocation-1500.test.ts`
|
||
runs the same assertions in vitest.
|
||
|
||
**After CG-10** the generated files rank #3/#4 instead of #1/#2, `cycle.go` delivers 38.9%
|
||
(it delivered nothing), and `runPayrollCycleAll` + the real `s.store.Upsert(ctx, slip)`
|
||
reach the agent. Those assertions are now live regressions. What remains `it.fails` is
|
||
`payslip_builder.go`: it ranks #6, the tier's `maxFiles` is 4, and the render loop still
|
||
spends by file size — CG-12's job.
|
||
|
||
**Finding, deliberately left unfixed:** `runPayrollCycleAll` calls `s.store.Upsert` on a
|
||
`*payslipstore.Store`, but the graph resolves that edge to the **generated**
|
||
`internal/gen/fkit/payroll/store.go` `Store.Upsert`. Same-name method resolution across two
|
||
packages that both define `Store.Upsert` picks the wrong receiver. It is upstream of
|
||
allocation — a wrong edge pulls the generated store into the subgraph. CG-10 mitigates the
|
||
*symptom* (the generated store is penalized on both score and graph mass, so it no longer
|
||
displaces the real one) without fixing the resolution bug itself, which belongs with the
|
||
same-name method resolution work (see `samename-method-resolution-1079`).
|
||
|
||
### 2. `self-query` — the same bug with no generated code in sight
|
||
|
||
The baseline above, promoted to a fixture: this repo, *"how does explore allocate its output
|
||
budget across files"*. `scripts/agent-eval/*.mjs` mention `explore` and `BUDGET`
|
||
incidentally — they are eval harnesses, not the allocator — and they are small enough to
|
||
ship whole, while `src/mcp/tools.ts` is large enough to be clipped.
|
||
|
||
At 493 indexed files (small tier): the script corpus takes **71.8%** of the delivered
|
||
envelope (79.4% allocated) against `tools.ts`'s **18.5%**, despite `tools.ts` scoring 46 vs
|
||
10, carrying 2.3× the graph mass and 3× the distinct term hits.
|
||
|
||
This fixture reads the **live** index of this repo, so unlike `payroll-go` its exact numbers
|
||
move as the repo changes. Its assertions are relative for that reason (answer group vs
|
||
incidental group, largest delivered file), never fixed percentages. Two things to know:
|
||
|
||
- The `<500`-file tier boundary is close. This repo indexes 493 files including the new
|
||
fixture; crossing 500 flips `maxOutputChars` 18,000 → 24,000, `maxFiles` 5 → 8 and
|
||
`maxCharsPerFile` 3,800 → 6,500, which moves every number in the table above. Re-baseline
|
||
after the crossing rather than treating the drift as a regression.
|
||
- Adding the `payroll-go` fixture itself moved the count 472 → 493. Its Go files match none
|
||
of this query's terms, so they change the tier arithmetic and nothing else.
|
||
|
||
### Reproducing
|
||
|
||
The query explores this repo, so **uncommitted edits to `src/mcp/tools.ts` change the
|
||
result** — the index picks them up and scores shift (the same query on the CG-4 working
|
||
tree reported tools.ts at 13–19% depending on the sync state). Measure against a clean
|
||
tree: restore `src/mcp/tools.ts` from `main`, remove `src/mcp/explore-diagnostics.ts`,
|
||
`codegraph sync`, then run the built `dist/` binary (which still carries the instrument).
|
||
Restore afterwards.
|
||
|
||
---
|
||
|
||
## CG-14 — locking the allocation down
|
||
|
||
The allocation change is one function plus three render-loop bounds, and every way it can
|
||
regress is silent: nothing throws, the response just gets less useful and the agent falls
|
||
back to Read. So the coverage is built around the question "would this test go red if the
|
||
lever were removed?" rather than around line coverage.
|
||
|
||
### Where the coverage lives
|
||
|
||
| File | Owns |
|
||
|---|---|
|
||
| `__tests__/explore-proportional-allocation.test.ts` | `allocateExploreBudget` in isolation — the split, the cliff, the tier invariant, envelope safety, spine weighting, degenerate inputs |
|
||
| `__tests__/explore-allocation-e2e.test.ts` | The same behaviours through the real render loop on real indexed projects — the self-query fixture's shape, degenerate result sets, the diffuse-query control |
|
||
| `__tests__/explore-allocation-1500.test.ts` | The reporter's Go shape (CG-6 fixture 1), plus the hard-ceiling stress case |
|
||
| `scripts/agent-eval/probe-allocation.mjs` | Both CG-6 fixtures against the built `dist/`, including the **live** self-query arm that reads this repo's own index |
|
||
|
||
The split between the last two is deliberate. The self-query fixture reads a moving target
|
||
(this repo), so its exact numbers drift with the tree and it belongs out of band, where a
|
||
drift is a number to re-baseline rather than a red suite. `npm test` owns a synthetic
|
||
**mirror** of it instead: same three roles, same size asymmetry, fixed.
|
||
|
||
### The two bounds are not the same bound
|
||
|
||
Worth stating once, because a test that conflates them looks right and passes for the wrong
|
||
reason:
|
||
|
||
- **`maxOutputChars` bounds the RESERVATIONS.** `sum(allowances) <= pool <= maxOutputChars`,
|
||
exactly, at every tier and every candidate shape.
|
||
- **`hardCeiling` — `min(maxOutputChars * 1.5, 25000)` — bounds the RESPONSE.** The render
|
||
loop is allowed a bounded overshoot (the whole-file grace, an oversize first cluster), so
|
||
a response legitimately exceeds the envelope. `payroll-go` does exactly that: 19.3K
|
||
delivered against a 13,000 envelope and a 19,500 ceiling.
|
||
|
||
Only the 25K is absolute. Above it the host writes the result to a file the agent Reads
|
||
back, which is the failure the tool exists to prevent.
|
||
|
||
### Mutation-tested, not just green
|
||
|
||
Each lever was removed from `src/mcp/tools.ts` in turn and the suite re-run. Every one is
|
||
covered by at least one failing test — a lever with no red test is a lever that can be
|
||
deleted by accident:
|
||
|
||
| Mutation | Tests that go red |
|
||
|---|---|
|
||
| Render loop reverted to pre-CG-12 (`fileBudget = maxCharsPerFile`, whole-file bound `maxCharsPerFile * 3`) | 5 e2e + 2 payroll |
|
||
| Proportional split replaced with an equal one | 4 unit |
|
||
| Cliff disabled (`cliffAt = 0`) | 7 unit + 3 payroll |
|
||
| Spine boost and cliff exemption removed | 3 unit |
|
||
| `MIN_CHARS` floor removed | 2 unit |
|
||
| `MAX_SHARE` ceiling removed | 4 unit |
|
||
|
||
The first row is the one that matters most: it reproduces #1500 on the synthetic fixture
|
||
verbatim, with the half-as-relevant file taking the larger share purely on size.
|
||
|
||
| file | score | pre-CG-12 | CG-12 |
|
||
|---|---|---|---|
|
||
| `src/mcp/allocator.ts` | 77.5 | 4,843 (39.7%) | 9,335 (80.1%) |
|
||
| `src/util/budget-math.ts` | 36.0 | 6,079 (49.8%) | 1,037 (8.9%) |
|
||
|
||
### Two defects the coverage surfaced
|
||
|
||
Both were found by writing the invariant rather than by reading the code:
|
||
|
||
1. **Rounded shares could exceed the pool.** `Math.round` on each file's proportional slice
|
||
let the reservations sum past `pool` by up to half a char per file — small, but it made
|
||
"reservations fit the envelope" false rather than exact. Both terms now floor.
|
||
2. **A non-finite score produced a NaN allowance.** An `Infinity` weight makes every share
|
||
`Infinity / Infinity`. Scores are finite sums in the pipeline so it was unreachable, but
|
||
the failure mode is a NaN handed to the render loop. `weightOf` now fails safe to 0.
|
||
|
||
### Calibration vs. invariant
|
||
|
||
`EXPLORE_ALLOCATION` is exported so the tests can read it. Invariant tests (envelope safety,
|
||
tier monotonicity, the floor) reference the constants and hold at any value; one test pins
|
||
the literals, so re-tuning a constant is a visible decision that says *re-run the probe*
|
||
rather than a silent re-calibration of the fixtures.
|