docs: CG-15 agent A/B of the #1500 allocation change — gate fails on the control

Three repos, both arms codegraph-on, sonnet/high, 3 runs per arm.

PASS on the two medium repos: client-go (the reporter's Go shape, 2,001 of
2,454 files generated) and excalidraw hold Read 0 in every run of both arms,
excalidraw goes 34s -> 24s at the median with one fewer explore call, and the
generated clientsets/informers that took 10.5%% of a baseline envelope appear
in no new run.

FAIL on express, the small control, in 1 run of 3: 4 Reads and 52s against a
baseline that read once. Not agent variance — replaying that run's query
deterministically, lib/utils.js goes from 6,380 bytes whole to a 583-byte
cluster stub and the envelope shrinks 13.8K -> 9.2K against an unchanged
13,000 budget. The diagnostic shows the allocator was right and the render
loop was not: utils.js is the top-ranked file, was reserved 3,870 chars, and
spent 583. The whole-file bound (allowance + grace = 4,450) lands just under
the file's 5,293 bytes, so the whole-file render is declined and the unspent
reservation is dropped rather than redistributed.

Bar 1 is the hard gate, so per CG-15's acceptance rule the design goes back to
CG-12 — the budget is not to be widened to compensate. Two candidate fixes are
written up in the design doc.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-04 01:26:47 -05:00
co-authored by Claude Opus 5
parent edce18f586
commit c7103c7f2f
3 changed files with 270 additions and 0 deletions
@@ -278,6 +278,28 @@ Status legend: ✅ done+validated · 🔬 hole identified · ⬜ not started.
(Verify the exact supported set against `src/extraction/languages/` and
`src/resolution/frameworks/` before starting — this table is a starting point.)
### Retrieval A/Bs that are not coverage work
Coverage decides whether a flow *exists* in the graph. A second class of change decides
whether the answer explore returns is *sufficient* — how the byte envelope is divided across
the files it found. Same pass bar (Read → 0, no wall-clock regression), same `--model sonnet
--effort high` rule, but the harness is `ab-new-vs-baseline.sh` (new build vs baseline build,
**both codegraph-on**) rather than `run-all.sh`'s with-vs-without, because the question is
whether a change to codegraph helped, not whether codegraph helps.
| Change | Repos | Result |
|---|---|---|
| **Score-proportional byte allocation** (#1500, epic CG-1) — 2026-08-04, `feature/CG-1` vs `main`, 3 runs/arm | client-go (Go, 2,454 f, 2,001 generated — the reporter's shape), excalidraw (TS, 672 f), express (JS, 147 f, control) | **Gate FAILED.** Read 0/0/0 both arms on client-go and excalidraw; excalidraw **34s → 24s median with one fewer explore call**; generated clientsets/informers drop from 10.5% of a baseline envelope to 0% in every new run. But express regressed in 1 of 3 runs (**4 Reads, 52s**) from a reproducible non-agent cause: a file whose proportional reservation lands below its own size no longer renders whole and its cluster render leaves the reservation **unspent** (`lib/utils.js` 6,380 B whole → 583 B stub, envelope 13.8K → 9.2K). Full record: [`docs/benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) |
Two harness lessons from that run, both now baked into `ab-new-vs-baseline.sh`:
- **Name codegraph in the prompt for this class of A/B.** Whether the agent picks the tool at
all is an adoption axis a retrieval change does not touch; a run that never calls explore
measures nothing about how explore divides its bytes (one pre-run: 0 codegraph calls, 3
Reads). It is not a forced-Read-0 — the agent stays free to fall back, which is the bar.
- **`CODEGRAPH_NO_PROMPT_HOOK=1` on both arms.** The machine's ambient front-load hook resolves
to whatever is in `dist/`, which the script itself rewrites between arms.
---
## 7. Known limits & gotchas (from the excalidraw/django work)
+56
View File
@@ -446,3 +446,59 @@ Both were found by writing the invariant rather than by reading the code:
tier monotonicity, the floor) reference the constants and hold at any value; one test pins
the literals, so re-tuning a constant is a visible decision that says *re-run the probe*
rather than a silent re-calibration of the fixtures.
## CG-15 — what the agent A/B found: a reservation can go unspent
The agent A/B (full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md))
passed on both medium repos — client-go and excalidraw at Read 0 in every run, excalidraw
**34s → 24s median with one fewer explore call**, the generated clientsets that took 10.5% of a
baseline envelope gone from every new run — and **failed on the small control**, express, in 1
run of 3: 4 Reads of `lib/utils.js` where the baseline made 1 Read of a different file.
It is not agent variance. Replaying that run's own query deterministically:
| `lib/utils.js` (5,293 B, 272 lines) | baseline | CG-12 |
|---|---|---|
| delivered | **6,380 (46.1%) whole** | **583 (7.7%) cluster stub** |
| source envelope (budget 13,000) | 13,849 | 9,241 |
The diagnostic says the allocator was right and the render loop was not:
```
allocation 12,398 reserved of 12,400 pool · nothing cliffed
# deliv% bytes reserved score flags render file
1 5.7% 583 3,870 56.0 named entry central clusters lib/utils.js
```
`utils.js` is the **top-ranked** file and was **reserved 3,870 chars — of which it spent 583.**
The whole-file bound (`src/mcp/tools.ts:4008`) is
`allowance + min(GRACE_MAX, allowance * GRACE_FRACTION)` = `3,870 + 580` = 4,450, just under the
file's 5,293 bytes, so the whole-file render is declined; the fallback cluster render has three
matched symbols to work with and emits 583 chars. The remaining 3,287 chars of the reservation
are **not redistributed — they are lost**, which is why the response shrank by a third against
an unchanged budget.
This is CG-12's own acceptance criterion (*"no file that was previously unclipped becomes
clipped"*) failing. CG-14 recorded one instance as a documented exception (`memory-budget.ts`);
this is the same defect in the wild, where it costs an agent round-trip. It is **not** systemic:
on both medium repos the render loop saturates (`[over budget] [TRUNCATED]`, 23,599 of a 23,600
pool reserved) so there is nothing left to lose. It needs a file whose reservation lands below
its own size while its matched-symbol set is thin — likeliest on small repos, where per-file
reservations are smallest.
### The two candidate fixes
Widening the envelope is explicitly **not** one of them — that is the dial iter2 already proved
doesn't work, and the bytes here were reserved for this file already.
1. **Let a large-enough reservation buy the whole file.** Render whole when
`allowance >= k * fileSize` for some `k < 1` (utils.js sits at 0.73), letting the bounded
overshoot the hard ceiling already tolerates absorb the difference. Smaller change, matches
the observed shape.
2. **Redistribute the shortfall.** Once the render loop knows a file's realised size, hand what
it cannot spend to the next-ranked file. The stronger invariant — *the pool is spent* — and
it fixes the shrinking-envelope symptom directly rather than case by case.
They compose; (1) alone would have carried this case. Whichever lands needs the express shape as
a hermetic fixture — a mid-sized top-ranked file with few matched symbols, sized just above its
reservation — because nothing in the current suite has that shape, which is how it shipped.