docs: CG-15 agent A/B of the #1500 allocation change — gate fails on the control
Three repos, both arms codegraph-on, sonnet/high, 3 runs per arm. PASS on the two medium repos: client-go (the reporter's Go shape, 2,001 of 2,454 files generated) and excalidraw hold Read 0 in every run of both arms, excalidraw goes 34s -> 24s at the median with one fewer explore call, and the generated clientsets/informers that took 10.5%% of a baseline envelope appear in no new run. FAIL on express, the small control, in 1 run of 3: 4 Reads and 52s against a baseline that read once. Not agent variance — replaying that run's query deterministically, lib/utils.js goes from 6,380 bytes whole to a 583-byte cluster stub and the envelope shrinks 13.8K -> 9.2K against an unchanged 13,000 budget. The diagnostic shows the allocator was right and the render loop was not: utils.js is the top-ranked file, was reserved 3,870 chars, and spent 583. The whole-file bound (allowance + grace = 4,450) lands just under the file's 5,293 bytes, so the whole-file render is declined and the unspent reservation is dropped rather than redistributed. Bar 1 is the hard gate, so per CG-15's acceptance rule the design goes back to CG-12 — the budget is not to be widened to compensate. Two candidate fixes are written up in the design doc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
edce18f586
commit
c7103c7f2f
@@ -278,6 +278,28 @@ Status legend: ✅ done+validated · 🔬 hole identified · ⬜ not started.
|
||||
(Verify the exact supported set against `src/extraction/languages/` and
|
||||
`src/resolution/frameworks/` before starting — this table is a starting point.)
|
||||
|
||||
### Retrieval A/Bs that are not coverage work
|
||||
|
||||
Coverage decides whether a flow *exists* in the graph. A second class of change decides
|
||||
whether the answer explore returns is *sufficient* — how the byte envelope is divided across
|
||||
the files it found. Same pass bar (Read → 0, no wall-clock regression), same `--model sonnet
|
||||
--effort high` rule, but the harness is `ab-new-vs-baseline.sh` (new build vs baseline build,
|
||||
**both codegraph-on**) rather than `run-all.sh`'s with-vs-without, because the question is
|
||||
whether a change to codegraph helped, not whether codegraph helps.
|
||||
|
||||
| Change | Repos | Result |
|
||||
|---|---|---|
|
||||
| **Score-proportional byte allocation** (#1500, epic CG-1) — 2026-08-04, `feature/CG-1` vs `main`, 3 runs/arm | client-go (Go, 2,454 f, 2,001 generated — the reporter's shape), excalidraw (TS, 672 f), express (JS, 147 f, control) | **Gate FAILED.** Read 0/0/0 both arms on client-go and excalidraw; excalidraw **34s → 24s median with one fewer explore call**; generated clientsets/informers drop from 10.5% of a baseline envelope to 0% in every new run. But express regressed in 1 of 3 runs (**4 Reads, 52s**) from a reproducible non-agent cause: a file whose proportional reservation lands below its own size no longer renders whole and its cluster render leaves the reservation **unspent** (`lib/utils.js` 6,380 B whole → 583 B stub, envelope 13.8K → 9.2K). Full record: [`docs/benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) |
|
||||
|
||||
Two harness lessons from that run, both now baked into `ab-new-vs-baseline.sh`:
|
||||
|
||||
- **Name codegraph in the prompt for this class of A/B.** Whether the agent picks the tool at
|
||||
all is an adoption axis a retrieval change does not touch; a run that never calls explore
|
||||
measures nothing about how explore divides its bytes (one pre-run: 0 codegraph calls, 3
|
||||
Reads). It is not a forced-Read-0 — the agent stays free to fall back, which is the bar.
|
||||
- **`CODEGRAPH_NO_PROMPT_HOOK=1` on both arms.** The machine's ambient front-load hook resolves
|
||||
to whatever is in `dist/`, which the script itself rewrites between arms.
|
||||
|
||||
---
|
||||
|
||||
## 7. Known limits & gotchas (from the excalidraw/django work)
|
||||
|
||||
@@ -446,3 +446,59 @@ Both were found by writing the invariant rather than by reading the code:
|
||||
tier monotonicity, the floor) reference the constants and hold at any value; one test pins
|
||||
the literals, so re-tuning a constant is a visible decision that says *re-run the probe*
|
||||
rather than a silent re-calibration of the fixtures.
|
||||
|
||||
## CG-15 — what the agent A/B found: a reservation can go unspent
|
||||
|
||||
The agent A/B (full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md))
|
||||
passed on both medium repos — client-go and excalidraw at Read 0 in every run, excalidraw
|
||||
**34s → 24s median with one fewer explore call**, the generated clientsets that took 10.5% of a
|
||||
baseline envelope gone from every new run — and **failed on the small control**, express, in 1
|
||||
run of 3: 4 Reads of `lib/utils.js` where the baseline made 1 Read of a different file.
|
||||
|
||||
It is not agent variance. Replaying that run's own query deterministically:
|
||||
|
||||
| `lib/utils.js` (5,293 B, 272 lines) | baseline | CG-12 |
|
||||
|---|---|---|
|
||||
| delivered | **6,380 (46.1%) whole** | **583 (7.7%) cluster stub** |
|
||||
| source envelope (budget 13,000) | 13,849 | 9,241 |
|
||||
|
||||
The diagnostic says the allocator was right and the render loop was not:
|
||||
|
||||
```
|
||||
allocation 12,398 reserved of 12,400 pool · nothing cliffed
|
||||
# deliv% bytes reserved score flags render file
|
||||
1 5.7% 583 3,870 56.0 named entry central clusters lib/utils.js
|
||||
```
|
||||
|
||||
`utils.js` is the **top-ranked** file and was **reserved 3,870 chars — of which it spent 583.**
|
||||
The whole-file bound (`src/mcp/tools.ts:4008`) is
|
||||
`allowance + min(GRACE_MAX, allowance * GRACE_FRACTION)` = `3,870 + 580` = 4,450, just under the
|
||||
file's 5,293 bytes, so the whole-file render is declined; the fallback cluster render has three
|
||||
matched symbols to work with and emits 583 chars. The remaining 3,287 chars of the reservation
|
||||
are **not redistributed — they are lost**, which is why the response shrank by a third against
|
||||
an unchanged budget.
|
||||
|
||||
This is CG-12's own acceptance criterion (*"no file that was previously unclipped becomes
|
||||
clipped"*) failing. CG-14 recorded one instance as a documented exception (`memory-budget.ts`);
|
||||
this is the same defect in the wild, where it costs an agent round-trip. It is **not** systemic:
|
||||
on both medium repos the render loop saturates (`[over budget] [TRUNCATED]`, 23,599 of a 23,600
|
||||
pool reserved) so there is nothing left to lose. It needs a file whose reservation lands below
|
||||
its own size while its matched-symbol set is thin — likeliest on small repos, where per-file
|
||||
reservations are smallest.
|
||||
|
||||
### The two candidate fixes
|
||||
|
||||
Widening the envelope is explicitly **not** one of them — that is the dial iter2 already proved
|
||||
doesn't work, and the bytes here were reserved for this file already.
|
||||
|
||||
1. **Let a large-enough reservation buy the whole file.** Render whole when
|
||||
`allowance >= k * fileSize` for some `k < 1` (utils.js sits at 0.73), letting the bounded
|
||||
overshoot the hard ceiling already tolerates absorb the difference. Smaller change, matches
|
||||
the observed shape.
|
||||
2. **Redistribute the shortfall.** Once the render loop knows a file's realised size, hand what
|
||||
it cannot spend to the next-ranked file. The stronger invariant — *the pool is spent* — and
|
||||
it fixes the shrinking-envelope symptom directly rather than case by case.
|
||||
|
||||
They compose; (1) alone would have carried this case. Whichever lands needs the express shape as
|
||||
a hermetic fixture — a mid-sized top-ranked file with few matched symbols, sized just above its
|
||||
reservation — because nothing in the current suite has that shape, which is how it shipped.
|
||||
|
||||
Reference in New Issue
Block a user