docs: CG-15 agent A/B of the #1500 allocation change — gate fails on the control
Three repos, both arms codegraph-on, sonnet/high, 3 runs per arm. PASS on the two medium repos: client-go (the reporter's Go shape, 2,001 of 2,454 files generated) and excalidraw hold Read 0 in every run of both arms, excalidraw goes 34s -> 24s at the median with one fewer explore call, and the generated clientsets/informers that took 10.5%% of a baseline envelope appear in no new run. FAIL on express, the small control, in 1 run of 3: 4 Reads and 52s against a baseline that read once. Not agent variance — replaying that run's query deterministically, lib/utils.js goes from 6,380 bytes whole to a 583-byte cluster stub and the envelope shrinks 13.8K -> 9.2K against an unchanged 13,000 budget. The diagnostic shows the allocator was right and the render loop was not: utils.js is the top-ranked file, was reserved 3,870 chars, and spent 583. The whole-file bound (allowance + grace = 4,450) lands just under the file's 5,293 bytes, so the whole-file render is declined and the unspent reservation is dropped rather than redistributed. Bar 1 is the hard gate, so per CG-15's acceptance rule the design goes back to CG-12 — the budget is not to be widened to compensate. Two candidate fixes are written up in the design doc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
edce18f586
commit
c7103c7f2f
@@ -278,6 +278,28 @@ Status legend: ✅ done+validated · 🔬 hole identified · ⬜ not started.
|
||||
(Verify the exact supported set against `src/extraction/languages/` and
|
||||
`src/resolution/frameworks/` before starting — this table is a starting point.)
|
||||
|
||||
### Retrieval A/Bs that are not coverage work
|
||||
|
||||
Coverage decides whether a flow *exists* in the graph. A second class of change decides
|
||||
whether the answer explore returns is *sufficient* — how the byte envelope is divided across
|
||||
the files it found. Same pass bar (Read → 0, no wall-clock regression), same `--model sonnet
|
||||
--effort high` rule, but the harness is `ab-new-vs-baseline.sh` (new build vs baseline build,
|
||||
**both codegraph-on**) rather than `run-all.sh`'s with-vs-without, because the question is
|
||||
whether a change to codegraph helped, not whether codegraph helps.
|
||||
|
||||
| Change | Repos | Result |
|
||||
|---|---|---|
|
||||
| **Score-proportional byte allocation** (#1500, epic CG-1) — 2026-08-04, `feature/CG-1` vs `main`, 3 runs/arm | client-go (Go, 2,454 f, 2,001 generated — the reporter's shape), excalidraw (TS, 672 f), express (JS, 147 f, control) | **Gate FAILED.** Read 0/0/0 both arms on client-go and excalidraw; excalidraw **34s → 24s median with one fewer explore call**; generated clientsets/informers drop from 10.5% of a baseline envelope to 0% in every new run. But express regressed in 1 of 3 runs (**4 Reads, 52s**) from a reproducible non-agent cause: a file whose proportional reservation lands below its own size no longer renders whole and its cluster render leaves the reservation **unspent** (`lib/utils.js` 6,380 B whole → 583 B stub, envelope 13.8K → 9.2K). Full record: [`docs/benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) |
|
||||
|
||||
Two harness lessons from that run, both now baked into `ab-new-vs-baseline.sh`:
|
||||
|
||||
- **Name codegraph in the prompt for this class of A/B.** Whether the agent picks the tool at
|
||||
all is an adoption axis a retrieval change does not touch; a run that never calls explore
|
||||
measures nothing about how explore divides its bytes (one pre-run: 0 codegraph calls, 3
|
||||
Reads). It is not a forced-Read-0 — the agent stays free to fall back, which is the bar.
|
||||
- **`CODEGRAPH_NO_PROMPT_HOOK=1` on both arms.** The machine's ambient front-load hook resolves
|
||||
to whatever is in `dist/`, which the script itself rewrites between arms.
|
||||
|
||||
---
|
||||
|
||||
## 7. Known limits & gotchas (from the excalidraw/django work)
|
||||
|
||||
Reference in New Issue
Block a user