docs: CG-21 A/B — the gate passes, all four bars (#1500)

Re-runs CG-15's agent A/B on the fixed build: same harness, same three
prompts, same baseline ref, n=6 per arm on express and excalidraw.

Read = 0 in all 15 new-arm runs. The express regression that routed the
defect to CG-21 does not reproduce in 6 attempts, and the baseline now
reads in 4 of 6 while the new arm reads in none (median 24.5s -> 21.5s),
so the control beats the arm it previously lost to. client-go holds
92.7-96.2% answer share against a baseline run at 53.8%.

Excalidraw's new arm is ~8s slower at the median and that is recorded as
NOT attributable to the build rather than waved through: explore's own
latency is 374ms vs 372ms on the same query and index, the deterministic
responses differ by +2% with one byte-identical, and the unchanged main
build's own median moved 34s -> 26.5s between the two sessions — the same
magnitude as the gap.

Bars were not re-baselined; they are CG-15's four, applied to a larger
sample. The CG-15 section is kept intact and marked superseded, because
its root-cause analysis is the record of why the fix looks like it does.
This commit is contained in:
Colby McHenry
2026-08-04 02:40:10 -05:00
parent fca7d87047
commit abee46c5e4
3 changed files with 139 additions and 0 deletions
+38
View File
@@ -616,3 +616,41 @@ Two traps the first drafts fell into, both of which made a test pass on the defe
legitimately under-spends it (the fixture's `response.ts`: 1,635 delivered of 5,292 reserved).
The assertion is per-file — *delivered >= min(reservation, file size)* — which is what express's
`utils.js` violated and a pool-sum assertion does not express.
### A third condition, found by review rather than by a test
A buy must also **fit the render ceiling**. The whole-file branch refuses to slice a file
mid-method, so a whole render that overruns `renderCeiling` is skipped *entirely* — meaning a
buy approved by the funding pool but refused by the ceiling trades a clustered section for **no
section**. That is the same trade the funding pool exists to refuse, arriving by another route.
It is reachable only on the 24K tiers, which is why neither new fixture can see it:
| tier | envelope | `renderCeiling` = `min(1.5x, 25000) - 600` | funding line = `reservedTotal + 0.15x` |
|---|---|---|---|
| small | 13,000 | 18,900 | ~14,350 — cannot cross |
| medium/large | 24,000 | **24,400** | ~27,200 when saturated — **crosses by ~2.8K** |
So `buysWhole` carries `totalChars + size + FILE_OVERHEAD <= renderCeiling` as well; failing it
drops through to the cluster path, which is bounded by `headroom` and always renders something.
The grace arm is deliberately untouched — a file within a sliver of its reservation that still
does not fit is genuinely at the end of a full response, and that behaviour predates the epic.
Verified inert on all three A/B repos (excalidraw and client-go byte-identical across 3 queries
each, express reproducer unchanged), so it did not invalidate the measurement below.
### The agent A/B (CG-15's gate, re-run)
Full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md)
§ "Re-run after CG-21". **All four bars pass**, at n=6 per arm on express and excalidraw:
- **Read = 0 in all 15 new-arm runs.** The express regression that routed the defect here (4
Reads of `lib/utils.js`) does not reproduce in 6 attempts — and the *baseline* reads in 4 of
6, so the control now beats the arm it previously lost to. Median 24.5s → 21.5s.
- **client-go** — the reporter's shape — holds 92.796.2% answer share against a baseline run
at 53.8%.
- **Excalidraw's ~8s median gap is not attributable to the change.** Explore's own latency is
374 ms vs 372 ms (n=5, same query and index); deterministic responses differ by +2% with one
byte-identical; and the *unchanged* `main` build's own median moved 34s → 26.5s between the
CG-15 session and this one — the same magnitude as the gap. Agent wall-clock on this repo is
noise-dominated at this sample size, which is the known shape (host-model thinking dominates,
not tool latency).