Re-runs CG-15's agent A/B on the fixed build: same harness, same three
prompts, same baseline ref, n=6 per arm on express and excalidraw.
Read = 0 in all 15 new-arm runs. The express regression that routed the
defect to CG-21 does not reproduce in 6 attempts, and the baseline now
reads in 4 of 6 while the new arm reads in none (median 24.5s -> 21.5s),
so the control beats the arm it previously lost to. client-go holds
92.7-96.2% answer share against a baseline run at 53.8%.
Excalidraw's new arm is ~8s slower at the median and that is recorded as
NOT attributable to the build rather than waved through: explore's own
latency is 374ms vs 372ms on the same query and index, the deterministic
responses differ by +2% with one byte-identical, and the unchanged main
build's own median moved 34s -> 26.5s between the two sessions — the same
magnitude as the gap.
Bars were not re-baselined; they are CG-15's four, applied to a larger
sample. The CG-15 section is kept intact and marked superseded, because
its root-cause analysis is the record of why the fix looks like it does.
Three repos, both arms codegraph-on, sonnet/high, 3 runs per arm.
PASS on the two medium repos: client-go (the reporter's Go shape, 2,001 of
2,454 files generated) and excalidraw hold Read 0 in every run of both arms,
excalidraw goes 34s -> 24s at the median with one fewer explore call, and the
generated clientsets/informers that took 10.5%% of a baseline envelope appear
in no new run.
FAIL on express, the small control, in 1 run of 3: 4 Reads and 52s against a
baseline that read once. Not agent variance — replaying that run's query
deterministically, lib/utils.js goes from 6,380 bytes whole to a 583-byte
cluster stub and the envelope shrinks 13.8K -> 9.2K against an unchanged
13,000 budget. The diagnostic shows the allocator was right and the render
loop was not: utils.js is the top-ranked file, was reserved 3,870 chars, and
spent 583. The whole-file bound (allowance + grace = 4,450) lands just under
the file's 5,293 bytes, so the whole-file render is declined and the unspent
reservation is dropped rather than redistributed.
Bar 1 is the hard gate, so per CG-15's acceptance rule the design goes back to
CG-12 — the budget is not to be widened to compensate. Two candidate fixes are
written up in the design doc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Colby McHenry
