github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: CG-22 — the epic's gate, re-run at CG-15's exact setup (#1500)

Browse Source 
CG-21 fixed the unspent-reservation defect and re-ran the A/B itself. CG-22 is
the gate proper: CG-15's setup, unchanged, measured independently of the task
that wrote the fix. RUNS=3, both arms codegraph-on, sonnet/high,
CODEGRAPH_NO_PROMPT_HOOK=1 on both, baseline pinned to 49c11fc by SHA, fresh
clones of the same three repos and the same three questions.

All four bars pass. Read = 0 in all 12 new-arm runs (express 3, excalidraw 3,
client-go 6) while the baseline reads in 3 of 3 express runs and 1 of 6
client-go runs; the express run that failed CG-15 with 4 Reads of lib/utils.js
now reads nothing and receives the file whole. Answer share >= 66.6% in every
new run. Medians: express 26s -> 24s, excalidraw 26s -> 26s, client-go
35s -> 36.5s at n=6 with fully overlapping ranges.

Deterministic core re-measured on BOTH builds in one session rather than
quoted: lib/utils.js renders whole at 6,380 B on baseline and on HEAD (583 B
stub under CG-12), and the source envelope goes 13,849 -> 14,913 against an
unchanged 13,000 budget, so the reservation is spent and the envelope stops
shrinking. client-go's +1.5s median is attributed away from the build: explore
latency 669 vs 668 ms (n=5) and the new build's deterministic response is both
smaller (15.8K vs 18.9K) and more concentrated (top file 50.2% vs 35.7%).

Two counter-points recorded as measured, not smoothed: excalidraw's new arm
runs below its baseline on answer share (66.6-81.9 vs 75.5-92.7, all well over
the bar), and this session's client-go baseline sampled well (85.6-100%), so
the #1500 gap is smaller here than in CG-21's session.

CHANGELOG: the two #1500 bullets were multi-sentence paragraphs carrying
implementation detail. Rewritten to house style as four bullets that lead with
the symptom, with the mechanism, the banner catalogue and the old-behaviour
contrast dropped; the @LeDuyViet credit and the re-index note stay.

Suite green on the measured build: 171 files, 2,868 passed, 6 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-04 12:58:26 -05:00
co-authored by Claude Opus 5
parent abee46c5e4
commit c65d56ceba
4 changed files with 192 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/explore-budget-allocation.md
+10
View File
@@ -640,6 +640,16 @@ each, express reproducer unchanged), so it did not invalidate the measurement be
### The agent A/B (CG-15's gate, re-run)
> **The epic's gate is CG-22, not this section.** This run was measured by the task that wrote
> the fix; CG-22 re-ran it at CG-15's exact setup (`RUNS=3`, fresh clones, baseline pinned to
> `49c11fc` by SHA) and it **passes all four bars there too** — Read = 0 in all 12 new-arm runs
> while the baseline reads in 3 of 3 express runs, and the deterministic reproducer holds with
> both builds re-measured in one session (`lib/utils.js` 6,380 B whole on both, envelope
> 13,849 → 14,913). See
> [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md)
> § CG-22. The two counter-points that section records — excalidraw's answer share running
> below its baseline, and client-go's +1.5s median — are unresolved-but-attributed, not hidden.
Full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md)
§ "Re-run after CG-21". **All four bars pass**, at n=6 per arm on express and excalidraw: