github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: CG-22 — the epic's gate, re-run at CG-15's exact setup (#1500)

Browse Source 
CG-21 fixed the unspent-reservation defect and re-ran the A/B itself. CG-22 is
the gate proper: CG-15's setup, unchanged, measured independently of the task
that wrote the fix. RUNS=3, both arms codegraph-on, sonnet/high,
CODEGRAPH_NO_PROMPT_HOOK=1 on both, baseline pinned to 49c11fc by SHA, fresh
clones of the same three repos and the same three questions.

All four bars pass. Read = 0 in all 12 new-arm runs (express 3, excalidraw 3,
client-go 6) while the baseline reads in 3 of 3 express runs and 1 of 6
client-go runs; the express run that failed CG-15 with 4 Reads of lib/utils.js
now reads nothing and receives the file whole. Answer share >= 66.6% in every
new run. Medians: express 26s -> 24s, excalidraw 26s -> 26s, client-go
35s -> 36.5s at n=6 with fully overlapping ranges.

Deterministic core re-measured on BOTH builds in one session rather than
quoted: lib/utils.js renders whole at 6,380 B on baseline and on HEAD (583 B
stub under CG-12), and the source envelope goes 13,849 -> 14,913 against an
unchanged 13,000 budget, so the reservation is spent and the envelope stops
shrinking. client-go's +1.5s median is attributed away from the build: explore
latency 669 vs 668 ms (n=5) and the new build's deterministic response is both
smaller (15.8K vs 18.9K) and more concentrated (top file 50.2% vs 35.7%).

Two counter-points recorded as measured, not smoothed: excalidraw's new arm
runs below its baseline on answer share (66.6-81.9 vs 75.5-92.7, all well over
the bar), and this session's client-go baseline sampled well (85.6-100%), so
the #1500 gap is smaller here than in CG-21's session.

CHANGELOG: the two #1500 bullets were multi-sentence paragraphs carrying
implementation detail. Rewritten to house style as four bullets that lead with
the symptom, with the mechanism, the banner catalogue and the old-behaviour
contrast dropped; the @LeDuyViet credit and the re-index note stay.

Suite green on the measured build: 171 files, 2,868 passed, 6 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-04 12:58:26 -05:00
co-authored by Claude Opus 5
parent abee46c5e4
commit c65d56ceba
4 changed files with 192 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/dynamic-dispatch-coverage-playbook.md
+1
View File
@@ -291,6 +291,7 @@ whether a change to codegraph helped, not whether codegraph helps.
|---|---|---|
| **Score-proportional byte allocation** (#1500, epic CG-1) — 2026-08-04, `feature/CG-1` vs `main`, 3 runs/arm | client-go (Go, 2,454 f, 2,001 generated — the reporter's shape), excalidraw (TS, 672 f), express (JS, 147 f, control) | **Gate FAILED.** Read 0/0/0 both arms on client-go and excalidraw; excalidraw **34s → 24s median with one fewer explore call**; generated clientsets/informers drop from 10.5% of a baseline envelope to 0% in every new run. But express regressed in 1 of 3 runs (**4 Reads, 52s**) from a reproducible non-agent cause: a file whose proportional reservation lands below its own size no longer renders whole and its cluster render leaves the reservation **unspent** (`lib/utils.js` 6,380 B whole → 583 B stub, envelope 13.8K → 9.2K). Full record: [`docs/benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) |
| **↳ re-run after CG-21** — 2026-08-04, `feature/CG-1` @ `fca7d87` vs the same `main`, **6 runs/arm** on express + excalidraw, 3 on client-go | same three repos | **Gate PASSES, all four bars.** **Read = 0 in all 15 new-arm runs** — the express regression does not reproduce in 6 attempts, and the *baseline* now reads in 4 of 6 while the new arm reads in none; express median **24.5s → 21.5s**. client-go answer share 92.7–96.2% vs a baseline run at 53.8%. Excalidraw's new arm is ~8s slower at the median, **not attributed to the build**: explore's own latency is 374 ms vs 372 ms (n=5), deterministic responses differ by +2% with one byte-identical, and the *unchanged* `main` build's own median moved 34s → 26.5s between the two sessions. Deterministic core: `lib/utils.js` 583 B stub → **6,268 B whole**, envelope 9.2K → **14.5K** on an unchanged budget |
| **↳ the gate (CG-22)** — 2026-08-04, `feature/CG-1` @ `abee46c` vs `main` @ `49c11fc` pinned by SHA, **CG-15's exact setup**: `RUNS=3`, fresh clones, 3 runs/arm (6 on client-go, two pooled batches) | same three repos (excalidraw now 677 f; same tier) | **Gate PASSES, all four bars — re-measured independently of the task that wrote the fix.** **Read = 0 in all 12 new-arm runs**, while the *baseline* reads in **3 of 3** express runs and 1 of 6 client-go runs. Express median 26s → **24s**, `lib/utils.js` delivered whole in every new run (6,455–7,673 B) and never opened. client-go answer share **89.8–97.8%** (baseline 85.6–100% — this session's baseline sampled well, so the gap is smaller than CG-21's); generated clientsets/informers take **0%** in all 12 runs. One honest counter-point: excalidraw's new arm runs **below** its baseline on answer share (66.6–81.9 vs 75.5–92.7), still far above the 50% bar. client-go's +1.5s median at n=6 is **not the build** — explore's own latency is 669 vs 668 ms and the new build's deterministic response is *smaller* (15.8K vs 18.9K) and more concentrated (50.2% vs 35.7% on the top file). Deterministic core, both builds re-measured in-session: `lib/utils.js` **6,380 B whole on both**, envelope **13,849 → 14,913** on an unchanged 13,000 budget |
Two harness lessons from that run, both now baked into `ab-new-vs-baseline.sh`:
docs/design/explore-budget-allocation.md
+10
View File
@@ -640,6 +640,16 @@ each, express reproducer unchanged), so it did not invalidate the measurement be
### The agent A/B (CG-15's gate, re-run)
> **The epic's gate is CG-22, not this section.** This run was measured by the task that wrote
> the fix; CG-22 re-ran it at CG-15's exact setup (`RUNS=3`, fresh clones, baseline pinned to
> `49c11fc` by SHA) and it **passes all four bars there too** — Read = 0 in all 12 new-arm runs
> while the baseline reads in 3 of 3 express runs, and the deterministic reproducer holds with
> both builds re-measured in one session (`lib/utils.js` 6,380 B whole on both, envelope
> 13,849 → 14,913). See
> [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md)
> § CG-22. The two counter-points that section records — excalidraw's answer share running
> below its baseline, and client-go's +1.5s median — are unresolved-but-attributed, not hidden.
Full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md)
§ "Re-run after CG-21". **All four bars pass**, at n=6 per arm on express and excalidraw: