github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: CG-21 A/B — the gate passes, all four bars (#1500)

Browse Source 
Re-runs CG-15's agent A/B on the fixed build: same harness, same three
prompts, same baseline ref, n=6 per arm on express and excalidraw.

Read = 0 in all 15 new-arm runs. The express regression that routed the
defect to CG-21 does not reproduce in 6 attempts, and the baseline now
reads in 4 of 6 while the new arm reads in none (median 24.5s -> 21.5s),
so the control beats the arm it previously lost to. client-go holds
92.7-96.2% answer share against a baseline run at 53.8%.

Excalidraw's new arm is ~8s slower at the median and that is recorded as
NOT attributable to the build rather than waved through: explore's own
latency is 374ms vs 372ms on the same query and index, the deterministic
responses differ by +2% with one byte-identical, and the unchanged main
build's own median moved 34s -> 26.5s between the two sessions — the same
magnitude as the gap.

Bars were not re-baselined; they are CG-15's four, applied to a larger
sample. The CG-15 section is kept intact and marked superseded, because
its root-cause analysis is the record of why the fix looks like it does.
This commit is contained in:
Colby McHenry
2026-08-04 02:40:10 -05:00
parent fca7d87047
commit abee46c5e4
3 changed files with 139 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/dynamic-dispatch-coverage-playbook.md
+1
View File
@@ -290,6 +290,7 @@ whether a change to codegraph helped, not whether codegraph helps.
| Change | Repos | Result |
|---|---|---|
| **Score-proportional byte allocation** (#1500, epic CG-1) — 2026-08-04, `feature/CG-1` vs `main`, 3 runs/arm | client-go (Go, 2,454 f, 2,001 generated — the reporter's shape), excalidraw (TS, 672 f), express (JS, 147 f, control) | **Gate FAILED.** Read 0/0/0 both arms on client-go and excalidraw; excalidraw **34s → 24s median with one fewer explore call**; generated clientsets/informers drop from 10.5% of a baseline envelope to 0% in every new run. But express regressed in 1 of 3 runs (**4 Reads, 52s**) from a reproducible non-agent cause: a file whose proportional reservation lands below its own size no longer renders whole and its cluster render leaves the reservation **unspent** (`lib/utils.js` 6,380 B whole → 583 B stub, envelope 13.8K → 9.2K). Full record: [`docs/benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) |
| **↳ re-run after CG-21** — 2026-08-04, `feature/CG-1` @ `fca7d87` vs the same `main`, **6 runs/arm** on express + excalidraw, 3 on client-go | same three repos | **Gate PASSES, all four bars.** **Read = 0 in all 15 new-arm runs** — the express regression does not reproduce in 6 attempts, and the *baseline* now reads in 4 of 6 while the new arm reads in none; express median **24.5s → 21.5s**. client-go answer share 92.7–96.2% vs a baseline run at 53.8%. Excalidraw's new arm is ~8s slower at the median, **not attributed to the build**: explore's own latency is 374 ms vs 372 ms (n=5), deterministic responses differ by +2% with one byte-identical, and the *unchanged* `main` build's own median moved 34s → 26.5s between the two sessions. Deterministic core: `lib/utils.js` 583 B stub → **6,268 B whole**, envelope 9.2K → **14.5K** on an unchanged budget |
Two harness lessons from that run, both now baked into `ab-new-vs-baseline.sh`: