From abee46c5e41ddbb00b62788056bb0f007ef1678e Mon Sep 17 00:00:00 2001 From: Colby McHenry Date: Tue, 4 Aug 2026 02:40:10 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20CG-21=20A/B=20=E2=80=94=20the=20gate=20?= =?UTF-8?q?passes,=20all=20four=20bars=20(#1500)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-runs CG-15's agent A/B on the fixed build: same harness, same three prompts, same baseline ref, n=6 per arm on express and excalidraw. Read = 0 in all 15 new-arm runs. The express regression that routed the defect to CG-21 does not reproduce in 6 attempts, and the baseline now reads in 4 of 6 while the new arm reads in none (median 24.5s -> 21.5s), so the control beats the arm it previously lost to. client-go holds 92.7-96.2% answer share against a baseline run at 53.8%. Excalidraw's new arm is ~8s slower at the median and that is recorded as NOT attributable to the build rather than waved through: explore's own latency is 374ms vs 372ms on the same query and index, the deterministic responses differ by +2% with one byte-identical, and the unchanged main build's own median moved 34s -> 26.5s between the two sessions — the same magnitude as the gap. Bars were not re-baselined; they are CG-15's four, applied to a larger sample. The CG-15 section is kept intact and marked superseded, because its root-cause analysis is the record of why the fix looks like it does. --- docs/benchmarks/explore-allocation-ab-1500.md | 100 ++++++++++++++++++ .../dynamic-dispatch-coverage-playbook.md | 1 + docs/design/explore-budget-allocation.md | 38 +++++++ 3 files changed, 139 insertions(+) diff --git a/docs/benchmarks/explore-allocation-ab-1500.md b/docs/benchmarks/explore-allocation-ab-1500.md index 61a5db1..95ed95f 100644 --- a/docs/benchmarks/explore-allocation-ab-1500.md +++ b/docs/benchmarks/explore-allocation-ab-1500.md @@ -12,6 +12,12 @@ only an agent A/B proves the agent stopped reading. the CG-15 acceptance rule the allocation design goes back to CG-12 — the budget is *not* to be widened to compensate. Root cause and the smallest honest fix are in [§Root cause](#root-cause). +> **Superseded.** This section is the CG-15 measurement, kept because it is what routed the +> defect to CG-21 and because the root-cause analysis is the record of why. The defect was +> fixed and the A/B re-run: see +> [§Re-run after CG-21](#re-run-after-cg-21--the-gate-passes) at the bottom, where **all four +> bars pass** on a larger sample. Nothing below was re-baselined. + --- ## Method @@ -190,3 +196,97 @@ node scripts/agent-eval/parse-run.mjs /tmp/ab-express/run-new-2.jsonl --answer ' CODEGRAPH_EXPLORE_DEBUG=1 node dist/bin/codegraph.js \ explore "res.send Content-Type ETag generateETag setETag" --path ``` + +--- + +# Re-run after CG-21 — the gate passes + +**Date:** 2026-08-04 · **New:** `feature/CG-1` @ `fca7d87` (CG-21) · **Baseline:** `main` +(unchanged) · same harness, same three prompts, same repos, `--model sonnet --effort high`, +both arms codegraph-on. **n=6 per arm** on express and excalidraw (two pooled batches of 3 — +same build, same prompts, same baseline ref), n=3 on client-go. + +**Verdict: all four bars pass.** Bar 1 — the hard gate that failed above — is clean: +**Read = 0 in all 15 new-arm runs**, including the express control where the defect bit. + +### Read and wall-clock + +`explore` / `Read` are per-run counts; duration is the median with the range beneath. + +| repo | arm | n | explore | **Read** | duration | +|---|---|---|---|---|---| +| **express** (control) | **new** | 6 | 2,2,2,2,1,1 | **0 ×6** | **21.5s** (18–30) | +| | baseline | 6 | 2,1,2,2,2,2 | **1 in 4 of 6** | 24.5s (19–29) | +| **excalidraw** | **new** | 6 | 3,2,2,3,4,2 | **0 ×6** | 34.5s (28–43) | +| | baseline | 6 | 2,4,2,2,2,2 | 0 ×6 | 26.5s (24–45) | +| **client-go** | **new** | 3 | 2,4,4 | **0 ×3** | 45s (44–52) | +| | baseline | 3 | 4,2,4 | 0 ×3 | 43s (40–49) | + +The express row is the fix, measured end to end: the CG-12 arm made **4 Reads of +`lib/utils.js`** in 1 run of 3; the CG-21 arm makes **none in 6**, while the *baseline* reads +in 4 of 6 — so the control now beats the baseline it previously lost to, on both Read and +median wall-clock. + +### Envelope share (bar 2) + +| repo | new | baseline | +|---|---|---| +| express | 100% ×6 | 100% ×6 | +| excalidraw | 65.8 / 79.6 / 78.9% | 82.8 / 78.9 / 85.1% | +| client-go | **96.0 / 96.2 / 92.7%** | 86.7 / **53.8** / 80.2% | + +client-go — the #1500 shape — is where the change is supposed to show, and does: the new arm +never drops below 92.7% while the baseline has a 53.8% run. Excalidraw's new arm runs a few +points lower than its baseline; every run is far above the 50% bar and the ranges are within +this harness's run-to-run spread. + +### The excalidraw wall-clock gap is not the build + +Excalidraw's new arm is ~8s slower at the median, which reads like a bar-3 failure until it is +attributed. Three measurements say it is session variance, not the change: + +1. **Explore's own latency is unchanged.** Same query, same index, 5 reps per build: + **median 374 ms new vs 372 ms baseline** (new 371–524, baseline 365–395). The change cannot + cost 8s of wall-clock through a tool that costs the same 0.37s. +2. **The responses are the same size.** Deterministic replay of three excalidraw queries on + both builds: 23,993 vs 20,388, 23,038 vs 25,277, and one **byte-identical** — +2% overall, + in both directions. No truncation in any of the 12 runs, either arm. +3. **The identical baseline build moved 34s → 26.5s between sessions.** `main` did not change + between the CG-15 measurement above and this one, yet its excalidraw median dropped ~8s — + the same magnitude, and in the opposite direction to the CG-15 result (where *new* was 24s + and *baseline* 34s). Between-session variance on this repo is as large as the effect. + +So the honest statement is that excalidraw's wall-clock is **noise-dominated at n=6** and +cannot be attributed either way; express (n=6, the control) and client-go (n=3) show no +regression, and express improves. This is the known shape — agent wall-clock is dominated by +host-model thinking, not by tool latency. + +### Bars + +| # | Bar | Verdict | +|---|---|---| +| 1 | **Read stays at 0** | **PASS** — 0 in all 15 new-arm runs across 3 repos. The CG-15 failure (4 Reads of `lib/utils.js`) does not reproduce in 6 attempts | +| 2 | Correct-file share > 50% | **PASS** — every new run ≥ 65.8%; client-go 92.7–96.2% vs a baseline run at 53.8% | +| 3 | No wall-clock regression | **PASS** — express 24.5s → 21.5s, client-go 43s → 45s (overlapping). Excalidraw's +8s is not attributable to the build (see above) | +| 4 | No regression on the control | **PASS** — express is the control and improves on both axes | + +Bars were **not** re-baselined: they are the same four from CG-15, applied to a larger sample. + +### Deterministic core + +The reproducer that routed the defect to CG-21, on the shipped build: + +| `lib/utils.js` (5,293 B, 272 lines) | baseline | CG-12 | **CG-21** | +|---|---|---|---| +| delivered | 6,380 (46.1%) whole | **583 (7.7%) stub** | **6,268 (39.3%) whole** | +| source envelope (13,000 budget) | 13,849 | **9,241** | **14,505** | + +``` +allocation 12,398 reserved of 12,400 pool · nothing cliffed + # deliv% bytes reserved score flags render file + 1 39.3% 6,268 3,870 56.0 named entry central whole lib/utils.js + 2 36.8% 5,868 5,875 91.4 entry clusters lib/response.js + 3 14.8% 2,369 2,653 34.5 entry central clusters lib/application.js +``` + +Design and coverage: [`../design/explore-budget-allocation.md`](../design/explore-budget-allocation.md) § CG-21. diff --git a/docs/design/dynamic-dispatch-coverage-playbook.md b/docs/design/dynamic-dispatch-coverage-playbook.md index 865ddb6..9b6fe91 100644 --- a/docs/design/dynamic-dispatch-coverage-playbook.md +++ b/docs/design/dynamic-dispatch-coverage-playbook.md @@ -290,6 +290,7 @@ whether a change to codegraph helped, not whether codegraph helps. | Change | Repos | Result | |---|---|---| | **Score-proportional byte allocation** (#1500, epic CG-1) — 2026-08-04, `feature/CG-1` vs `main`, 3 runs/arm | client-go (Go, 2,454 f, 2,001 generated — the reporter's shape), excalidraw (TS, 672 f), express (JS, 147 f, control) | **Gate FAILED.** Read 0/0/0 both arms on client-go and excalidraw; excalidraw **34s → 24s median with one fewer explore call**; generated clientsets/informers drop from 10.5% of a baseline envelope to 0% in every new run. But express regressed in 1 of 3 runs (**4 Reads, 52s**) from a reproducible non-agent cause: a file whose proportional reservation lands below its own size no longer renders whole and its cluster render leaves the reservation **unspent** (`lib/utils.js` 6,380 B whole → 583 B stub, envelope 13.8K → 9.2K). Full record: [`docs/benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) | +| **↳ re-run after CG-21** — 2026-08-04, `feature/CG-1` @ `fca7d87` vs the same `main`, **6 runs/arm** on express + excalidraw, 3 on client-go | same three repos | **Gate PASSES, all four bars.** **Read = 0 in all 15 new-arm runs** — the express regression does not reproduce in 6 attempts, and the *baseline* now reads in 4 of 6 while the new arm reads in none; express median **24.5s → 21.5s**. client-go answer share 92.7–96.2% vs a baseline run at 53.8%. Excalidraw's new arm is ~8s slower at the median, **not attributed to the build**: explore's own latency is 374 ms vs 372 ms (n=5), deterministic responses differ by +2% with one byte-identical, and the *unchanged* `main` build's own median moved 34s → 26.5s between the two sessions. Deterministic core: `lib/utils.js` 583 B stub → **6,268 B whole**, envelope 9.2K → **14.5K** on an unchanged budget | Two harness lessons from that run, both now baked into `ab-new-vs-baseline.sh`: diff --git a/docs/design/explore-budget-allocation.md b/docs/design/explore-budget-allocation.md index cff4d45..92d9f2c 100644 --- a/docs/design/explore-budget-allocation.md +++ b/docs/design/explore-budget-allocation.md @@ -616,3 +616,41 @@ Two traps the first drafts fell into, both of which made a test pass on the defe legitimately under-spends it (the fixture's `response.ts`: 1,635 delivered of 5,292 reserved). The assertion is per-file — *delivered >= min(reservation, file size)* — which is what express's `utils.js` violated and a pool-sum assertion does not express. + +### A third condition, found by review rather than by a test + +A buy must also **fit the render ceiling**. The whole-file branch refuses to slice a file +mid-method, so a whole render that overruns `renderCeiling` is skipped *entirely* — meaning a +buy approved by the funding pool but refused by the ceiling trades a clustered section for **no +section**. That is the same trade the funding pool exists to refuse, arriving by another route. + +It is reachable only on the 24K tiers, which is why neither new fixture can see it: + +| tier | envelope | `renderCeiling` = `min(1.5x, 25000) - 600` | funding line = `reservedTotal + 0.15x` | +|---|---|---|---| +| small | 13,000 | 18,900 | ~14,350 — cannot cross | +| medium/large | 24,000 | **24,400** | ~27,200 when saturated — **crosses by ~2.8K** | + +So `buysWhole` carries `totalChars + size + FILE_OVERHEAD <= renderCeiling` as well; failing it +drops through to the cluster path, which is bounded by `headroom` and always renders something. +The grace arm is deliberately untouched — a file within a sliver of its reservation that still +does not fit is genuinely at the end of a full response, and that behaviour predates the epic. +Verified inert on all three A/B repos (excalidraw and client-go byte-identical across 3 queries +each, express reproducer unchanged), so it did not invalidate the measurement below. + +### The agent A/B (CG-15's gate, re-run) + +Full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) +§ "Re-run after CG-21". **All four bars pass**, at n=6 per arm on express and excalidraw: + +- **Read = 0 in all 15 new-arm runs.** The express regression that routed the defect here (4 + Reads of `lib/utils.js`) does not reproduce in 6 attempts — and the *baseline* reads in 4 of + 6, so the control now beats the arm it previously lost to. Median 24.5s → 21.5s. +- **client-go** — the reporter's shape — holds 92.7–96.2% answer share against a baseline run + at 53.8%. +- **Excalidraw's ~8s median gap is not attributable to the change.** Explore's own latency is + 374 ms vs 372 ms (n=5, same query and index); deterministic responses differ by +2% with one + byte-identical; and the *unchanged* `main` build's own median moved 34s → 26.5s between the + CG-15 session and this one — the same magnitude as the gap. Agent wall-clock on this repo is + noise-dominated at this sample size, which is the known shape (host-model thinking dominates, + not tool latency).