github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: CG-21 A/B — the gate passes, all four bars (#1500)

Browse Source 
Re-runs CG-15's agent A/B on the fixed build: same harness, same three
prompts, same baseline ref, n=6 per arm on express and excalidraw.

Read = 0 in all 15 new-arm runs. The express regression that routed the
defect to CG-21 does not reproduce in 6 attempts, and the baseline now
reads in 4 of 6 while the new arm reads in none (median 24.5s -> 21.5s),
so the control beats the arm it previously lost to. client-go holds
92.7-96.2% answer share against a baseline run at 53.8%.

Excalidraw's new arm is ~8s slower at the median and that is recorded as
NOT attributable to the build rather than waved through: explore's own
latency is 374ms vs 372ms on the same query and index, the deterministic
responses differ by +2% with one byte-identical, and the unchanged main
build's own median moved 34s -> 26.5s between the two sessions — the same
magnitude as the gap.

Bars were not re-baselined; they are CG-15's four, applied to a larger
sample. The CG-15 section is kept intact and marked superseded, because
its root-cause analysis is the record of why the fix looks like it does.
This commit is contained in:
Colby McHenry
2026-08-04 02:40:10 -05:00
parent fca7d87047
commit abee46c5e4
3 changed files with 139 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/benchmarks/explore-allocation-ab-1500.md
+100
View File
@@ -12,6 +12,12 @@ only an agent A/B proves the agent stopped reading.
the CG-15 acceptance rule the allocation design goes back to CG-12 — the budget is *not* to be
widened to compensate. Root cause and the smallest honest fix are in [§Root cause](#root-cause).
> **Superseded.** This section is the CG-15 measurement, kept because it is what routed the
> defect to CG-21 and because the root-cause analysis is the record of why. The defect was
> fixed and the A/B re-run: see
> [§Re-run after CG-21](#re-run-after-cg-21--the-gate-passes) at the bottom, where **all four
> bars pass** on a larger sample. Nothing below was re-baselined.
---
## Method
@@ -190,3 +196,97 @@ node scripts/agent-eval/parse-run.mjs /tmp/ab-express/run-new-2.jsonl --answer '
CODEGRAPH_EXPLORE_DEBUG=1 node dist/bin/codegraph.js \
explore "res.send Content-Type ETag generateETag setETag" --path <express>
```
---
# Re-run after CG-21 — the gate passes
**Date:** 2026-08-04 · **New:** `feature/CG-1` @ `fca7d87` (CG-21) · **Baseline:** `main`
(unchanged) · same harness, same three prompts, same repos, `--model sonnet --effort high`,
both arms codegraph-on. **n=6 per arm** on express and excalidraw (two pooled batches of 3 —
same build, same prompts, same baseline ref), n=3 on client-go.
**Verdict: all four bars pass.** Bar 1 — the hard gate that failed above — is clean:
**Read = 0 in all 15 new-arm runs**, including the express control where the defect bit.
### Read and wall-clock
`explore` / `Read` are per-run counts; duration is the median with the range beneath.
| repo | arm | n | explore | **Read** | duration |
|---|---|---|---|---|---|
| **express** (control) | **new** | 6 | 2,2,2,2,1,1 | **0 ×6** | **21.5s** (18–30) |
| | baseline | 6 | 2,1,2,2,2,2 | **1 in 4 of 6** | 24.5s (19–29) |
| **excalidraw** | **new** | 6 | 3,2,2,3,4,2 | **0 ×6** | 34.5s (28–43) |
| | baseline | 6 | 2,4,2,2,2,2 | 0 ×6 | 26.5s (24–45) |
| **client-go** | **new** | 3 | 2,4,4 | **0 ×3** | 45s (44–52) |
| | baseline | 3 | 4,2,4 | 0 ×3 | 43s (40–49) |
The express row is the fix, measured end to end: the CG-12 arm made **4 Reads of
`lib/utils.js`** in 1 run of 3; the CG-21 arm makes **none in 6**, while the *baseline* reads
in 4 of 6 — so the control now beats the baseline it previously lost to, on both Read and
median wall-clock.
### Envelope share (bar 2)
| repo | new | baseline |
|---|---|---|
| express | 100% ×6 | 100% ×6 |
| excalidraw | 65.8 / 79.6 / 78.9% | 82.8 / 78.9 / 85.1% |
| client-go | **96.0 / 96.2 / 92.7%** | 86.7 / **53.8** / 80.2% |
client-go — the #1500 shape — is where the change is supposed to show, and does: the new arm
never drops below 92.7% while the baseline has a 53.8% run. Excalidraw's new arm runs a few
points lower than its baseline; every run is far above the 50% bar and the ranges are within
this harness's run-to-run spread.
### The excalidraw wall-clock gap is not the build
Excalidraw's new arm is ~8s slower at the median, which reads like a bar-3 failure until it is
attributed. Three measurements say it is session variance, not the change:
1. **Explore's own latency is unchanged.** Same query, same index, 5 reps per build:
**median 374 ms new vs 372 ms baseline** (new 371–524, baseline 365–395). The change cannot
cost 8s of wall-clock through a tool that costs the same 0.37s.
2. **The responses are the same size.** Deterministic replay of three excalidraw queries on
both builds: 23,993 vs 20,388, 23,038 vs 25,277, and one **byte-identical** — +2% overall,
in both directions. No truncation in any of the 12 runs, either arm.
3. **The identical baseline build moved 34s → 26.5s between sessions.** `main` did not change
between the CG-15 measurement above and this one, yet its excalidraw median dropped ~8s —
the same magnitude, and in the opposite direction to the CG-15 result (where *new* was 24s
and *baseline* 34s). Between-session variance on this repo is as large as the effect.
So the honest statement is that excalidraw's wall-clock is **noise-dominated at n=6** and
cannot be attributed either way; express (n=6, the control) and client-go (n=3) show no
regression, and express improves. This is the known shape — agent wall-clock is dominated by
host-model thinking, not by tool latency.
### Bars
| # | Bar | Verdict |
|---|---|---|
| 1 | **Read stays at 0** | **PASS** — 0 in all 15 new-arm runs across 3 repos. The CG-15 failure (4 Reads of `lib/utils.js`) does not reproduce in 6 attempts |
| 2 | Correct-file share > 50% | **PASS** — every new run ≥ 65.8%; client-go 92.7–96.2% vs a baseline run at 53.8% |
| 3 | No wall-clock regression | **PASS** — express 24.5s → 21.5s, client-go 43s → 45s (overlapping). Excalidraw's +8s is not attributable to the build (see above) |
| 4 | No regression on the control | **PASS** — express is the control and improves on both axes |
Bars were **not** re-baselined: they are the same four from CG-15, applied to a larger sample.
### Deterministic core
The reproducer that routed the defect to CG-21, on the shipped build:
| `lib/utils.js` (5,293 B, 272 lines) | baseline | CG-12 | **CG-21** |
|---|---|---|---|
| delivered | 6,380 (46.1%) whole | **583 (7.7%) stub** | **6,268 (39.3%) whole** |
| source envelope (13,000 budget) | 13,849 | **9,241** | **14,505** |
```
allocation 12,398 reserved of 12,400 pool · nothing cliffed
# deliv% bytes reserved score flags render file
1 39.3% 6,268 3,870 56.0 named entry central whole lib/utils.js
2 36.8% 5,868 5,875 91.4 entry clusters lib/response.js
3 14.8% 2,369 2,653 34.5 entry central clusters lib/application.js
```
Design and coverage: [`../design/explore-budget-allocation.md`](../design/explore-budget-allocation.md) § CG-21.
docs/design/dynamic-dispatch-coverage-playbook.md
+1
View File
@@ -290,6 +290,7 @@ whether a change to codegraph helped, not whether codegraph helps.
| Change | Repos | Result |
|---|---|---|
| **Score-proportional byte allocation** (#1500, epic CG-1) — 2026-08-04, `feature/CG-1` vs `main`, 3 runs/arm | client-go (Go, 2,454 f, 2,001 generated — the reporter's shape), excalidraw (TS, 672 f), express (JS, 147 f, control) | **Gate FAILED.** Read 0/0/0 both arms on client-go and excalidraw; excalidraw **34s → 24s median with one fewer explore call**; generated clientsets/informers drop from 10.5% of a baseline envelope to 0% in every new run. But express regressed in 1 of 3 runs (**4 Reads, 52s**) from a reproducible non-agent cause: a file whose proportional reservation lands below its own size no longer renders whole and its cluster render leaves the reservation **unspent** (`lib/utils.js` 6,380 B whole → 583 B stub, envelope 13.8K → 9.2K). Full record: [`docs/benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) |
| **↳ re-run after CG-21** — 2026-08-04, `feature/CG-1` @ `fca7d87` vs the same `main`, **6 runs/arm** on express + excalidraw, 3 on client-go | same three repos | **Gate PASSES, all four bars.** **Read = 0 in all 15 new-arm runs** — the express regression does not reproduce in 6 attempts, and the *baseline* now reads in 4 of 6 while the new arm reads in none; express median **24.5s → 21.5s**. client-go answer share 92.7–96.2% vs a baseline run at 53.8%. Excalidraw's new arm is ~8s slower at the median, **not attributed to the build**: explore's own latency is 374 ms vs 372 ms (n=5), deterministic responses differ by +2% with one byte-identical, and the *unchanged* `main` build's own median moved 34s → 26.5s between the two sessions. Deterministic core: `lib/utils.js` 583 B stub → **6,268 B whole**, envelope 9.2K → **14.5K** on an unchanged budget |
Two harness lessons from that run, both now baked into `ab-new-vs-baseline.sh`:
docs/design/explore-budget-allocation.md
+38
View File
@@ -616,3 +616,41 @@ Two traps the first drafts fell into, both of which made a test pass on the defe
legitimately under-spends it (the fixture's `response.ts`: 1,635 delivered of 5,292 reserved).
The assertion is per-file — *delivered >= min(reservation, file size)* — which is what express's
`utils.js` violated and a pool-sum assertion does not express.
### A third condition, found by review rather than by a test
A buy must also **fit the render ceiling**. The whole-file branch refuses to slice a file
mid-method, so a whole render that overruns `renderCeiling` is skipped *entirely* — meaning a
buy approved by the funding pool but refused by the ceiling trades a clustered section for **no
section**. That is the same trade the funding pool exists to refuse, arriving by another route.
It is reachable only on the 24K tiers, which is why neither new fixture can see it:
| tier | envelope | `renderCeiling` = `min(1.5x, 25000) - 600` | funding line = `reservedTotal + 0.15x` |
|---|---|---|---|
| small | 13,000 | 18,900 | ~14,350 — cannot cross |
| medium/large | 24,000 | **24,400** | ~27,200 when saturated — **crosses by ~2.8K** |
So `buysWhole` carries `totalChars + size + FILE_OVERHEAD <= renderCeiling` as well; failing it
drops through to the cluster path, which is bounded by `headroom` and always renders something.
The grace arm is deliberately untouched — a file within a sliver of its reservation that still
does not fit is genuinely at the end of a full response, and that behaviour predates the epic.
Verified inert on all three A/B repos (excalidraw and client-go byte-identical across 3 queries
each, express reproducer unchanged), so it did not invalidate the measurement below.
### The agent A/B (CG-15's gate, re-run)
Full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md)
§ "Re-run after CG-21". **All four bars pass**, at n=6 per arm on express and excalidraw:
- **Read = 0 in all 15 new-arm runs.** The express regression that routed the defect here (4
Reads of `lib/utils.js`) does not reproduce in 6 attempts — and the *baseline* reads in 4 of
6, so the control now beats the arm it previously lost to. Median 24.5s → 21.5s.
- **client-go** — the reporter's shape — holds 92.7–96.2% answer share against a baseline run
at 53.8%.
- **Excalidraw's ~8s median gap is not attributable to the change.** Explore's own latency is
374 ms vs 372 ms (n=5, same query and index); deterministic responses differ by +2% with one
byte-identical; and the *unchanged* `main` build's own median moved 34s → 26.5s between the
CG-15 session and this one — the same magnitude as the gap. Agent wall-clock on this repo is
noise-dominated at this sample size, which is the known shape (host-model thinking dominates,
not tool latency).