docs(benchmarks): record the CG-26 A/B — the invariant holds on every path
Deterministic 6-repo table, the three agent A/Bs (django, excalidraw, okhttp, 2 runs/arm, Read 0 in all 12 runs), and an honest read of the two repos that deliver a few hundred fewer source chars: at the CG-31 tip both were over-filled by the flat-200 section overhead and paid for it by discarding their epilogue whole. Also: CHANGELOG entries for the two user-visible changes, and the memory note now carries the fourth accounting gap plus the two lessons — hold the REMAINDER when a full reservation no longer fits, and never skip a file over an accounting difference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
7cbde95ce2
commit
5f32478b57
@@ -30,6 +30,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|||||||
- A file built around one very long function no longer takes the whole `codegraph_explore` answer for itself — or disappears from it. Previously such a file was shown in full however big it was, which used up the room every file after it needed, and when the function was larger than the entire response the file was dropped without a word. These files now come back as a bounded window on whole lines — the signature and the top of the body, plus the call site when the call path runs through it — with the rest one follow-up `codegraph_explore` away.
|
- A file built around one very long function no longer takes the whole `codegraph_explore` answer for itself — or disappears from it. Previously such a file was shown in full however big it was, which used up the room every file after it needed, and when the function was larger than the entire response the file was dropped without a word. These files now come back as a bounded window on whole lines — the signature and the top of the body, plus the call site when the call path runs through it — with the rest one follow-up `codegraph_explore` away.
|
||||||
- `codegraph_explore` no longer lets the first file in an answer spend the room set aside for the files below it, so the rest of the answer still arrives. Previously a large file near the top could quietly use up everything left, and the files ranked under it — each already judged relevant enough to include — were dropped with no source at all; on one question only one of six made it into the answer. Every file now keeps what it was given, and a question that really is about one file still concentrates on that file.
|
- `codegraph_explore` no longer lets the first file in an answer spend the room set aside for the files below it, so the rest of the answer still arrives. Previously a large file near the top could quietly use up everything left, and the files ranked under it — each already judged relevant enough to include — were dropped with no source at all; on one question only one of six made it into the answer. Every file now keeps what it was given, and a question that really is about one file still concentrates on that file.
|
||||||
- When a `codegraph_explore` answer runs right up against its size limit, it now drops the trailing notes rather than a whole file's source. Previously the last file was cut even though trimming the notes alone would have fit, so a file that had already been read, ranked and rendered was thrown away at the last moment. Across a range of real projects this returns one more file and up to 20% more source per call.
|
- When a `codegraph_explore` answer runs right up against its size limit, it now drops the trailing notes rather than a whole file's source. Previously the last file was cut even though trimming the notes alone would have fit, so a file that had already been read, ranked and rendered was thrown away at the last moment. Across a range of real projects this returns one more file and up to 20% more source per call.
|
||||||
|
- Every file `codegraph_explore` decides to include now actually arrives. A file shown in full could still spend room set aside for files below it — the fix above covered files shown as excerpts but not files shown whole — and the answer's own size bookkeeping under-counted each file's heading, so the answer ran past its limit and a fully prepared file was discarded at the end. A file that no longer fits whole is now shown as excerpts instead of vanishing, and one that overshoots by a little is trimmed to fit rather than dropped.
|
||||||
|
- The list of files an answer could not cover — the "explore these names for their source" pointers — is no longer thrown away when the answer is full. It is now budgeted for and trimmed to fit, so a full answer still tells you what it left out and which names to ask for next, instead of ending with no pointers at all.
|
||||||
- The blast-radius section of `codegraph_explore` flagged "no covering tests found" whenever no test called a symbol directly — falsely branding helpers that tests exercise through their callers as untested (about 40% of flagged symbols in a measured sample). The check now follows caller chains up to 3 hops and reports indirect coverage as "tested via callers"; when nothing is found it states exactly what was checked instead of an unconditional warning. Thanks @inth3shadows for measuring the false-positive rate. (#1475)
|
- The blast-radius section of `codegraph_explore` flagged "no covering tests found" whenever no test called a symbol directly — falsely branding helpers that tests exercise through their callers as untested (about 40% of flagged symbols in a measured sample). The check now follows caller chains up to 3 hops and reports indirect coverage as "tested via callers"; when nothing is found it states exactly what was checked instead of an unconditional warning. Thanks @inth3shadows for measuring the false-positive rate. (#1475)
|
||||||
|
|
||||||
## [1.5.0] - 2026-07-21
|
## [1.5.0] - 2026-07-21
|
||||||
|
|||||||
@@ -0,0 +1,156 @@
|
|||||||
|
# Agent A/B — the end-to-end reservation invariant (task CG-26)
|
||||||
|
|
||||||
|
**Date:** 2026-08-06 · **New:** `bugfix/CG-26` @ `7cbde95` · **Baseline:** `bugfix/CG-31` @ `c54e008` ·
|
||||||
|
**Harness:** `scripts/agent-eval/ab-new-vs-baseline.sh`, `--model sonnet --effort high`,
|
||||||
|
**both arms codegraph-on**, CLI blocked (0 contamination in every run),
|
||||||
|
`CODEGRAPH_NO_PROMPT_HOOK=1`. Every index measured on was **fully rebuilt**, never
|
||||||
|
incrementally synced (CG-33).
|
||||||
|
|
||||||
|
Baseline is the CG-31 tip, not `main`, so every number here isolates CG-26. Read the three
|
||||||
|
in sequence: `explore-oversize-member-ab-cg30.md` → `explore-displacement-guard-ab-cg31.md` →
|
||||||
|
this one.
|
||||||
|
|
||||||
|
**The invariant:** every admitted file receives at least its reservation before any file draws
|
||||||
|
on carry-forward slack. CG-30 bounded an oversize cluster member; CG-31 gave the cluster path a
|
||||||
|
displacement guard. This closes the three holes left over — and each one was starving a file that
|
||||||
|
had been admitted, reserved, and in the worst case *rendered*.
|
||||||
|
|
||||||
|
**Verdict: no behavioural regression on three repos, and the response is honest about its own
|
||||||
|
budget for the first time.** No repo truncates. No repo loses a file; okhttp gains one. Two repos
|
||||||
|
trade a few hundred source chars on their LAST-ranked file for the pointer list that names what
|
||||||
|
the response could not cover — bytes the CG-31 tip only had because it over-filled a ceiling it
|
||||||
|
mis-measured and then discarded the whole epilogue.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The three holes
|
||||||
|
|
||||||
|
**1. The whole-file arms had no displacement guard.** The BUY arm's fit test read
|
||||||
|
`totalChars + fileContent.length + FILE_OVERHEAD <= renderCeiling` — room before the ceiling,
|
||||||
|
which belongs to every file the loop has not reached — while its own source-space sibling
|
||||||
|
(`owedBelow`) refused exactly that trade. GRACE was not fit-tested at all. Measured on okhttp:
|
||||||
|
`CallServerInterceptor.kt` shipped **8,499 chars against a 5,964 funded ceiling**, and the rank-6
|
||||||
|
file below it delivered nothing. Both arms now test the render they actually produce against
|
||||||
|
`fundedHeadroom`, and a whole render that does not fit **falls through to clustering** instead of
|
||||||
|
skipping the file — a clustered section traded for no section is the trade the funding pool exists
|
||||||
|
to refuse.
|
||||||
|
|
||||||
|
**2. Section overhead was charged at a flat 200 chars.** A real header — path plus up to
|
||||||
|
`maxSymbolsInFileHeader` symbol names — runs 300–500. Everything downstream is expressed in those
|
||||||
|
units (`headroom`, `fundedHeadroom`, every fit test), so the under-count was not a rounding error:
|
||||||
|
it funded promises out of bytes that did not exist. okhttp allocated **26,601 chars against a
|
||||||
|
24,400 ceiling** and the final truncation threw a fully-rendered section away. Sections are
|
||||||
|
charged their real cost now; `owedPayableBelow` holds back each pending file's reservation *plus a
|
||||||
|
per-file overhead estimated from that file's own symbols*; and a marginal overrun **trims the
|
||||||
|
weakest cluster** — or windows the last one into the room that is left — rather than skipping a
|
||||||
|
file over a ~300-char accounting difference.
|
||||||
|
|
||||||
|
**3. `owedPayableBelow` held all-or-nothing.** CG-31 was right that a promise the ceiling cannot
|
||||||
|
reach is not a claim on this file's bytes — but it dropped the *partial* case. When the last
|
||||||
|
admitted file's FULL reservation no longer fit, nothing at all was held for it. On the
|
||||||
|
precise-query fixture the rank-5 file took 4,134 chars against a 2,948 reservation while rank 6 —
|
||||||
|
admitted, reserved 2,539 — was left **4 chars** and skipped. It now holds the remainder, while
|
||||||
|
that remainder is still worth a section (`MIN_CHARS`).
|
||||||
|
|
||||||
|
## The epilogue, budgeted instead of discarded
|
||||||
|
|
||||||
|
CG-31 handed this forward: the loop reserved a flat **600** chars for an epilogue that measures
|
||||||
|
1,064 (gin), 1,788 (django), 2,231 (excalidraw), and four of six suite repos survived by
|
||||||
|
discarding the epilogue **whole** — shipping with no pointer list and no reminders at all. A
|
||||||
|
margin sweep was run and deliberately not shipped, because tuning one constant against the suite
|
||||||
|
is the trap CG-30's record warns about.
|
||||||
|
|
||||||
|
The fix is not a bigger constant. The epilogue is **two things**:
|
||||||
|
|
||||||
|
- a **floor** the render loop reserves, sized from the real strings: the one line that says an
|
||||||
|
uncovered area exists and that another explore — not a Read — reaches it, plus a pointer for
|
||||||
|
every file whose bytes were deliberately WITHHELD (a cliffed file's bytes were traded away on
|
||||||
|
the promise that the agent can still name it — CG-12; if the ceiling eats that name the trade
|
||||||
|
was a silent drop);
|
||||||
|
- an **elastic tail** — the rest of the pointer list and the reminders — fitted, in priority
|
||||||
|
order and entry by entry, to the room that is actually left once the loop is done.
|
||||||
|
|
||||||
|
So a saturated response now lands with as much of its epilogue as it can pay for, instead of none
|
||||||
|
of it, and `renderCeiling` is `hardCeiling − floor` rather than `hardCeiling − 600`.
|
||||||
|
|
||||||
|
## Deterministic measurement — the primary evidence
|
||||||
|
|
||||||
|
Same clean-rebuilt index, same query, both builds. One `codegraph_explore` per repo.
|
||||||
|
Reproduce with `node scripts/agent-eval/probe-suite-envelope.mjs` (added by this task).
|
||||||
|
|
||||||
|
| repo | base source | new source | Δ | base files | new files | ceiling behaviour |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| django | 20,791 | **20,878** | +87 | 6 | 6 | was discarding its epilogue |
|
||||||
|
| tokio | 21,521 | **21,607** | +86 | 5 | 5 | was discarding its epilogue |
|
||||||
|
| okhttp | 19,034 | 18,870 | −164 | 5 | **6** | +1 file delivered; keeps its pointer list |
|
||||||
|
| excalidraw | 20,204 | 19,652 | −552 | 8 | 8 | keeps its pointer list |
|
||||||
|
| gin | 10,776 | 10,776 | 0 | 4 | 4 | byte-identical |
|
||||||
|
| alamofire | 11,662 | 11,662 | 0 | 2 | 2 | byte-identical |
|
||||||
|
|
||||||
|
Queries are the CG-30/CG-31 ones, unchanged.
|
||||||
|
|
||||||
|
**Read the two negatives honestly.** They are not starvation — they are the reverse. At the CG-31
|
||||||
|
tip both responses were *over-filled*: the loop under-counted its own section overhead, spent past
|
||||||
|
the render ceiling, and the hard-ceiling cut then took the epilogue away to pay for it. okhttp
|
||||||
|
also had a file rendered and dropped. Now the accounting is exact, so the loop stops where it
|
||||||
|
said it would, and the ~500 chars go to the pointer list naming the files the response could not
|
||||||
|
cover (2 on excalidraw, both `max-files` skips). No admitted file is starved in either.
|
||||||
|
|
||||||
|
**gin and alamofire are byte-identical between the builds** — neither saturates, so neither the
|
||||||
|
guard nor the epilogue fit engages. That is what a control should show.
|
||||||
|
|
||||||
|
**Fixtures.** `__tests__/explore-reservation-invariant.test.ts` (14 tests; **3 fail on the CG-31
|
||||||
|
tip**) pins the invariant on all three render paths and in both directions — the rank-#1 file when
|
||||||
|
files below it overspend, and an admitted lower-ranked file when the top one does — plus the two
|
||||||
|
things the ceiling must no longer do (allocate past itself; drop a rendered section) and the
|
||||||
|
concentration it must not flatten. `__tests__/explore-displacement-guard.test.ts` (CG-31, 11
|
||||||
|
tests) still passes unchanged.
|
||||||
|
|
||||||
|
**Allocation fixtures** — `scripts/agent-eval/allocation-fixtures.json`: **both PASS**. The
|
||||||
|
self-query gate changed shape and the reason is recorded in `afterCG26`: the envelope-denominated
|
||||||
|
`answerShareAtLeast` reads 47.5% here against 51.0% at the CG-31 tip while `tools.ts` delivers
|
||||||
|
**byte-identical** source in both arms. What moved is the denominator — the response now delivers
|
||||||
|
a fifth admitted file (`memory-budget.ts`, rank 4, paid its full 3,123-char reservation; the CG-31
|
||||||
|
tip rendered it and let the ceiling drop the section) and keeps epilogue prose it used to discard.
|
||||||
|
Both are the improvements this epic exists to make. The gate is now denominated in delivered
|
||||||
|
SOURCE, where the answer group reads 55.5%, and it passes on both arms.
|
||||||
|
|
||||||
|
## Agent runs
|
||||||
|
|
||||||
|
| | django new | django base | excalidraw new | excalidraw base | okhttp new | okhttp base |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| runs | 2 | 2 | 2 | 2 | 2 | 2 |
|
||||||
|
| duration (s) | **42** | 43 [36–50] | 53 [45–62] | 45 [37–54] | 49 [39–59] | 42 [32–52] |
|
||||||
|
| tool calls | 4 [3–4] | 4 [3–5] | **3** [2–4] | 5 [4–5] | 4 | 4 [3–5] |
|
||||||
|
| Read | **0** | 0 | **0** | 0 | **0** | 0 |
|
||||||
|
| Grep/Glob | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
| codegraph calls | 3 [2–3] | 3 [2–3] | **3** [2–3] | 4 [3–4] | 3 | 3 [2–4] |
|
||||||
|
| occupancy share | **36.6%** | 37.6% | **38.2%** | 47.2% | 44.8% | 40.8% |
|
||||||
|
| allocation efficiency | **99.2%** | 94.7% | 81.9% | 82.5% | 75.8% | 87.6% |
|
||||||
|
|
||||||
|
Prompts are the deterministic queries with "Trace the flow end to end." appended.
|
||||||
|
|
||||||
|
**Read is 0 in all 12 runs, in both arms.** Sufficiency, pooled per call: **0 "Read a file we
|
||||||
|
returned" and 0 recall misses on every repo in both arms** — the responses that deliver a few
|
||||||
|
hundred fewer chars do not send the agent back to the file.
|
||||||
|
|
||||||
|
**Where the new arm looks worse, and why it is not read as a regression:**
|
||||||
|
|
||||||
|
- *okhttp allocation efficiency, 75.8% vs 87.6%, and occupancy 44.8% vs 40.8%.* The new arm's
|
||||||
|
envelope is 99,411 chars against 89,297 — it returns one more file and more source overall, and
|
||||||
|
the metric is the share of returned bytes the answer *cited*. Same trade the CG-31 record noted
|
||||||
|
on this repo; the metric's own documentation says it is relative and must not be read as waste.
|
||||||
|
- *Duration on excalidraw and okhttp.* n=2 with fully overlapping ranges (45–62 vs 37–54;
|
||||||
|
39–59 vs 32–52), on a machine also running the other arm's build. excalidraw's new arm does the
|
||||||
|
same work in **3 tool calls against 5** and holds **9 points less context**.
|
||||||
|
|
||||||
|
## Residuals
|
||||||
|
|
||||||
|
None from this task. The render-loop budget is now exact end to end: `totalChars` counts
|
||||||
|
`flow.text`, the real per-section cost, and the epilogue floor; `allocatedChars ≤ hardCeiling` on
|
||||||
|
every suite repo; and the final section-boundary truncation is now unreachable in normal
|
||||||
|
operation (it stays as the backstop).
|
||||||
|
|
||||||
|
One thing deliberately NOT changed: the pointer list still caps at 10 files. Trimming happens
|
||||||
|
from the bottom of the rank order and the "+N more files" tail is rewritten to confess every entry
|
||||||
|
dropped, so the count is never silently wrong.
|
||||||
Reference in New Issue
Block a user