docs(benchmarks): record the CG-36 cluster-starvation measurement
The issue blamed the density tiebreak; both real cases lost on maxImportance, so ranking was left alone. Full before/after table, the one cost (okhttp's rank-6 file, squeezed out by reservations that were already structurally over-subscribed), and what ships to keep it measurable.
This commit is contained in:
Colby McHenry
@@ -0,0 +1,164 @@
|
||||
# Deterministic measurement — cluster starvation inside one file (task CG-36)
|
||||
|
||||
**Date:** 2026-08-06 · **Baseline:** `feature/CG-24` @ `76ab1fe` (the CG-24 epic tip) ·
|
||||
**Harness:** `scripts/agent-eval/probe-file-spend.mjs` and `probe-suite-envelope.mjs` over the
|
||||
deterministic 6-repo corpus at `/tmp/codegraph-corpus`, clean full-rebuilt indexes (CG-33), plus
|
||||
two hermetic fixtures through `probe-allocation.mjs`. No agent A/B: the claim is which bytes go
|
||||
to which file, and the agent runs are far too noisy to see a 2K shift.
|
||||
|
||||
**Verdict: the defect is real, the diagnosis in the issue was half wrong, and the fix holds.**
|
||||
All 8 starvation flags across the suite clear; net **+1,012 source chars**. One repo (okhttp)
|
||||
trades its weakest file for +7,196 chars in the two that answer the question — stated in full
|
||||
below rather than smoothed over.
|
||||
|
||||
---
|
||||
|
||||
## The defect
|
||||
|
||||
A file's ranked clusters were all-or-nothing past the first one. The top-ranked cluster was
|
||||
always taken — shrunk to the highest-importance whole symbol ranges that fit when it overran —
|
||||
and every cluster below it was rendered **whole** and then either fit the remainder or was
|
||||
dropped entirely. On a file whose top-ranked cluster is trivial, that discards the answer.
|
||||
|
||||
What made it invisible: the response stays FULL. The unspent reservation carries forward exactly
|
||||
as CG-31 designed, so a lower-scoring file takes the bytes and every envelope-share measure still
|
||||
reads healthy. Measured on the epic tip:
|
||||
|
||||
| repo | file | score | reserved | spent | share |
|
||||
|---|---|---|---|---|---|
|
||||
| django | `db/models/sql/query.py` | 83 | 7,947 | 1,923 | **24%** |
|
||||
| django | `contrib/admin/filters.py` | 18 | 2,271 | 8,057 | **355%** |
|
||||
| okhttp | `.../RealInterceptorChain.kt` | 86 | 6,058 | 1,474 | **24%** |
|
||||
| okhttp | `.../CallServerInterceptor.kt` | 20 | 1,974 | 5,832 | **295%** |
|
||||
|
||||
A score-83 file spends a quarter of its reservation while a score-18 file takes 3.5× its own.
|
||||
|
||||
## What the issue got wrong
|
||||
|
||||
The issue named two candidate fix points and suspected the first: cluster ranking breaks ties on
|
||||
**density** (`score / span`) after `hasSpine` → `maxImportance`, which structurally favours a
|
||||
small trivial cluster over a large answer-bearing one. Dumping the cluster set says otherwise —
|
||||
in **both** real cases the loser lost on `maxImportance`, not on density:
|
||||
|
||||
```
|
||||
django/db/models/sql/query.py budget 10,135 spent 1,923
|
||||
KEPT 1379–1400 span 22 score 14 maxImp 6 (check_related_objects — a glue symbol)
|
||||
DROPPED 306– 929 span 624 score 290 maxImp 3 (133 members: the whole Query class)
|
||||
|
||||
okhttp .../RealInterceptorChain.kt budget 6,058 spent 1,474
|
||||
KEPT 16– 44 span 29 score 44 maxImp 6 (package decl + import block)
|
||||
DROPPED 113– 373 span 261 score 171 maxImp 3 (73 members: the chain itself)
|
||||
```
|
||||
|
||||
`maxImportance` first is deliberate and protective — it is what stops Alamofire's `Session.swift`
|
||||
from losing its budget to the top-of-file property list — so **ranking was not touched**. The
|
||||
lever is the second fix point: stop dropping the loser whole.
|
||||
|
||||
## The change
|
||||
|
||||
Two sites, one rule — *hold the remainder while it is still worth a section*, which is CG-26's
|
||||
between-FILES lesson applied between CLUSTERS.
|
||||
|
||||
1. **Selection.** A later cluster is now shrunk into what is left of the file's budget, by the
|
||||
same whole-member rule the first cluster already used. Below `MIN_CHARS` (700) the remainder
|
||||
cannot hold a readable block, so it stays a drop rather than a stutter of fragments the next
|
||||
call's dedup has to shred around. The never-empty windowing floors may overrun that room; the
|
||||
first cluster is allowed that overshoot, a later one is not.
|
||||
2. **The ceiling trim.** When the exact section cost overruns `renderCeiling`, the weakest chosen
|
||||
cluster is re-rendered into the room that remains before being dropped. This one is worth
|
||||
naming on its own: on excalidraw's `typeChecks.ts` the section-cost estimate missed by
|
||||
**13 chars** and a 1,512-char cluster — the file's highest-*scoring* one, last in rank order
|
||||
only because rank breaks ties on density — was thrown away to pay for it. Recovered 1,501 of
|
||||
excalidraw's 1,449-char loss.
|
||||
|
||||
## Suite result
|
||||
|
||||
`node scripts/agent-eval/probe-file-spend.mjs`, 6 repos, clean rebuilds. Only files whose spend
|
||||
moved are listed; score is the candidate's ranking score, reserved its allocation.
|
||||
|
||||
| repo | file | score | reserved | before | after |
|
||||
|---|---|---|---|---|---|
|
||||
| django | `db/models/sql/query.py` | 83 | 7,947 | 1,923 | **10,082** |
|
||||
| django | `contrib/admin/filters.py` | 18 | 2,271 | 8,057 | 2,198 |
|
||||
| django | `utils/autoreload.py` | 12 | 1,747 | 3,145 | 1,709 |
|
||||
| django | `db/models/fields/related_descriptors.py` | 11 | 1,660 | 2,516 | 1,493 |
|
||||
| excalidraw | `element/src/typeChecks.ts` | 23 | 2,740 | 3,102 | 2,573 |
|
||||
| excalidraw | `excalidraw/types.ts` | 14 | 1,942 | 819 | 1,372 |
|
||||
| okhttp | `.../RealInterceptorChain.kt` | 86 | 6,058 | 1,474 | **6,038** |
|
||||
| okhttp | `.../Interceptor.kt` | 64 | 4,697 | 2,027 | **4,659** |
|
||||
| okhttp | `.../RealCall.kt` | 52 | 3,972 | 3,628 | 3,922 |
|
||||
| okhttp | `.../Call.kt` | 54 | 4,097 | 4,097 | 2,073 |
|
||||
| okhttp | `.../CallServerInterceptor.kt` | 20 | 1,974 | 5,832 | 1,959 |
|
||||
| okhttp | `androidMain/.../AndroidDns.kt` | 21 | 1,999 | 1,812 | **0** |
|
||||
| tokio | `task/local.rs` | 40 | 4,361 | 4,599 | 4,798 |
|
||||
| tokio | `runtime/task/harness.rs` | 14 | 1,981 | 2,565 | 2,341 |
|
||||
| gin | `routergroup.go` | 87 | 5,782 | 3,273 | **5,632** |
|
||||
| gin | `tree.go` | 17 | 1,693 | 892 | 1,969 |
|
||||
| gin | `ginS/gins.go` | 26 | 2,213 | 4,431 | 2,171 |
|
||||
| alamofire | `Source/Core/Session.swift` | 34 | 2,792 | 2,797 | 3,396 |
|
||||
| alamofire | `Source/Core/Request.swift` | 148 | 9,100 | 8,865 | 8,453 |
|
||||
|
||||
Bytes move up the score order in every repo. Envelope totals:
|
||||
|
||||
| repo | source before | after | Δ | files | ceiling |
|
||||
|---|---|---|---|---|---|
|
||||
| django | 20,878 | 20,719 | −159 | 6 → 6 | 24,963 ≤ 25,000 |
|
||||
| excalidraw | 19,652 | 19,704 | +52 | 8 → 8 | 24,813 ≤ 25,000 |
|
||||
| okhttp | 18,870 | 18,651 | −219 | 6 → **5** | 24,985 ≤ 25,000 |
|
||||
| tokio | 21,607 | 21,582 | −25 | 5 → 5 | 24,777 ≤ 25,000 |
|
||||
| gin | 10,776 | 11,952 | **+1,176** | 4 → 4 | 14,655 ≤ 19,500 |
|
||||
| alamofire | 11,662 | 11,849 | +187 | 2 → 2 | 12,862 ≤ 19,500 |
|
||||
| **total** | **103,445** | **104,457** | **+1,012** | | |
|
||||
|
||||
Starvation flags: **8 → 0**.
|
||||
|
||||
## The one cost, stated plainly
|
||||
|
||||
okhttp drops its rank-6 file, `androidMain/.../AndroidDns.kt` (score 21, a platform DNS helper on
|
||||
a question about the interceptor chain), and 219 source chars, in exchange for +4,564 to
|
||||
`RealInterceptorChain.kt` and +2,632 to `Interceptor.kt` — the two files that answer the question.
|
||||
|
||||
This is not a new defect and it is not the fix over-reaching. okhttp's reservations are
|
||||
**structurally over-subscribed**: the allocator splits `maxOutputChars` charging a flat
|
||||
`FILE_OVERHEAD` of 200 per file while a real header runs 300–500, so the sum of promises
|
||||
(~22,800 source + ~2,100 of real headers) exceeds what the ~24,760-char render ceiling can hold.
|
||||
`owedPayableBelow` already refuses to hold bytes back for a file it can see will be dropped, and
|
||||
AndroidDns.kt is the file past that line. On the epic tip it survived only because the files above
|
||||
it under-spent — by luck, not by design. Closing the over-subscription means charging the
|
||||
allocator per-file header estimates rather than the flat 200; that is a wider change than this
|
||||
issue, and CG-26 deliberately kept `FILE_OVERHEAD` as the allocator's own constant.
|
||||
|
||||
## What did NOT change
|
||||
|
||||
- **Cluster ranking.** `hasSpine` → `maxImportance` → density → score → span, untouched.
|
||||
- **Alamofire `Session.swift`.** The shape density-first exists for; it *gains* 599 chars.
|
||||
- **The factory-closure outcome (CG-27).** `probe-factory-closure.mjs`: 7 of 11 inner closure
|
||||
definitions delivered, identical to the epic tip.
|
||||
- **The reservation invariant (CG-31/CG-26).** `explore-reservation-invariant.test.ts` green;
|
||||
every repo stays at or under its hard ceiling.
|
||||
- **All four allocation fixtures pass** — `payroll-go`, `self-query`, and the two added here.
|
||||
|
||||
## What ships so this stays measurable
|
||||
|
||||
- `scripts/agent-eval/probe-file-spend.mjs` — the standing per-file reservation-vs-delivered
|
||||
sweep. It flags a **pair**, never a single file: a large share unspent *while* a materially
|
||||
lower-scoring file overspends. Either alone is legitimate (a small file has less to say;
|
||||
carry-forward is the mechanism that hands its slack down), which is why the envelope probe
|
||||
could never see this. Exit code 1 on any flag, so it gates.
|
||||
- `__tests__/fixtures/starved-cluster-ts/` — django's and okhttp's shape reduced to a fixture.
|
||||
Fails on the epic tip (28.8% of reservation, neither `proceed` nor `writeAndRead` delivered),
|
||||
passes with the fix.
|
||||
- `__tests__/fixtures/dense-header-ts/` — the `Session.swift` shape, byte-identical on both
|
||||
builds. The counterweight: it fails if a future change lets density outrank importance again.
|
||||
- `spendShareAtLeast` in `probe-allocation.mjs`, and `__tests__/explore-cluster-starvation.test.ts`
|
||||
pinning both fixtures in `npm test`.
|
||||
|
||||
## Method note
|
||||
|
||||
None of this is visible in the rendered markdown. To see it you must dump the cluster set —
|
||||
patch `dist/mcp/tools.js` just before `let assembled = assembleSection(chosenIndices);` and log
|
||||
`fileBudget` / `projectedChars` / each ranked cluster's span, score, `maxImportance`, chosen flag
|
||||
and members. Reading only the response makes member-selection effects look like budget effects.
|
||||
Equally, a source-chars diff between builds is not automatically a regression: excalidraw's
|
||||
−1,449 on the first cut was the elastic epilogue expanding into room a 13-char accounting error
|
||||
had released, not source lost to allocation.
|
||||
@@ -47,18 +47,21 @@ change at all**.
|
||||
| **CG-25** | Recognize `Generated by <tool> by running <command>` banners. Precision held by requiring two `by` clauses, so ordinary prose does not match. |
|
||||
| **CG-28** | Damp declaration-only files that nothing in the index depends on. Does not stack with the generated penalty (`Math.min`), and naming a declaration symbol exempts its file. |
|
||||
| **CG-33 / CG-35** | Incremental sync converges with a rebuild, plus a regression suite that fails when the fix is disabled. |
|
||||
| **CG-36** | A later cluster is shrunk into the remainder rather than dropped whole — at selection, and again in the ceiling trim. All 8 starvation flags across the suite clear; +1,012 source chars net. `explore-cluster-starvation-cg36.md`. |
|
||||
|
||||
Deterministic across the 6-repo suite: no repo truncates, none loses a file,
|
||||
okhttp gains one, every repo lands at or under the 25,000 hard ceiling.
|
||||
|
||||
## Open
|
||||
|
||||
**[CG-36]** — a file's non-first clusters are never shrunk, so a trivial cluster
|
||||
starves the answer-bearing one. Found as a byproduct of CG-27's measurement and
|
||||
confirmed firing on the epic tip: `query.py` leaves 81% of its budget unspent,
|
||||
keeping a score-14 cluster and dropping a score-290 one. Two candidate fix
|
||||
points — the density tiebreak in selection, and the never-shrink rule — and
|
||||
density-first must keep working (the `Session.swift` case).
|
||||
Nothing. CG-36, the last one, shipped 2026-08-06.
|
||||
|
||||
Its own measurement is worth carrying forward, because the issue named the wrong
|
||||
fix point: both real cases (`query.py`, `RealInterceptorChain.kt`) lost on
|
||||
`maxImportance`, **not** on the density tiebreak the issue suspected. Ranking was
|
||||
left alone; the never-shrink rule was the lever. Full numbers and the one cost
|
||||
(okhttp trades its rank-6 file for +7,196 chars in the two that answer the
|
||||
question) in `explore-cluster-starvation-cg36.md`.
|
||||
|
||||
## Closed because measurement contradicted them
|
||||
|
||||
|
||||
Reference in New Issue
Block a user