github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(explore): hold back what is still owed below a clustered render (CG-31)

Browse Source 
Carry-forward slack let a file spend what the files ABOVE it left on the
table. Nothing held back what was promised BELOW it. The whole-file BUY arm
has always refused that trade (`owedBelow`); the cluster path read `headroom`
— what is left before the hard ceiling — instead of what is still owed, so
`fileBudget` and `SPINE_CEILING` could pay a 1.5x overshoot out of another
file's reservation.

`fundedHeadroom` is the same inequality in the units the cluster path spends
in: source PLUS the per-section overhead each unreached file will charge.
Floored at the file's own reservation — a kept promise is not a displacement —
and it is <= `headroom` by construction, so it is the only bound the three
render sites need. The skeleton path's `bodyCap` takes it too.

Measured on `__tests__/fixtures/displacement-ts` (a 4-stage pipeline padded
past 500 files, where the 24K envelope genuinely saturates the 24.4K render
ceiling):

  before  ingest.ts emitted 9,301 on a 6,289 spendable, then lost the whole
          section to the final ceiling — 0 delivered. types.ts and sink.ts
          skipped `budget-whole-file`. 3 of 6 admitted files delivered.
  after   ingest.ts bounded to the 4,913 actually free. 6 of 6 delivered,
          envelope 14,908 -> 22,066.

The self-query allocation fixture flips back to PASS with it, on a clean full
rebuild of this repo's index (CG-33). Its `afterCG30` verdict blamed an
over-RESERVED incidental file; the reservation was identical in both arms —
the file was over-SPENDING. Recorded honestly in `afterCG31`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-06 02:44:59 -05:00
co-authored by Claude Opus 5
parent 0d014a6582
commit 089dcc276f
13 changed files with 1321 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/mcp/explore-diagnostics.ts
+27 -1
View File
@@ -103,6 +103,15 @@ interface FileRecord extends ExploreCandidateMeta {
* a file spending over its reservation.
*/
spendable: number | null;
/**
* The DISPLACEMENT-GUARDED bound (CG-31): how much this file may render
* without spending a reservation still owed to a file the loop has not
* reached. `spendable` is what the file was promised, this is what is
* actually still there to pay it with — when it sits below `spendable`, the
* difference is the overshoot the guard refused, and the files below this one
* in the table are the reason. `null` until the render loop reaches the file.
*/
funded: number | null;
render?: ExploreRenderMode;
/**
* Source chars this call did NOT re-send because an earlier call in the
@@ -152,6 +161,8 @@ export interface ExploreDiagnosticFile extends ExploreCandidateMeta {
allowance: number | null;
/** Reservation + inherited slack — the bound the render paths actually use. */
spendable: number | null;
/** Same bound after holding back what is still owed to unreached files. */
funded: number | null;
render: ExploreRenderMode | null;
skipped: ExploreSkipReason | null;
clipped: boolean;
@@ -375,7 +386,7 @@ export class ExploreDiagnostics {
/** Record one ranked candidate's scoring inputs, in final sort order. */
noteCandidate(path: string, meta: ExploreCandidateMeta): void {
this.files.set(path, {
path, ...meta, allowance: null, spendable: null,
path, ...meta, allowance: null, spendable: null, funded: null,
dedupSavedChars: 0, dedupCovered: [],
emittedChars: 0, finalChars: 0, share: 0, allocatedShare: 0, clipped: false,
});
@@ -413,6 +424,15 @@ export class ExploreDiagnostics {
if (rec) rec.spendable = chars;
}
/**
* What the render loop will let this file spend once the reservations still
* owed BELOW it are held back (CG-31). Called alongside `recordSpendable`.
*/
recordFunded(path: string, chars: number): void {
const rec = this.files.get(path);
if (rec) rec.funded = chars;
}
/** A candidate rendered source into the response. */
recordRender(path: string, render: ExploreRenderMode, sourceChars: number, clipped: boolean): void {
const rec = this.files.get(path);
@@ -561,6 +581,7 @@ export class ExploreDiagnostics {
kinds: r.kinds,
allowance: r.allowance,
spendable: r.spendable,
funded: r.funded,
render: r.render ?? null,
skipped: r.skipped ?? null,
clipped: r.clipped,
@@ -732,6 +753,11 @@ export function renderTable(report: ExploreDiagnosticReport): string {
if (f.spendable !== null && f.allowance !== null && f.spendable !== f.allowance) {
out.push(` spendable: ${num(f.spendable)} (reservation + inherited slack)`);
}
// Only when the displacement guard actually bit: the gap is what this file
// was refused so the files below it could still be paid.
if (f.funded !== null && f.spendable !== null && f.funded < f.spendable) {
out.push(` funded: ${num(f.funded)} (capped — ${num(f.spendable - f.funded)} held back for files not yet rendered)`);
}
if (f.dedupSavedChars > 0) {
const spans = f.dedupCovered.slice(0, 6).map(([a, b]) => (a === b ? `${a}` : `${a}-${b}`)).join(',');
const more = f.dedupCovered.length > 6 ? `,+${f.dedupCovered.length - 6}` : '';
src/mcp/tools.ts
+60 -7
View File
@@ -4048,6 +4048,11 @@ export class ToolHandler {
// and no file is ever cut BELOW the reservation it was promised.
let reservedSoFar = 0;
let sourceSpent = 0;
// How many admitted files the loop has already drawn a reservation for.
// Pairs with `reservedSoFar` to say how many reservations are still owed
// BELOW the current file — the render-space overhead of those pending
// sections has to be held back too, not just their source (CG-31).
let admittedSoFar = 0;
// Funding line for the whole-file BUY rule: the response's SOURCE may reach
// everything the allocator promised plus one bounded overshoot, and no more.
// Measured against the promise rather than `renderCeiling` on purpose — the
@@ -4055,6 +4060,7 @@ export class ToolHandler {
// what, so funding a buy from it just moves the shortfall to whichever file
// the loop reaches last. See WHOLE_FILE_BUY_OVERSHOOT_FRACTION.
const reservedTotal = [...allocation.allowances.values()].reduce((sum, n) => sum + n, 0);
const admittedTotal = allocation.allowances.size;
const sourceCeiling = reservedTotal + Math.round(
budget.maxOutputChars * EXPLORE_ALLOCATION.WHOLE_FILE_BUY_OVERSHOOT_FRACTION,
);
@@ -4093,7 +4099,39 @@ export class ToolHandler {
Math.max(reserved, Math.round(budget.maxOutputChars * EXPLORE_ALLOCATION.MAX_SHARE)),
);
reservedSoFar += reserved;
admittedSoFar++;
diag?.recordSpendable(filePath, allowance);
// DISPLACEMENT GUARD, in render space (CG-31). `allowance` says what this
// file MAY spend; it does not say the bytes are still there to spend. The
// hard ceiling is shared with every file the loop has not reached yet, and
// their reservations are promises the allocator already made — so what is
// left before the ceiling is not all ours: `owedRenderBelow` of it is
// spoken for. Subtracting it is the same inequality the whole-file BUY arm
// enforces with `owedBelow` (see below), moved into the units the cluster
// path actually spends in — source PLUS the per-section overhead each
// pending file will charge.
//
// Floored at this file's OWN reservation, never below: a kept promise is
// not a displacement, and cutting a file under what it earned is the
// failure this whole allocation layer exists to prevent. When the
// reservations genuinely cannot all fit under the ceiling (the response
// preamble is charged to the same ceiling but not to the allocator's
// envelope), the floor means the shortfall lands on the LAST file rather
// than being taken out of the top one — same as before this guard.
//
// Slack still reaches the file: a file above that under-spends leaves
// `totalChars` lower, which raises `headroom` one-for-one, so the
// carry-forward the `allowance` line grants is exactly the carry-forward
// this bound funds.
const owedBelow = Math.max(0, reservedTotal - reservedSoFar);
const owedRenderBelow = owedBelow
+ EXPLORE_ALLOCATION.FILE_OVERHEAD * Math.max(0, admittedTotal - admittedSoFar);
const headroom = Math.max(0, renderCeiling - totalChars - EXPLORE_ALLOCATION.FILE_OVERHEAD);
const fundedHeadroom = Math.max(
Math.min(reserved, headroom),
headroom - owedRenderBelow,
);
diag?.recordFunded(filePath, Math.min(allowance, fundedHeadroom));
const absPath = validatePathWithinRoot(projectRoot, filePath);
if (!absPath || !existsSync(absPath)) {
diag?.recordSkip(filePath, 'unreadable');
@@ -4306,7 +4344,13 @@ export class ToolHandler {
// response and starve the co-flow file (harness.rs's poll). The native agent
// windows such a file too (~190 lines at a time), so this mimics, not
// truncates. Always emit ≥1 (never an empty section).
const bodyCap = allowance;
//
// Held to `fundedHeadroom` as well (CG-31) so this path cannot spend a
// reservation still owed below it either. It never exceeds `allowance`
// today, so the bound only bites once the ceiling is genuinely tight —
// but "every render path" has to mean every one, or the guard is just a
// detour the next god-file takes.
const bodyCap = Math.min(allowance, fundedHeadroom);
const bodyIds = new Set<string>();
let bodyChars = 0;
for (const n of syms.filter(n => prio(n) < 99 && n.endLine >= n.startLine).sort((a, b) => prio(a) - prio(b))) {
@@ -4434,8 +4478,10 @@ export class ToolHandler {
// rather than a size cap — a buy that fits the line only by spending a
// lower-ranked file's reservation is the trade that dropped
// `payslip_builder.go`, and it is refused here. Self-limiting: each buy
// grows `sourceSpent`, so the pool cannot be spent twice.
const owedBelow = Math.max(0, reservedTotal - reservedSoFar);
// grows `sourceSpent`, so the pool cannot be spent twice. (`owedBelow` is
// computed once at the top of the iteration — the cluster path below
// enforces the same inequality in render space; see `fundedHeadroom`.)
//
// Third condition on the BUY arm only: it must also FIT. A whole render
// that overruns `renderCeiling` is skipped ENTIRELY a few lines below (the
// branch refuses to slice a file mid-method), so attempting a buy that
@@ -4940,13 +4986,20 @@ export class ToolHandler {
// top-scoring file at the same 3,800 as the weakest one, while the whole-file
// branch above handed a small file 3x that. The reservation is the whole point
// of CG-12 — bytes follow relevance, not file size.
const headroom = Math.max(0, renderCeiling - totalChars - 200);
const fileBudget = Math.min(allowance, headroom);
//
// `fundedHeadroom`, not `headroom` (CG-31): what is left before the hard
// ceiling includes every unreached file's reservation, and spending that
// is how one clustered file zeroed five admitted peers. It is ≤ `headroom`
// by construction, so it is the only bound these three lines need.
const fileBudget = Math.min(allowance, fundedHeadroom);
// Spine ceiling: a flow-path cluster may exceed the reservation (the call path
// IS the answer and clipping it forces the Read), but bounded — 1.5x the
// reservation and never past the ceiling — so a pathological long in-file
// spine can't run away or starve co-flow files entirely.
const SPINE_CEILING = Math.min(Math.round(allowance * 1.5), headroom);
// spine can't run away or starve co-flow files entirely. The 1.5x is drawn
// from the shared envelope, so it is exactly the overshoot the displacement
// guard has to fund: past `fundedHeadroom` the extra half-reservation is
// another file's, not spare room.
const SPINE_CEILING = Math.min(Math.round(allowance * 1.5), fundedHeadroom);
const chosenIndices = new Set<number>();
// Final renders (deduped, shrunk where oversize) by cluster index. Computed
// during selection and reused at emission so the two never disagree.