fix(explore): shrink a later cluster into the remainder instead of dropping it (CG-36)
A file's ranked clusters were all-or-nothing past the first one: the top-ranked cluster was taken (shrunk to fit when it had to be) and every cluster below it was rendered whole, then either fit the remainder or was dropped entirely. On a file whose top-ranked cluster is TRIVIAL that discards the answer — django's `db/models/sql/query.py` kept a 22-line glue cluster and dropped the 624-line `Query` body, spending 1,923 of a 7,947 reservation; okhttp's `RealInterceptorChain.kt` did the same behind its import header. The response stayed full, which is why this was invisible: the unspent reservation carried forward exactly as designed and a file scoring a fifth as much took the bytes. Two sites, the same rule — hold the remainder while it is still worth a section (CG-26's between-FILES lesson, applied between CLUSTERS): - selection now shrinks a later cluster into what is left of the file's budget, by the same whole-member rule the first cluster already used; - the ceiling trim re-renders the weakest cluster into the room that remains before dropping it. On excalidraw's `typeChecks.ts` the section-cost estimate missed by 13 chars and a 1,512-char cluster — the file's highest-SCORING one — was thrown away to pay for it. Cluster RANKING is untouched: measured, both real cases lost on `maxImportance`, not on the density tiebreak the issue suspected, and density-first is what keeps Alamofire's `Session.swift` from burying its methods under the property list. Suite (6 repos, clean-rebuilt indexes): all 8 starvation flags cleared, +1,012 source chars net. django's `sql/query.py` 1,923 -> 10,082 of 7,947, okhttp's `RealInterceptorChain.kt` 1,474 -> 6,038 of 6,058, gin's `routergroup.go` 3,273 -> 5,632. okhttp trades its rank-6 file (score 21) for +7,196 chars in the two files that answer the question. Ships two fixtures pulling in opposite directions (`starved-cluster-ts` and `dense-header-ts`), a `spendShareAtLeast` gate in probe-allocation, and probe-file-spend.mjs — a standing per-file reservation-vs-delivered sweep.
This commit is contained in:
Colby McHenry
@@ -22,7 +22,14 @@
|
||||
"actually about) and `incidental` (what wins the envelope today on name collisions).",
|
||||
"Assertions are on the DELIVERED envelope unless suffixed `Allocated`; delivered is",
|
||||
"what the agent got, allocated is what the render loop chose before the hard ceiling.",
|
||||
"Shares are fractions of the whole response, meta-text included, so they never sum to 1."
|
||||
"Shares are fractions of the whole response, meta-text included, so they never sum to 1.",
|
||||
"",
|
||||
"CG-36 adds two more fixtures and a per-file `spendShareAtLeast` gate. The share gates",
|
||||
"above ask which files WON the envelope; that one asks whether a file that won its share",
|
||||
"then spent it. `starved-cluster` and `dense-header` are the two halves of the same",
|
||||
"tradeoff and must be read together — one fails if a trivial cluster starves the",
|
||||
"answer-bearing one, the other fails if the fix for that buries a query's own methods",
|
||||
"under a dense declaration block."
|
||||
],
|
||||
"fixtures": [
|
||||
{
|
||||
@@ -105,6 +112,124 @@
|
||||
"verdict": "ALL GATES PASS. Answer group 78.7% (from 25.6% at baseline), generated layer 0.0% (from 57.4%). All four hand-written files deliver source, including payslip_builder.go — `func (s *Service) BuildPayslip`, the 'calculate' half of the question, finally reaches the agent. The generated files are still NAMED with their symbols and line numbers under 'Not shown above', so withholding their bytes costs ~100 chars each instead of ~4,500 and stays one follow-up explore away."
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "starved-cluster",
|
||||
"title": "CG-36 — a trivial top-ranked cluster starving the answer-bearing one",
|
||||
"kind": "fixture",
|
||||
"path": "__tests__/fixtures/starved-cluster-ts",
|
||||
"query": "how does a request travel from sendRequest to the socket",
|
||||
"rationale": [
|
||||
"django's `db/models/sql/query.py` and okhttp's `RealInterceptorChain.kt`, reduced",
|
||||
"to a fixture. `chain.ts` holds a one-line `describeChain` helper at the top —",
|
||||
"trivial, but a direct callee of the query's entry point, so its cluster carries",
|
||||
"the file's highest per-symbol importance — and, past the cluster gap, the",
|
||||
"`RequestChain` class that actually answers the question. The helper's cluster wins",
|
||||
"the one guaranteed-and-shrinkable slot; the class then does not fit the remainder.",
|
||||
"",
|
||||
"Before CG-36 the class was dropped WHOLE and the file delivered 1,985 of a 6,904",
|
||||
"reservation. That is not merely unspent budget: the slack carries forward to",
|
||||
"lower-ranked files, so the response stays full and every envelope-share gate",
|
||||
"passes while the answer is missing. Hence `spendShareAtLeast`."
|
||||
],
|
||||
"groups": {
|
||||
"answer": [
|
||||
"src/pipeline/chain.ts",
|
||||
"src/transport/**",
|
||||
"src/app/client.ts"
|
||||
],
|
||||
"incidental": [
|
||||
"src/app/config.ts",
|
||||
"src/pipeline/framing.ts"
|
||||
]
|
||||
},
|
||||
"assert": {
|
||||
"topFileGroup": "answer",
|
||||
"spendShareAtLeast": {
|
||||
"src/pipeline/chain.ts": 0.6
|
||||
},
|
||||
"mustDeliverBytes": [
|
||||
"src/pipeline/chain.ts"
|
||||
],
|
||||
"$mustContainComment": "The two ends of the in-file flow: the chain hop and the transport hop it terminates in. Both live in the cluster that used to be dropped whole.",
|
||||
"mustContain": [
|
||||
"async proceed(request: PipelineRequest)",
|
||||
"private async writeAndRead(request: PipelineRequest)"
|
||||
]
|
||||
},
|
||||
"baseline": {
|
||||
"measuredOn": "2026-08-06",
|
||||
"note": "The CG-24 epic tip (76ab1fe), before CG-36. 3,725 chars of source delivered in total.",
|
||||
"delivered": {
|
||||
"src/pipeline/chain.ts": 1985,
|
||||
"src/app/client.ts": 997,
|
||||
"src/pipeline/types.ts": 743
|
||||
},
|
||||
"verdict": "FAILS spendShareAtLeast and both needles. chain.ts spends 1,985 of its 6,904 reservation (28.8%) — it keeps the `describeChain` cluster and drops the `RequestChain` cluster whole, so neither `proceed` nor `writeAndRead` reaches the agent. Nothing else in the response is wrong: the file still ranks #1 by score and is still reserved the largest slice."
|
||||
},
|
||||
"afterCG36": {
|
||||
"measuredOn": "2026-08-06",
|
||||
"note": "10,802 chars of source delivered in total, nothing truncated.",
|
||||
"delivered": {
|
||||
"src/pipeline/chain.ts": 9062,
|
||||
"src/app/client.ts": 997,
|
||||
"src/pipeline/types.ts": 743
|
||||
},
|
||||
"verdict": "ALL GATES PASS. The `RequestChain` cluster is now SHRUNK into the remainder by the same whole-member rule the first cluster already used, instead of being dropped whole, so chain.ts delivers 9,062 chars including `proceed`, `advance` and `writeAndRead` — the whole in-file flow the question asks for."
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dense-header",
|
||||
"title": "CG-36 — the Session.swift shape density-first ranking exists for",
|
||||
"kind": "fixture",
|
||||
"path": "__tests__/fixtures/dense-header-ts",
|
||||
"query": "how does perform create a URLRequest and start the task",
|
||||
"rationale": [
|
||||
"The counterweight to `starved-cluster`, and the reason CG-36 did NOT touch cluster",
|
||||
"ranking. `session.ts` opens with a 60-line property list and a run of trivial",
|
||||
"accessors — many adjacent, individually worthless declarations, i.e. the densest",
|
||||
"block in the file — while `perform`, `didCreateURLRequest` and `task`, which the",
|
||||
"query names, sit ~200 lines below it.",
|
||||
"",
|
||||
"Ranked on density alone the header block takes the file's whole budget and the",
|
||||
"methods are buried; that is Alamofire's Session.swift, the case the",
|
||||
"importance-then-density order was built for. Any future change to selection or",
|
||||
"shrinking has to keep this passing as well as `starved-cluster` — they pull in",
|
||||
"opposite directions, which is exactly why both are here."
|
||||
],
|
||||
"groups": {
|
||||
"answer": [
|
||||
"src/net/**",
|
||||
"src/core/request-builder.ts",
|
||||
"src/core/task-factory.ts"
|
||||
],
|
||||
"incidental": [
|
||||
"src/core/queue.ts"
|
||||
]
|
||||
},
|
||||
"assert": {
|
||||
"topFileGroup": "answer",
|
||||
"answerShareOfSourceAtLeast": 0.8,
|
||||
"spendShareAtLeast": {
|
||||
"src/net/session.ts": 0.6
|
||||
},
|
||||
"$mustContainComment": "All three named symbols are deep in the file, past the dense header block. If density ever outranks importance again, these are the first thing to go.",
|
||||
"mustContain": [
|
||||
"async perform(url: string, method: string",
|
||||
"didCreateURLRequest(request: URLRequest)",
|
||||
"task(request: URLRequest, identifier: number)"
|
||||
]
|
||||
},
|
||||
"afterCG36": {
|
||||
"measuredOn": "2026-08-06",
|
||||
"note": "11,695 chars of source delivered, BYTE-IDENTICAL to the CG-24 epic tip (76ab1fe) — this fixture pins behaviour CG-36 deliberately left alone.",
|
||||
"delivered": {
|
||||
"src/net/session.ts": 9007,
|
||||
"src/core/types.ts": 1957,
|
||||
"src/core/task-factory.ts": 731
|
||||
},
|
||||
"verdict": "ALL GATES PASS, on the epic tip and on CG-36 alike. session.ts spends 9,007 of its 9,009 reservation and the response carries all three named methods from the bottom of the file. The dense header block is not what won the budget."
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "self-query",
|
||||
"title": "This repo — incidental `explore`/`BUDGET` matches in the agent-eval scripts",
|
||||
|
||||
@@ -199,6 +199,23 @@ function evaluate(fixture, report, text) {
|
||||
: 'not among the ranked candidates',
|
||||
);
|
||||
}
|
||||
// Reservation-vs-delivered, per file (CG-36). The share gates above ask which
|
||||
// files won the envelope; this asks whether a file that WON its share then
|
||||
// actually spent it. A file can rank #1, be reserved the largest slice, and
|
||||
// still deliver a quarter of it because the cluster carrying the answer was
|
||||
// dropped whole instead of shrunk — and the share gates read that as a pass,
|
||||
// since the unspent bytes carry forward and the envelope stays full.
|
||||
for (const [path, floor] of Object.entries(want.spendShareAtLeast ?? {})) {
|
||||
const rec = report.files.find((f) => f.path === path);
|
||||
const spent = rec && rec.allowance ? rec.finalChars / rec.allowance : 0;
|
||||
add(
|
||||
`${path} spends >= ${pct(floor)} of its reservation`,
|
||||
!!rec && rec.allowance > 0 && spent >= floor,
|
||||
rec
|
||||
? `${num(rec.finalChars)} delivered of a ${num(rec.allowance ?? 0)} reservation (${pct(spent)})`
|
||||
: 'not among the ranked candidates',
|
||||
);
|
||||
}
|
||||
for (const needle of want.mustContain ?? []) {
|
||||
add(`response contains "${needle}"`, text.includes(needle), text.includes(needle) ? 'present' : 'absent');
|
||||
}
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Per-file reservation-vs-delivered sweep for `codegraph_explore` (CG-36).
|
||||
*
|
||||
* `probe-suite-envelope.mjs` answers "how much source did the response deliver";
|
||||
* this answers the question one level down — "did the bytes go to the files that
|
||||
* earned them". The CG-36 defect was invisible to the envelope probe because the
|
||||
* envelope stayed full: a rank-#3 file spent 24% of its reservation, the slack
|
||||
* carried forward exactly as designed, and a far weaker file spent 3.5x its own.
|
||||
* The response looked healthy; the ANSWER-bearing file had been starved.
|
||||
*
|
||||
* So the flag here is a PAIR, not a per-file threshold: a file that leaves a
|
||||
* large share of its reservation unspent WHILE a materially lower-scoring file
|
||||
* spends well over its own. Either alone is legitimate — a small file simply has
|
||||
* less to say, and carry-forward is the mechanism that hands its slack down.
|
||||
*
|
||||
* Numbers come from the CG-4 diagnostic (`CODEGRAPH_EXPLORE_DEBUG`), so this
|
||||
* measures the shipping allocator rather than re-deriving shares from markdown.
|
||||
*
|
||||
* Usage (needs a current `npm run build`, and full-REBUILT indexes — CG-33):
|
||||
* node scripts/agent-eval/probe-file-spend.mjs
|
||||
* node scripts/agent-eval/probe-file-spend.mjs --json > /tmp/new.json
|
||||
* node scripts/agent-eval/probe-file-spend.mjs --baseline /tmp/base.json
|
||||
* node scripts/agent-eval/probe-file-spend.mjs django --all # every file, not just flags
|
||||
* CORPUS=/tmp/codegraph-corpus node scripts/agent-eval/probe-file-spend.mjs
|
||||
*
|
||||
* Exit code is 1 when any repo carries a starvation flag, so this can gate.
|
||||
*/
|
||||
import { mkdtempSync, readFileSync, rmSync, existsSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
|
||||
const CORPUS = process.env.CORPUS ?? '/tmp/codegraph-corpus';
|
||||
|
||||
/** Same six repos and queries the CG-30/CG-31/CG-26 envelope tables use. */
|
||||
const SUITE = [
|
||||
{ id: 'django', q: 'How does a QuerySet turn into SQL and fetch rows from the database?' },
|
||||
{ id: 'excalidraw', q: 'How does updating an element re-render the canvas on screen?' },
|
||||
{ id: 'okhttp', q: 'How does a call go through the interceptor chain to the network?' },
|
||||
{ id: 'tokio', q: 'How does a spawned task get scheduled and run by a worker?' },
|
||||
{ id: 'gin', q: 'How does a registered route handler get invoked for an incoming HTTP request?' },
|
||||
{ id: 'alamofire', q: 'How does a request get built and sent through the session?' },
|
||||
];
|
||||
|
||||
/**
|
||||
* Starvation thresholds. A flag needs BOTH sides — the starved file and the
|
||||
* overspending one it lost the bytes to.
|
||||
*
|
||||
* `MIN_RESERVED` keeps the noise out: under it, "80% unspent" is a few hundred
|
||||
* chars and means nothing. `SCORE_RATIO` is what makes the pair meaningful —
|
||||
* a higher-scoring file underspending while a *comparable* one overspends is
|
||||
* ordinary; the defect is a materially weaker file taking the bytes.
|
||||
*/
|
||||
const STARVED_SHARE = 0.5; // spent < half its reservation
|
||||
const OVERSPEND_RATIO = 1.5; // spent > 1.5x its own reservation
|
||||
const SCORE_RATIO = 2; // ...while scoring less than half the starved file
|
||||
const MIN_RESERVED = 2000; // ignore files whose reservation is too small to matter
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
const asJson = argv.includes('--json');
|
||||
const showAll = argv.includes('--all');
|
||||
const baselineAt = argv.includes('--baseline') ? argv[argv.indexOf('--baseline') + 1] : null;
|
||||
const only = argv.filter((a) => !a.startsWith('--') && a !== baselineAt);
|
||||
|
||||
const say = (s = '') => { if (!asJson) console.log(s); };
|
||||
const num = (n) => Math.round(n).toLocaleString('en-US');
|
||||
const pct = (f) => `${(f * 100).toFixed(1)}%`;
|
||||
|
||||
const load = (rel) => import(pathToFileURL(resolve(rel)).href);
|
||||
const idx = await load('dist/index.js');
|
||||
const toolsMod = await load('dist/mcp/tools.js');
|
||||
const CodeGraph = idx.default?.default ?? idx.default ?? idx.CodeGraph;
|
||||
const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler;
|
||||
if (typeof CodeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') {
|
||||
console.error('could not resolve CodeGraph/ToolHandler from dist/ — run `npm run build`');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pair up the starved with the overspenders they lost bytes to. Only files the
|
||||
* render loop actually reached (a reservation and a render mode) take part —
|
||||
* a cliffed or max-files file never had bytes to spend.
|
||||
*/
|
||||
function findStarvation(files) {
|
||||
const spenders = files.filter(
|
||||
(f) => f.allowance !== null && f.allowance > 0 && f.render && f.render !== 'backref',
|
||||
);
|
||||
const flags = [];
|
||||
for (const s of spenders) {
|
||||
if (s.allowance < MIN_RESERVED) continue;
|
||||
if (s.finalChars >= s.allowance * STARVED_SHARE) continue;
|
||||
for (const o of spenders) {
|
||||
if (o.path === s.path) continue;
|
||||
if (o.finalChars <= o.allowance * OVERSPEND_RATIO) continue;
|
||||
if (o.score * SCORE_RATIO > s.score) continue;
|
||||
flags.push({
|
||||
starved: s.path,
|
||||
starvedScore: s.score,
|
||||
starvedReserved: s.allowance,
|
||||
starvedSpent: s.finalChars,
|
||||
overspent: o.path,
|
||||
overspentScore: o.score,
|
||||
overspentReserved: o.allowance,
|
||||
overspentSpent: o.finalChars,
|
||||
});
|
||||
}
|
||||
}
|
||||
return flags;
|
||||
}
|
||||
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'cg-spend-'));
|
||||
const results = [];
|
||||
try {
|
||||
for (const { id, q } of SUITE) {
|
||||
if (only.length > 0 && !only.includes(id)) continue;
|
||||
const repo = join(CORPUS, id);
|
||||
if (!existsSync(join(repo, '.codegraph', 'codegraph.db'))) {
|
||||
say(`${id}: no index at ${repo} — skipped`);
|
||||
continue;
|
||||
}
|
||||
const sidecar = join(tmp, `${id}.jsonl`);
|
||||
process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar;
|
||||
const cg = CodeGraph.openSync(repo);
|
||||
const h = new ToolHandler(cg);
|
||||
await h.execute('codegraph_explore', { query: q });
|
||||
try { cg.close?.(); } catch { /* best effort */ }
|
||||
const report = JSON.parse(readFileSync(sidecar, 'utf8').trim().split('\n').pop());
|
||||
const files = report.files.map((f) => ({
|
||||
path: f.path,
|
||||
rank: f.rank,
|
||||
score: f.score,
|
||||
allowance: f.allowance,
|
||||
spendable: f.spendable,
|
||||
finalChars: f.finalChars,
|
||||
render: f.render,
|
||||
skipped: f.skipped,
|
||||
spent: f.allowance ? f.finalChars / f.allowance : null,
|
||||
}));
|
||||
results.push({
|
||||
repo: id,
|
||||
sourceChars: report.envelope.sourceChars,
|
||||
files,
|
||||
flags: findStarvation(files),
|
||||
});
|
||||
}
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
if (asJson) {
|
||||
console.log(JSON.stringify(results, null, 2));
|
||||
} else {
|
||||
const base = baselineAt ? JSON.parse(readFileSync(baselineAt, 'utf8')) : null;
|
||||
const byRepo = new Map((base ?? []).map((r) => [r.repo, r]));
|
||||
for (const r of results) {
|
||||
const b = byRepo.get(r.repo);
|
||||
say(`\n${r.repo} — ${num(r.sourceChars)} source chars`
|
||||
+ (b ? ` (baseline ${num(b.sourceChars)})` : ''));
|
||||
say(' # score reserved spent spent% render file');
|
||||
say('-'.repeat(96));
|
||||
const flagged = new Set(r.flags.flatMap((f) => [f.starved, f.overspent]));
|
||||
for (const f of r.files) {
|
||||
if (f.allowance === null || f.allowance === 0) continue;
|
||||
if (!showAll && !flagged.has(f.path) && f.spent > STARVED_SHARE && f.spent < OVERSPEND_RATIO) continue;
|
||||
const mark = flagged.has(f.path) ? '*' : ' ';
|
||||
say(
|
||||
`${String(f.rank).padStart(2)}${mark} ${String(Math.round(f.score)).padStart(6)} `
|
||||
+ `${num(f.allowance).padStart(9)} ${num(f.finalChars).padStart(7)} `
|
||||
+ `${pct(f.spent).padStart(7)} ${(f.render ?? f.skipped ?? '—').padEnd(13)} ${f.path}`,
|
||||
);
|
||||
}
|
||||
for (const f of r.flags) {
|
||||
say(` FLAG: ${f.starved} (score ${Math.round(f.starvedScore)}) spent `
|
||||
+ `${num(f.starvedSpent)}/${num(f.starvedReserved)} while ${f.overspent} `
|
||||
+ `(score ${Math.round(f.overspentScore)}) spent ${num(f.overspentSpent)}/${num(f.overspentReserved)}`);
|
||||
}
|
||||
}
|
||||
const total = results.reduce((n, r) => n + r.flags.length, 0);
|
||||
say('');
|
||||
say(total === 0
|
||||
? 'No file leaves a large share of its reservation unspent while a weaker file overspends.'
|
||||
: `STARVATION: ${total} flag(s) across `
|
||||
+ `${results.filter((r) => r.flags.length > 0).map((r) => r.repo).join(', ')}.`);
|
||||
if (base) {
|
||||
const worse = results.filter((r) => {
|
||||
const b = byRepo.get(r.repo);
|
||||
return b && (r.flags.length > b.flags.length || r.sourceChars < b.sourceChars);
|
||||
});
|
||||
say(worse.length === 0
|
||||
? 'No repo flags more or delivers less than the baseline.'
|
||||
: `REGRESSION vs baseline: ${worse.map((r) => r.repo).join(', ')}.`);
|
||||
}
|
||||
if (total > 0) process.exitCode = 1;
|
||||
}
|
||||
Reference in New Issue
Block a user