github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'main' into feature/CG-35

Browse Source
This commit is contained in:
Colby McHenry
2026-08-06 21:17:56 -05:00
parent 02ee151e46 969ea1ec37
commit 2c708caf7c
79 changed files with 8946 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/agent-eval/allocation-fixtures.json
+192 -7
View File
@@ -4,7 +4,14 @@
"explore budget allocation. Run them with `node scripts/agent-eval/probe-allocation.mjs`",
"against a built dist/.",
"",
"STATUS: BOTH FIXTURES PASS. CG-10 (relevance scoring) closed the RANKING half —",
"STATUS: BOTH FIXTURES PASS again as of CG-31. self-query's delivered-share gates",
"failed on the CG-30-only build (`afterCG30`) and CG-31 restored them (`afterCG31`):",
"the incidental file was not over-RESERVED at all — it was over-SPENDING, drawing on",
"the reservations of files the render loop had not reached yet. Bounding that put the",
"response back inside the envelope, so nothing truncates and every admitted file",
"delivers. Read the two blocks together; the CG-30 verdict's diagnosis was wrong.",
"",
"CG-10 (relevance scoring) closed the RANKING half —",
"nothing incidental reaches the envelope any more — and CG-12 (score-proportional",
"allocation with a relative cliff) closed the BYTE SPLIT: each file's share is reserved",
"before anything renders, and a file under 15% of the top weight gets no source at all,",
@@ -15,7 +22,14 @@
"actually about) and `incidental` (what wins the envelope today on name collisions).",
"Assertions are on the DELIVERED envelope unless suffixed `Allocated`; delivered is",
"what the agent got, allocated is what the render loop chose before the hard ceiling.",
"Shares are fractions of the whole response, meta-text included, so they never sum to 1."
"Shares are fractions of the whole response, meta-text included, so they never sum to 1.",
"",
"CG-36 adds two more fixtures and a per-file `spendShareAtLeast` gate. The share gates",
"above ask which files WON the envelope; that one asks whether a file that won its share",
"then spent it. `starved-cluster` and `dense-header` are the two halves of the same",
"tradeoff and must be read together — one fails if a trivial cluster starves the",
"answer-bearing one, the other fails if the fix for that buries a query's own methods",
"under a dense declaration block."
],
"fixtures": [
{
@@ -44,7 +58,9 @@
"internal/domain/**",
"cmd/**"
],
"incidental": ["internal/gen/**"]
"incidental": [
"internal/gen/**"
]
},
"assert": {
"answerShareAtLeast": 0.55,
@@ -96,6 +112,124 @@
"verdict": "ALL GATES PASS. Answer group 78.7% (from 25.6% at baseline), generated layer 0.0% (from 57.4%). All four hand-written files deliver source, including payslip_builder.go — `func (s *Service) BuildPayslip`, the 'calculate' half of the question, finally reaches the agent. The generated files are still NAMED with their symbols and line numbers under 'Not shown above', so withholding their bytes costs ~100 chars each instead of ~4,500 and stays one follow-up explore away."
}
},
{
"id": "starved-cluster",
"title": "CG-36 — a trivial top-ranked cluster starving the answer-bearing one",
"kind": "fixture",
"path": "__tests__/fixtures/starved-cluster-ts",
"query": "how does a request travel from sendRequest to the socket",
"rationale": [
"django's `db/models/sql/query.py` and okhttp's `RealInterceptorChain.kt`, reduced",
"to a fixture. `chain.ts` holds a one-line `describeChain` helper at the top —",
"trivial, but a direct callee of the query's entry point, so its cluster carries",
"the file's highest per-symbol importance — and, past the cluster gap, the",
"`RequestChain` class that actually answers the question. The helper's cluster wins",
"the one guaranteed-and-shrinkable slot; the class then does not fit the remainder.",
"",
"Before CG-36 the class was dropped WHOLE and the file delivered 1,985 of a 6,904",
"reservation. That is not merely unspent budget: the slack carries forward to",
"lower-ranked files, so the response stays full and every envelope-share gate",
"passes while the answer is missing. Hence `spendShareAtLeast`."
],
"groups": {
"answer": [
"src/pipeline/chain.ts",
"src/transport/**",
"src/app/client.ts"
],
"incidental": [
"src/app/config.ts",
"src/pipeline/framing.ts"
]
},
"assert": {
"topFileGroup": "answer",
"spendShareAtLeast": {
"src/pipeline/chain.ts": 0.6
},
"mustDeliverBytes": [
"src/pipeline/chain.ts"
],
"$mustContainComment": "The two ends of the in-file flow: the chain hop and the transport hop it terminates in. Both live in the cluster that used to be dropped whole.",
"mustContain": [
"async proceed(request: PipelineRequest)",
"private async writeAndRead(request: PipelineRequest)"
]
},
"baseline": {
"measuredOn": "2026-08-06",
"note": "The CG-24 epic tip (76ab1fe), before CG-36. 3,725 chars of source delivered in total.",
"delivered": {
"src/pipeline/chain.ts": 1985,
"src/app/client.ts": 997,
"src/pipeline/types.ts": 743
},
"verdict": "FAILS spendShareAtLeast and both needles. chain.ts spends 1,985 of its 6,904 reservation (28.8%) — it keeps the `describeChain` cluster and drops the `RequestChain` cluster whole, so neither `proceed` nor `writeAndRead` reaches the agent. Nothing else in the response is wrong: the file still ranks #1 by score and is still reserved the largest slice."
},
"afterCG36": {
"measuredOn": "2026-08-06",
"note": "10,802 chars of source delivered in total, nothing truncated.",
"delivered": {
"src/pipeline/chain.ts": 9062,
"src/app/client.ts": 997,
"src/pipeline/types.ts": 743
},
"verdict": "ALL GATES PASS. The `RequestChain` cluster is now SHRUNK into the remainder by the same whole-member rule the first cluster already used, instead of being dropped whole, so chain.ts delivers 9,062 chars including `proceed`, `advance` and `writeAndRead` — the whole in-file flow the question asks for."
}
},
{
"id": "dense-header",
"title": "CG-36 — the Session.swift shape density-first ranking exists for",
"kind": "fixture",
"path": "__tests__/fixtures/dense-header-ts",
"query": "how does perform create a URLRequest and start the task",
"rationale": [
"The counterweight to `starved-cluster`, and the reason CG-36 did NOT touch cluster",
"ranking. `session.ts` opens with a 60-line property list and a run of trivial",
"accessors — many adjacent, individually worthless declarations, i.e. the densest",
"block in the file — while `perform`, `didCreateURLRequest` and `task`, which the",
"query names, sit ~200 lines below it.",
"",
"Ranked on density alone the header block takes the file's whole budget and the",
"methods are buried; that is Alamofire's Session.swift, the case the",
"importance-then-density order was built for. Any future change to selection or",
"shrinking has to keep this passing as well as `starved-cluster` — they pull in",
"opposite directions, which is exactly why both are here."
],
"groups": {
"answer": [
"src/net/**",
"src/core/request-builder.ts",
"src/core/task-factory.ts"
],
"incidental": [
"src/core/queue.ts"
]
},
"assert": {
"topFileGroup": "answer",
"answerShareOfSourceAtLeast": 0.8,
"spendShareAtLeast": {
"src/net/session.ts": 0.6
},
"$mustContainComment": "All three named symbols are deep in the file, past the dense header block. If density ever outranks importance again, these are the first thing to go.",
"mustContain": [
"async perform(url: string, method: string",
"didCreateURLRequest(request: URLRequest)",
"task(request: URLRequest, identifier: number)"
]
},
"afterCG36": {
"measuredOn": "2026-08-06",
"note": "11,695 chars of source delivered, BYTE-IDENTICAL to the CG-24 epic tip (76ab1fe) — this fixture pins behaviour CG-36 deliberately left alone.",
"delivered": {
"src/net/session.ts": 9007,
"src/core/types.ts": 1957,
"src/core/task-factory.ts": 731
},
"verdict": "ALL GATES PASS, on the epic tip and on CG-36 alike. session.ts spends 9,007 of its 9,009 reservation and the response carries all three named methods from the bottom of the file. The dense header block is not what won the budget."
}
},
{
"id": "self-query",
"title": "This repo — incidental `explore`/`BUDGET` matches in the agent-eval scripts",
@@ -115,14 +249,31 @@
"The assertions are therefore relative — answer-vs-incidental, not fixed percentages."
],
"groups": {
"answer": ["src/mcp/**"],
"incidental": ["scripts/**"]
"answer": [
"src/mcp/**"
],
"incidental": [
"scripts/**"
]
},
"assert": {
"answerShareAtLeast": 0.5,
"$answerShareComment": [
"Denominated in DELIVERED SOURCE, not in the whole envelope (CG-26). The",
"envelope-denominated form of this gate moved for reasons that have nothing",
"to do with allocation: it fell when the epilogue stopped being discarded,",
"and it fell again when a fifth ADMITTED file finally got paid its",
"reservation instead of being dropped by the ceiling. Both are the",
"improvements this epic exists to make, and a gate that reads them as",
"regressions is measuring the denominator. The fixture's own rationale",
"already says the assertions are relative, answer-vs-incidental, not fixed",
"percentages. 0.5 is unchanged; only what it is a share OF."
],
"answerShareOfSourceAtLeast": 0.5,
"incidentalShareAtMost": 0.25,
"topFileGroup": "answer",
"mustDeliverBytes": ["src/mcp/tools.ts"]
"mustDeliverBytes": [
"src/mcp/tools.ts"
]
},
"baseline": {
"measuredOn": "2026-08-03",
@@ -153,6 +304,40 @@
"src/resolution/lru-cache.ts": 0.111
},
"verdict": "ALL GATES PASS. tools.ts takes 60.6% of the envelope, up from 18.5% at baseline and 32.9% after CG-10 — past the epic's >50% acceptance bar. The reversal is the whole point: memory-budget.ts no longer wins by being small enough to ship whole (it now clusters within its 3.1K reservation), and tools.ts is no longer clipped at maxCharsPerFile (11K reservation, ~3x the old flat cap). Exception to 'no previously-unclipped file becomes clipped': memory-budget.ts was unclipped-whole at 5,672 and is now clipped to its proportional share. That is the epic's own diagnosis of the bug, not a regression — it scored 18 against tools.ts's 58 and was taking the larger slice."
},
"afterCG30": {
"measuredOn": "2026-08-06",
"note": "23,688 delivered of 26,430 allocated, truncated at the 25,000 ceiling. Baseline (main) on the SAME index: 14,851 delivered of 25,221 allocated. tools.ts delivers 8,282 chars in BOTH arms — identical bytes; only the denominator moved.",
"delivered": {
"scripts/agent-eval/parse-run.mjs": 0.361,
"src/mcp/tools.ts": 0.35,
"src/mcp/explore-session-state.ts": 0.147,
"src/resolution/memory-budget.ts": 0.0
},
"verdict": "THREE GATES FAIL — and the cause is not the CG-30 bound. Allocation is unchanged between arms (parse-run.mjs 32.3% here vs 33.9% on main); what changed is that it now DELIVERS. On main its whole 8,548-char section was cut by the hard-ceiling truncation, so the incidental group scored 0.0% by luck, not by design, and the fixture passed on that. Bounding the oversize-member overshoot freed enough headroom that the response no longer truncates the same section away. Every file obeys the new bound on this repo (max ratio 1.40x of spendable, against the 1.5x ceiling). What the failure exposes is real and pre-existing: parse-run.mjs scores 18 against tools.ts's 58 yet is reserved a comparable slice — a low-scoring file taking a top-file share, which is epic CG-24's subject. Fix it there; do not tune the CG-30 bound to restore a pass that depended on truncation. SUPERSEDED by afterCG31 — the diagnosis above is wrong on one load-bearing point, see there."
},
"afterCG31": {
"measuredOn": "2026-08-06",
"note": "23,083 delivered of 23,080 allocated — inside the envelope, nothing truncated. Both arms measured on the SAME clean FULL REBUILD of this repo's index (CG-33: an incrementally-synced index diverges and shifts ranking). CG-30-only arm on that index: 23,692 delivered of 26,410 allocated, TRUNCATED.",
"delivered": {
"src/mcp/tools.ts": 0.359,
"scripts/agent-eval/parse-run.mjs": 0.187,
"src/mcp/explore-session-state.ts": 0.151,
"src/resolution/lru-cache.ts": 0.087
},
"verdict": "ALL FOUR GATES PASS. The afterCG30 verdict called parse-run.mjs over-RESERVED; it was not — its reservation is 4,314 in both arms. It was over-SPENDING: 8,548 chars, drawing on reservations belonging to files the render loop had not reached yet, which is the CG-31 defect. With the displacement guard it renders 4,314, tools.ts's identical 8,282 chars go from 35.0% to 35.9% of a response that no longer overruns, and lru-cache.ts (dropped as memory-budget.ts was on the CG-30 arm) delivers. Note what did NOT change: allocation. This fixture moved because the render loop stopped spending other files' bytes, not because anything was re-ranked."
},
"afterCG26": {
"measuredOn": "2026-08-06",
"note": "24,952 delivered of 24,949 allocated, nothing truncated — against the CG-31 tip's 23,083 on the SAME clean full rebuild of this repo's index. tools.ts delivers 8,282 chars in BOTH arms: identical bytes, unchanged reservation, unchanged rank. Total delivered SOURCE 21,228 against 18,105.",
"delivered": {
"src/mcp/tools.ts": 0.334,
"scripts/agent-eval/parse-run.mjs": 0.174,
"src/mcp/explore-session-state.ts": 0.141,
"src/resolution/memory-budget.ts": 0.126,
"src/resolution/lru-cache.ts": 0.081
},
"verdict": "ALL GATES PASS. The one that changed shape is answerShareAtLeast → answerShareOfSourceAtLeast: on the envelope denominator the answer group reads 47.5% here against 51.0% at the CG-31 tip, and neither number is about allocation. tools.ts's bytes are byte-identical between the arms; what moved is that the response now delivers a FIFTH admitted file (memory-budget.ts, rank 4, paid its full 3,123-char reservation — the CG-31 tip rendered it and then let the hard ceiling drop the whole section) and keeps epilogue prose it used to discard. Answer/incidental separation is unchanged and strong: tools.ts 33.4% against parse-run.mjs's 17.4%, incidental 17.4% (down from 18.7%), top delivered file still tools.ts. Measured in delivered source the answer group is 55.5%."
}
}
]
scripts/agent-eval/probe-allocation.mjs
+39
View File
@@ -150,6 +150,28 @@ function evaluate(fixture, report, text) {
`answer ${pct(share('answer'))} delivered (${pct(allocated.get('answer') ?? 0)} allocated)`,
);
}
// Same question against the SOURCE the response delivered rather than the
// whole envelope (CG-26). The envelope-denominated gate above moves whenever
// the response's prose does — the epilogue surviving instead of being
// discarded costs it a point, and every additional admitted file that gets
// paid dilutes it further — so it cannot tell "the answer was starved" from
// "everything else was also delivered". Allocation is about source bytes;
// measure it in source bytes.
if (want.answerShareOfSourceAtLeast !== undefined) {
const sourceBy = new Map();
let totalSource = 0;
for (const f of report.files) {
const g = groupOf(f.path, groups);
sourceBy.set(g, (sourceBy.get(g) ?? 0) + f.finalChars);
totalSource += f.finalChars;
}
const answerSource = totalSource > 0 ? (sourceBy.get('answer') ?? 0) / totalSource : 0;
add(
`answer group takes >= ${pct(want.answerShareOfSourceAtLeast)} of DELIVERED SOURCE`,
answerSource >= want.answerShareOfSourceAtLeast,
`answer ${num(sourceBy.get('answer') ?? 0)} of ${num(totalSource)} source chars (${pct(answerSource)})`,
);
}
if (want.incidentalShareAtMost !== undefined) {
add(
`incidental group takes <= ${pct(want.incidentalShareAtMost)} of the envelope`,
@@ -177,6 +199,23 @@ function evaluate(fixture, report, text) {
: 'not among the ranked candidates',
);
}
// Reservation-vs-delivered, per file (CG-36). The share gates above ask which
// files won the envelope; this asks whether a file that WON its share then
// actually spent it. A file can rank #1, be reserved the largest slice, and
// still deliver a quarter of it because the cluster carrying the answer was
// dropped whole instead of shrunk — and the share gates read that as a pass,
// since the unspent bytes carry forward and the envelope stays full.
for (const [path, floor] of Object.entries(want.spendShareAtLeast ?? {})) {
const rec = report.files.find((f) => f.path === path);
const spent = rec && rec.allowance ? rec.finalChars / rec.allowance : 0;
add(
`${path} spends >= ${pct(floor)} of its reservation`,
!!rec && rec.allowance > 0 && spent >= floor,
rec
? `${num(rec.finalChars)} delivered of a ${num(rec.allowance ?? 0)} reservation (${pct(spent)})`
: 'not among the ranked candidates',
);
}
for (const needle of want.mustContain ?? []) {
add(`response contains "${needle}"`, text.includes(needle), text.includes(needle) ? 'present' : 'absent');
}
scripts/agent-eval/probe-decl-only.mjs
+198
View File
@@ -0,0 +1,198 @@
#!/usr/bin/env node
/**
* CG-28 measurement probe — what a DECLARATION-ONLY file takes from an explore
* envelope, with and without a generated banner.
*
* The issue was filed because a Wrangler `worker-configuration.d.ts` scored 49
* at `pen 1.00` and took 60.7% of an envelope on generic identifier overlap
* (`ReadableStream`, `Body`, `ImageMetadata`, `Message`, …) with a prose query.
* CG-25 has since taught `GENERATED_CONTENT_PATTERNS` the Wrangler banner, so
* the first thing to measure is whether that alone settles it — it does, and
* this probe quantifies it. What CG-25 does NOT cover is a declaration-only file
* that carries no banner at all: a hand-maintained ambient `.d.ts`, vendored
* typings, module augmentation. This probe puts both shapes in ONE fixture
* against ONE envelope so the banner is the only difference between them.
* (`.pyi` is not an indexed extension, so Python stubs never enter the graph.)
*
* Findings and the full regression evidence:
* `docs/benchmarks/explore-declaration-only-cg28.md`.
*
* Fixture: `__tests__/fixtures/ambient-decls-ts/` — an upload path (route →
* stream → metadata → queue) competing with:
* types/worker-configuration.d.ts declaration-only, Wrangler banner (CG-25)
* types/platform-shims.d.ts declaration-only, hand-written, NO banner
* src/storage/types.ts declaration-only but IMPORTED — the control
* that must never be damped
*
* Variants (`--variant`):
* both as committed — the controlled comparison
* strip-banner the banner is deleted from worker-configuration.d.ts, so the
* two declaration files differ in NOTHING the ranker can see;
* the delta against `both` is exactly what CG-25 buys
*
* Usage (needs a current `npm run build`):
* node scripts/agent-eval/probe-decl-only.mjs
* node scripts/agent-eval/probe-decl-only.mjs --variant strip-banner
* node scripts/agent-eval/probe-decl-only.mjs --json
* node scripts/agent-eval/probe-decl-only.mjs --query "..."
*/
import { cpSync, mkdtempSync, readFileSync, writeFileSync, rmSync, existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
const HERE = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = resolve(HERE, '../..');
const FIXTURE = join(REPO_ROOT, '__tests__/fixtures/ambient-decls-ts');
const GENERATED_DECL = 'types/worker-configuration.d.ts';
const HANDWRITTEN_DECL = 'types/platform-shims.d.ts';
/**
* The query shapes. The flow ones are prose and name no symbol — the shape that
* let the original file in. The last one is the counter-case the issue requires:
* a question genuinely ABOUT a declared type must still reach the declaration.
*/
const QUERIES = [
{ id: 'flow-upload', kind: 'flow', text: 'how does an upload request stream the file body to storage and record image metadata' },
{ id: 'flow-pipe', kind: 'flow', text: 'where does the upload body get piped into the bucket and the metadata written' },
{ id: 'flow-generic', kind: 'flow', text: 'how are streams and messages and image metadata handled for uploads' },
{ id: 'flow-queue', kind: 'flow', text: 'what happens after an object is stored and the follow-up message is queued' },
{ id: 'type-shim', kind: 'type', text: 'UploadStorage StoredUploadObject ImageMetadataShim' },
{ id: 'type-prose', kind: 'type', text: 'what does the UploadStorage interface declare for putting an object' },
];
const argv = process.argv.slice(2);
const asJson = argv.includes('--json');
const at = (flag) => { const i = argv.indexOf(flag); return i >= 0 ? argv[i + 1] : undefined; };
const VARIANT = at('--variant') ?? 'both';
const ONE_QUERY = at('--query');
const ONE_ID = at('--only');
const say = (s = '') => { if (!asJson) console.log(s); };
const num = (n) => Math.round(n).toLocaleString('en-US');
const pct = (f) => `${(f * 100).toFixed(1)}%`;
if (!existsSync(join(REPO_ROOT, 'dist/index.js'))) {
console.error('dist/ not built — run `npm run build` first.');
process.exit(2);
}
const load = (rel) => import(pathToFileURL(resolve(REPO_ROOT, rel)).href);
const idxMod = await load('dist/index.js');
const toolsMod = await load('dist/mcp/tools.js');
const CodeGraph = idxMod.default?.default ?? idxMod.default ?? idxMod.CodeGraph;
const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler;
/** Copy the fixture, apply the variant, index it. Hermetic per run. */
function materialize(variant) {
const dir = mkdtempSync(join(tmpdir(), 'cg-decl-'));
cpSync(FIXTURE, dir, { recursive: true });
rmSync(join(dir, '.codegraph'), { recursive: true, force: true });
if (variant === 'strip-banner') {
const p = join(dir, GENERATED_DECL);
// Drop only the banner comment lines; every declaration stays.
const kept = readFileSync(p, 'utf8').split('\n').filter((l) => !/^\/\/ .*(Generated by Wrangler|Runtime types generated)/.test(l));
writeFileSync(p, kept.join('\n'));
} else if (variant !== 'both') {
rmSync(dir, { recursive: true, force: true });
throw new Error(`unknown --variant ${variant} (both | strip-banner)`);
}
return dir;
}
const queries = ONE_QUERY
? [{ id: 'custom', kind: 'flow', text: ONE_QUERY }]
: QUERIES.filter((q) => !ONE_ID || q.id === ONE_ID);
const dir = materialize(VARIANT);
let rows;
try {
let cg = CodeGraph.initSync(dir);
await cg.indexAll();
cg.close?.();
const sidecar = join(dir, 'diag.jsonl');
rows = [];
for (const q of queries) {
rmSync(sidecar, { force: true });
process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar;
cg = CodeGraph.openSync(dir);
const res = await new ToolHandler(cg).execute('codegraph_explore', { query: q.text });
const text = res.content?.[0]?.text ?? '';
cg.close?.();
delete process.env.CODEGRAPH_EXPLORE_DEBUG;
const report = JSON.parse(readFileSync(sidecar, 'utf8').trim().split('\n').pop());
const pick = (path) => {
const f = report.files.find((x) => x.path === path);
if (!f) return null;
return {
path, rank: f.rank, score: f.score, graph: f.graphScore, hits: f.termHits,
penalty: f.penalty, generated: f.generated, render: f.render,
named: f.named, entry: f.entry, central: f.central,
allocatedShare: f.allocatedShare, share: f.share,
emitted: f.emittedChars, final: f.finalChars, skipped: f.skipped,
};
};
const declPaths = new Set([GENERATED_DECL, HANDWRITTEN_DECL]);
const totalSource = report.files.reduce((a, f) => a + f.finalChars, 0);
const declSource = report.files
.filter((f) => declPaths.has(f.path))
.reduce((a, f) => a + f.finalChars, 0);
// "Named in the response but carrying no source" is the correct outcome for
// a cliffed declaration file — the agent can still fetch it in one call.
const namedInResponse = (p) => text.includes(p);
rows.push({
query: q.id, kind: q.kind, text: q.text,
envelope: report.envelope,
generatedDecl: pick(GENERATED_DECL),
handwrittenDecl: pick(HANDWRITTEN_DECL),
declSourceShare: totalSource > 0 ? declSource / totalSource : 0,
implSourceShare: totalSource > 0 ? (totalSource - declSource) / totalSource : 0,
topFile: report.files.filter((f) => f.finalChars > 0).sort((a, b) => b.finalChars - a.finalChars)[0]?.path ?? null,
generatedNamed: namedInResponse(GENERATED_DECL),
handwrittenNamed: namedInResponse(HANDWRITTEN_DECL),
files: report.files
.filter((f) => f.emittedChars > 0 || f.finalChars > 0)
.map((f) => ({ rank: f.rank, path: f.path, score: f.score, graph: f.graphScore, hits: f.termHits, penalty: f.penalty, generated: f.generated, declOnly: f.ambientDeclaration, named: f.named, entry: f.entry, central: f.central, render: f.render, final: f.finalChars, share: f.share })),
});
}
} finally {
rmSync(dir, { recursive: true, force: true });
}
if (asJson) {
console.log(JSON.stringify({ variant: VARIANT, rows }, null, 2));
} else {
say(`variant ${VARIANT}`);
say('');
for (const r of rows) {
say(`── ${r.query} [${r.kind}] "${r.text}"`);
say(` envelope ${num(r.envelope.chars)} chars · decl-only files hold ${pct(r.declSourceShare)} of delivered source`);
say(' # deliv% bytes score graph hits pen gen flags render file');
for (const f of r.files) {
const flags = [f.named && "named", f.entry && "entry", f.central && "central", f.declOnly && "decl-only"].filter(Boolean).join(" ") || "-";
say(
' ' + String(f.rank).padStart(2) + ' ' +
pct(f.share).padStart(6) + ' ' +
num(f.final).padStart(7) + ' ' +
Number(f.score).toFixed(1).padStart(5) + ' ' +
f.graph.toFixed(5).padStart(7) + ' ' +
String(f.hits).padStart(4) + ' ' +
f.penalty.toFixed(2).padStart(4) + ' ' +
(f.generated ? ' ✓ ' : ' ') + ' ' +
flags.padEnd(18) + ' ' +
(f.render ?? '-').padEnd(9) + ' ' +
f.path,
);
}
for (const [label, d, named] of [
['generated ', r.generatedDecl, r.generatedNamed],
['handwritten', r.handwrittenDecl, r.handwrittenNamed],
]) {
say(` ${label} ${d ? `rank #${d.rank}, score ${Number(d.score).toFixed(1)}, pen ${d.penalty.toFixed(2)}, ${num(d.final)} chars (${pct(d.share)})${d.final === 0 ? ` — ${d.skipped ?? d.render ?? 'not rendered'}` : ''}` : 'not a candidate'}${named ? ' · named in response' : ''}`);
}
say('');
}
}
scripts/agent-eval/probe-factory-closure.mjs
+147
View File
@@ -0,0 +1,147 @@
#!/usr/bin/env node
/**
* CG-27 measurement probe — what a factory-closure file actually delivers.
*
* `probe-allocation.mjs` measures how the envelope is split BETWEEN files. This
* one measures what comes back from WITHIN one file whose top-level symbol spans
* almost all of it: a `createFoo()` factory returning an object of closures
* (Svelte 5 rune stores, React hook modules, Zustand `create((set,get)=>({…}))`,
* IIFE module-pattern JS). The claim under test is a ranking one, not a byte one
* — CG-30 already bounds the bytes — so the number that matters is WHICH inner
* symbols reach the agent, not how many chars did.
*
* Prints, for the factory file: every line range the response delivered, and for
* each inner function whether its DEFINITION LINE is inside one of them.
*
* Usage (needs a current `npm run build`):
* node scripts/agent-eval/probe-factory-closure.mjs
* node scripts/agent-eval/probe-factory-closure.mjs --json
* node scripts/agent-eval/probe-factory-closure.mjs --query "..."
*/
import { cpSync, mkdtempSync, readFileSync, rmSync, existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
const HERE = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = resolve(HERE, '../..');
const FIXTURE = join(REPO_ROOT, '__tests__/fixtures/factory-closure-ts');
const targetAt = process.argv.indexOf('--target');
const TARGET = targetAt >= 0 ? process.argv[targetAt + 1] : 'src/stores/dashboard-store.ts';
const factoryAt = process.argv.indexOf('--factory');
const FACTORY = factoryAt >= 0 ? process.argv[factoryAt + 1] : 'createDashboardStore';
const argv = process.argv.slice(2);
const asJson = argv.includes('--json');
const queryAt = argv.indexOf('--query');
const QUERY = queryAt >= 0
? argv[queryAt + 1]
: 'how does the dashboard store refresh its metrics and apply a filter';
const say = (s = '') => { if (!asJson) console.log(s); };
const num = (n) => Math.round(n).toLocaleString('en-US');
const load = (rel) => import(pathToFileURL(resolve(REPO_ROOT, rel)).href);
if (!existsSync(join(REPO_ROOT, 'dist/index.js'))) {
console.error('dist/ not built — run `npm run build` first.');
process.exit(2);
}
const idxMod = await load('dist/index.js');
const toolsMod = await load('dist/mcp/tools.js');
const CodeGraph = idxMod.default?.default ?? idxMod.default ?? idxMod.CodeGraph;
const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler;
const dir = mkdtempSync(join(tmpdir(), 'cg-factory-'));
cpSync(FIXTURE, dir, { recursive: true });
rmSync(join(dir, '.codegraph'), { recursive: true, force: true });
let out;
try {
let cg = CodeGraph.initSync(dir);
await cg.indexAll();
// Inner function definitions, straight from the index — the symbols the file's
// enclosing factory range would otherwise swallow.
const nodes = cg.getNodesInFile(TARGET);
const factory = nodes.find((n) => n.name === FACTORY);
const inner = nodes
.filter((n) => (n.kind === 'function' || n.kind === 'method')
&& n.name !== FACTORY
&& factory && n.startLine > factory.startLine && n.endLine <= factory.endLine)
.sort((a, b) => a.startLine - b.startLine);
cg.close?.();
const sidecar = join(dir, 'diag.jsonl');
process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar;
cg = CodeGraph.openSync(dir);
const res = await new ToolHandler(cg).execute('codegraph_explore', { query: QUERY });
const text = res.content?.[0]?.text ?? '';
cg.close?.();
delete process.env.CODEGRAPH_EXPLORE_DEBUG;
const report = JSON.parse(readFileSync(sidecar, 'utf8').trim().split('\n').pop());
// Which source lines of the target file the response actually carries. The
// response numbers every delivered line `<n>\t<text>`; match them back against
// the file so a line number that merely appears in prose can't count.
const source = readFileSync(join(dir, TARGET), 'utf8').split('\n');
const delivered = new Set();
for (const line of text.split('\n')) {
const m = /^(\d+)\t(.*)$/.exec(line);
if (!m) continue;
const n = Number(m[1]);
if (n >= 1 && n <= source.length && source[n - 1] === m[2]) delivered.add(n);
}
// Collapse to ranges for display.
const ranges = [];
for (const n of [...delivered].sort((a, b) => a - b)) {
const last = ranges[ranges.length - 1];
if (last && n === last.end + 1) last.end = n;
else ranges.push({ start: n, end: n });
}
const covered = (n) => delivered.has(n.startLine);
const rec = report.files.find((f) => f.path === TARGET) ?? null;
out = {
query: QUERY,
target: TARGET,
fileLines: source.length,
factory: factory ? { name: factory.name, start: factory.startLine, end: factory.endLine } : null,
file: rec && {
rank: rec.rank, render: rec.render, clipped: rec.clipped,
emittedChars: rec.emittedChars, finalChars: rec.finalChars,
allowance: rec.allowance, spendable: rec.spendable, skipped: rec.skipped,
},
deliveredRanges: ranges,
deliveredLines: delivered.size,
inner: inner.map((n) => ({ name: n.name, start: n.startLine, end: n.endLine, delivered: covered(n) })),
innerDelivered: inner.filter(covered).length,
innerTotal: inner.length,
envelope: report.envelope,
allFiles: report.files
.filter((f) => f.emittedChars > 0 || f.finalChars > 0)
.map((f) => ({ rank: f.rank, path: f.path, render: f.render, emitted: f.emittedChars, final: f.finalChars })),
};
} finally {
rmSync(dir, { recursive: true, force: true });
}
if (asJson) {
console.log(JSON.stringify(out, null, 2));
} else {
say(`query "${out.query}"`);
say(`target ${out.target} — ${out.fileLines} lines, factory ${out.factory?.name} spans ${out.factory?.start}–${out.factory?.end}`);
say('');
say(' # render emitted final file');
for (const f of out.allFiles) {
say(` ${String(f.rank).padStart(2)} ${(f.render ?? '-').padEnd(10)} ${num(f.emitted).padStart(7)} ${num(f.final).padStart(7)} ${f.path}`);
}
say('');
say(`delivered lines of ${out.target}: ${out.deliveredLines}`);
say(` ranges: ${out.deliveredRanges.map((r) => `${r.start}-${r.end}`).join(', ') || '(none)'}`);
say('');
say(`inner symbols whose definition reached the agent: ${out.innerDelivered}/${out.innerTotal}`);
for (const n of out.inner) {
say(` ${n.delivered ? '✓' : '·'} ${n.name} (${n.start}–${n.end})`);
}
}
scripts/agent-eval/probe-file-spend.mjs
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env node
/**
* Per-file reservation-vs-delivered sweep for `codegraph_explore` (CG-36).
*
* `probe-suite-envelope.mjs` answers "how much source did the response deliver";
* this answers the question one level down — "did the bytes go to the files that
* earned them". The CG-36 defect was invisible to the envelope probe because the
* envelope stayed full: a rank-#3 file spent 24% of its reservation, the slack
* carried forward exactly as designed, and a far weaker file spent 3.5x its own.
* The response looked healthy; the ANSWER-bearing file had been starved.
*
* So the flag here is a PAIR, not a per-file threshold: a file that leaves a
* large share of its reservation unspent WHILE a materially lower-scoring file
* spends well over its own. Either alone is legitimate — a small file simply has
* less to say, and carry-forward is the mechanism that hands its slack down.
*
* Numbers come from the CG-4 diagnostic (`CODEGRAPH_EXPLORE_DEBUG`), so this
* measures the shipping allocator rather than re-deriving shares from markdown.
*
* Usage (needs a current `npm run build`, and full-REBUILT indexes — CG-33):
* node scripts/agent-eval/probe-file-spend.mjs
* node scripts/agent-eval/probe-file-spend.mjs --json > /tmp/new.json
* node scripts/agent-eval/probe-file-spend.mjs --baseline /tmp/base.json
* node scripts/agent-eval/probe-file-spend.mjs django --all # every file, not just flags
* CORPUS=/tmp/codegraph-corpus node scripts/agent-eval/probe-file-spend.mjs
*
* Exit code is 1 when any repo carries a starvation flag, so this can gate.
*/
import { mkdtempSync, readFileSync, rmSync, existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
const CORPUS = process.env.CORPUS ?? '/tmp/codegraph-corpus';
/** Same six repos and queries the CG-30/CG-31/CG-26 envelope tables use. */
const SUITE = [
{ id: 'django', q: 'How does a QuerySet turn into SQL and fetch rows from the database?' },
{ id: 'excalidraw', q: 'How does updating an element re-render the canvas on screen?' },
{ id: 'okhttp', q: 'How does a call go through the interceptor chain to the network?' },
{ id: 'tokio', q: 'How does a spawned task get scheduled and run by a worker?' },
{ id: 'gin', q: 'How does a registered route handler get invoked for an incoming HTTP request?' },
{ id: 'alamofire', q: 'How does a request get built and sent through the session?' },
];
/**
* Starvation thresholds. A flag needs BOTH sides — the starved file and the
* overspending one it lost the bytes to.
*
* `MIN_RESERVED` keeps the noise out: under it, "80% unspent" is a few hundred
* chars and means nothing. `SCORE_RATIO` is what makes the pair meaningful —
* a higher-scoring file underspending while a *comparable* one overspends is
* ordinary; the defect is a materially weaker file taking the bytes.
*/
const STARVED_SHARE = 0.5; // spent < half its reservation
const OVERSPEND_RATIO = 1.5; // spent > 1.5x its own reservation
const SCORE_RATIO = 2; // ...while scoring less than half the starved file
const MIN_RESERVED = 2000; // ignore files whose reservation is too small to matter
const argv = process.argv.slice(2);
const asJson = argv.includes('--json');
const showAll = argv.includes('--all');
const baselineAt = argv.includes('--baseline') ? argv[argv.indexOf('--baseline') + 1] : null;
const only = argv.filter((a) => !a.startsWith('--') && a !== baselineAt);
const say = (s = '') => { if (!asJson) console.log(s); };
const num = (n) => Math.round(n).toLocaleString('en-US');
const pct = (f) => `${(f * 100).toFixed(1)}%`;
const load = (rel) => import(pathToFileURL(resolve(rel)).href);
const idx = await load('dist/index.js');
const toolsMod = await load('dist/mcp/tools.js');
const CodeGraph = idx.default?.default ?? idx.default ?? idx.CodeGraph;
const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler;
if (typeof CodeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') {
console.error('could not resolve CodeGraph/ToolHandler from dist/ — run `npm run build`');
process.exit(2);
}
/**
* Pair up the starved with the overspenders they lost bytes to. Only files the
* render loop actually reached (a reservation and a render mode) take part —
* a cliffed or max-files file never had bytes to spend.
*/
function findStarvation(files) {
const spenders = files.filter(
(f) => f.allowance !== null && f.allowance > 0 && f.render && f.render !== 'backref',
);
const flags = [];
for (const s of spenders) {
if (s.allowance < MIN_RESERVED) continue;
if (s.finalChars >= s.allowance * STARVED_SHARE) continue;
for (const o of spenders) {
if (o.path === s.path) continue;
if (o.finalChars <= o.allowance * OVERSPEND_RATIO) continue;
if (o.score * SCORE_RATIO > s.score) continue;
flags.push({
starved: s.path,
starvedScore: s.score,
starvedReserved: s.allowance,
starvedSpent: s.finalChars,
overspent: o.path,
overspentScore: o.score,
overspentReserved: o.allowance,
overspentSpent: o.finalChars,
});
}
}
return flags;
}
const tmp = mkdtempSync(join(tmpdir(), 'cg-spend-'));
const results = [];
try {
for (const { id, q } of SUITE) {
if (only.length > 0 && !only.includes(id)) continue;
const repo = join(CORPUS, id);
if (!existsSync(join(repo, '.codegraph', 'codegraph.db'))) {
say(`${id}: no index at ${repo} — skipped`);
continue;
}
const sidecar = join(tmp, `${id}.jsonl`);
process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar;
const cg = CodeGraph.openSync(repo);
const h = new ToolHandler(cg);
await h.execute('codegraph_explore', { query: q });
try { cg.close?.(); } catch { /* best effort */ }
const report = JSON.parse(readFileSync(sidecar, 'utf8').trim().split('\n').pop());
const files = report.files.map((f) => ({
path: f.path,
rank: f.rank,
score: f.score,
allowance: f.allowance,
spendable: f.spendable,
finalChars: f.finalChars,
render: f.render,
skipped: f.skipped,
spent: f.allowance ? f.finalChars / f.allowance : null,
}));
results.push({
repo: id,
sourceChars: report.envelope.sourceChars,
files,
flags: findStarvation(files),
});
}
} finally {
rmSync(tmp, { recursive: true, force: true });
}
if (asJson) {
console.log(JSON.stringify(results, null, 2));
} else {
const base = baselineAt ? JSON.parse(readFileSync(baselineAt, 'utf8')) : null;
const byRepo = new Map((base ?? []).map((r) => [r.repo, r]));
for (const r of results) {
const b = byRepo.get(r.repo);
say(`\n${r.repo} — ${num(r.sourceChars)} source chars`
+ (b ? ` (baseline ${num(b.sourceChars)})` : ''));
say(' # score reserved spent spent% render file');
say('-'.repeat(96));
const flagged = new Set(r.flags.flatMap((f) => [f.starved, f.overspent]));
for (const f of r.files) {
if (f.allowance === null || f.allowance === 0) continue;
if (!showAll && !flagged.has(f.path) && f.spent > STARVED_SHARE && f.spent < OVERSPEND_RATIO) continue;
const mark = flagged.has(f.path) ? '*' : ' ';
say(
`${String(f.rank).padStart(2)}${mark} ${String(Math.round(f.score)).padStart(6)} `
+ `${num(f.allowance).padStart(9)} ${num(f.finalChars).padStart(7)} `
+ `${pct(f.spent).padStart(7)} ${(f.render ?? f.skipped ?? '—').padEnd(13)} ${f.path}`,
);
}
for (const f of r.flags) {
say(` FLAG: ${f.starved} (score ${Math.round(f.starvedScore)}) spent `
+ `${num(f.starvedSpent)}/${num(f.starvedReserved)} while ${f.overspent} `
+ `(score ${Math.round(f.overspentScore)}) spent ${num(f.overspentSpent)}/${num(f.overspentReserved)}`);
}
}
const total = results.reduce((n, r) => n + r.flags.length, 0);
say('');
say(total === 0
? 'No file leaves a large share of its reservation unspent while a weaker file overspends.'
: `STARVATION: ${total} flag(s) across `
+ `${results.filter((r) => r.flags.length > 0).map((r) => r.repo).join(', ')}.`);
if (base) {
const worse = results.filter((r) => {
const b = byRepo.get(r.repo);
return b && (r.flags.length > b.flags.length || r.sourceChars < b.sourceChars);
});
say(worse.length === 0
? 'No repo flags more or delivers less than the baseline.'
: `REGRESSION vs baseline: ${worse.map((r) => r.repo).join(', ')}.`);
}
if (total > 0) process.exitCode = 1;
}
scripts/agent-eval/probe-suite-envelope.mjs
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env node
/**
* Deterministic 6-repo envelope sweep for `codegraph_explore` (CG-26).
*
* The allocation issues (CG-30 / CG-31 / CG-26) are all decided by how the
* render loop divides a fixed byte ceiling, and the agent A/B is far too noisy
* to see a 2K byte shift. This runs the SAME six queries the CG-30 and CG-31
* benchmark tables use, against the same clean-rebuilt corpus indexes, and
* prints the numbers those tables are made of: source chars delivered, files in
* the final output, whether the hard ceiling cut anything, and whether the
* epilogue survived.
*
* Numbers come from the CG-4 diagnostic (`CODEGRAPH_EXPLORE_DEBUG`), so this
* measures the shipping allocator rather than re-deriving shares from markdown.
*
* Usage (needs a current `npm run build`, and full-REBUILT indexes — CG-33):
* node scripts/agent-eval/probe-suite-envelope.mjs
* node scripts/agent-eval/probe-suite-envelope.mjs --json > /tmp/new.json
* node scripts/agent-eval/probe-suite-envelope.mjs --baseline /tmp/base.json
* CORPUS=/tmp/codegraph-corpus node scripts/agent-eval/probe-suite-envelope.mjs
*/
import { mkdtempSync, readFileSync, rmSync, existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
const CORPUS = process.env.CORPUS ?? '/tmp/codegraph-corpus';
/** The six suite repos + the exact queries the CG-30/CG-31 tables were measured on. */
const SUITE = [
{ id: 'django', q: 'How does a QuerySet turn into SQL and fetch rows from the database?' },
{ id: 'excalidraw', q: 'How does updating an element re-render the canvas on screen?' },
{ id: 'okhttp', q: 'How does a call go through the interceptor chain to the network?' },
{ id: 'tokio', q: 'How does a spawned task get scheduled and run by a worker?' },
{ id: 'gin', q: 'How does a registered route handler get invoked for an incoming HTTP request?' },
{ id: 'alamofire', q: 'How does a request get built and sent through the session?' },
];
const argv = process.argv.slice(2);
const asJson = argv.includes('--json');
const baselineAt = argv.includes('--baseline') ? argv[argv.indexOf('--baseline') + 1] : null;
const only = argv.filter((a) => !a.startsWith('--') && a !== baselineAt);
const say = (s = '') => { if (!asJson) console.log(s); };
const num = (n) => Math.round(n).toLocaleString('en-US');
const load = (rel) => import(pathToFileURL(resolve(rel)).href);
const idx = await load('dist/index.js');
const toolsMod = await load('dist/mcp/tools.js');
const CodeGraph = idx.default?.default ?? idx.default ?? idx.CodeGraph;
const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler;
if (typeof CodeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') {
console.error('could not resolve CodeGraph/ToolHandler from dist/ — run `npm run build`');
process.exit(2);
}
const tmp = mkdtempSync(join(tmpdir(), 'cg-suite-'));
const results = [];
try {
for (const { id, q } of SUITE) {
if (only.length > 0 && !only.includes(id)) continue;
const repo = join(CORPUS, id);
if (!existsSync(join(repo, '.codegraph', 'codegraph.db'))) {
say(`${id}: no index at ${repo} — skipped`);
continue;
}
const sidecar = join(tmp, `${id}.jsonl`);
process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar;
const cg = CodeGraph.openSync(repo);
const h = new ToolHandler(cg);
const res = await h.execute('codegraph_explore', { query: q });
const text = res.content?.[0]?.text ?? '';
try { cg.close?.(); } catch { /* best effort */ }
const report = JSON.parse(readFileSync(sidecar, 'utf8').trim().split('\n').pop());
results.push({
repo: id,
sourceChars: report.envelope.sourceChars,
envelopeChars: report.envelope.chars,
allocatedChars: report.envelope.allocatedChars,
hardCeiling: report.budget.hardCeiling,
truncated: report.envelope.truncated,
files: report.selection.filesInFinalOutput,
// Did the response keep its trailing pointer list / notes, or did the
// hard ceiling spend them? This is CG-26's residual 1.
epilogueCut: text.includes('omitted for size'),
sectionCut: text.includes('output truncated to budget'),
notShown: text.includes('Not shown above'),
budgetNote: text.includes('**Explore budget:'),
});
}
} finally {
rmSync(tmp, { recursive: true, force: true });
}
if (asJson) {
console.log(JSON.stringify(results, null, 2));
} else {
const base = baselineAt ? JSON.parse(readFileSync(baselineAt, 'utf8')) : null;
const byRepo = new Map((base ?? []).map((r) => [r.repo, r]));
say('repo source Δ env files cut epilogue');
say('-'.repeat(74));
for (const r of results) {
const b = byRepo.get(r.repo);
const delta = b ? (r.sourceChars - b.sourceChars) : null;
const dStr = delta === null ? '' : (delta > 0 ? `+${num(delta)}` : num(delta));
const cut = r.sectionCut ? 'section' : r.epilogueCut ? 'epilogue' : '—';
const epi = [r.notShown ? 'not-shown' : null, r.budgetNote ? 'budget-note' : null]
.filter(Boolean).join('+') || 'none';
say(
`${r.repo.padEnd(12)} ${num(r.sourceChars).padStart(7)} ${dStr.padStart(8)} `
+ `${num(r.envelopeChars).padStart(7)} ${String(r.files).padStart(5)} ${cut.padEnd(12)} ${epi}`,
);
}
if (base) {
const lost = results.filter((r) => {
const b = byRepo.get(r.repo);
return b && (r.sourceChars < b.sourceChars || r.files < b.files);
});
say('');
say(lost.length === 0
? 'No repo delivers less source or fewer files than the baseline.'
: `REGRESSION: ${lost.map((r) => r.repo).join(', ')} deliver less than baseline.`);
}
}