diff --git a/CHANGELOG.md b/CHANGELOG.md index 6534bde..152c27e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Python classes used as values — `return SomeSerializer` from a factory method, `handler = SomeClass` aliases, registry dicts and lists, and classes passed as arguments — now produce reference edges in the graph. Previously these idioms were invisible, so on Django and Django REST Framework projects, asking for a serializer's callers or the impact of editing it missed the views that actually use it. Re-index after upgrading to pick up the new edges. (#1478) - When a file changed on disk after its last index sync, `codegraph_node` and `codegraph_explore` could return a different symbol's code under the requested name — current file bytes cut at outdated line positions — while presenting it as verbatim, trustworthy source. This hit hardest on projects queried through `projectPath` (for example, sub-projects of a monorepo), which have no live file watcher to flag pending edits. Both tools now verify each file against the index before showing sliced code: an out-of-date file is either shown whole with its full current source, or its code is withheld with a clear "changed on disk" notice — never served as a wrong slice. A fresh re-index restores normal output automatically. Thanks @inth3shadows for the thorough report and verification passes. (#1474) - A file built around one very long function no longer takes the whole `codegraph_explore` answer for itself — or disappears from it. Previously such a file was shown in full however big it was, which used up the room every file after it needed, and when the function was larger than the entire response the file was dropped without a word. These files now come back as a bounded window on whole lines — the signature and the top of the body, plus the call site when the call path runs through it — with the rest one follow-up `codegraph_explore` away. +- `codegraph_explore` no longer lets the first file in an answer spend the room set aside for the files below it, so the rest of the answer still arrives. Previously a large file near the top could quietly use up everything left, and the files ranked under it — each already judged relevant enough to include — were dropped with no source at all; on one question only one of six made it into the answer. Every file now keeps what it was given, and a question that really is about one file still concentrates on that file. - The blast-radius section of `codegraph_explore` flagged "no covering tests found" whenever no test called a symbol directly — falsely branding helpers that tests exercise through their callers as untested (about 40% of flagged symbols in a measured sample). The check now follows caller chains up to 3 hops and reports indirect coverage as "tested via callers"; when nothing is found it states exactly what was checked instead of an unconditional warning. Thanks @inth3shadows for measuring the false-positive rate. (#1475) ## [1.5.0] - 2026-07-21 diff --git a/__tests__/explore-displacement-guard.test.ts b/__tests__/explore-displacement-guard.test.ts new file mode 100644 index 0000000..aebf792 --- /dev/null +++ b/__tests__/explore-displacement-guard.test.ts @@ -0,0 +1,234 @@ +/** + * Regression fixture for CG-31 — a clustered render may not spend a reservation + * still owed to a file the loop has not reached. + * + * The allocator hands every admitted file a reservation (CG-12), and the render + * loop then walks the files in rank order. Carry-forward slack lets a file spend + * what the files ABOVE it left on the table, which is right; what was missing is + * the other half — nothing was held back for the files BELOW it. The whole-file + * BUY arm has always refused that trade (`owedBelow`, `tools.ts`); the cluster + * path had no equivalent, so `fileBudget`/`SPINE_CEILING` read what was left + * before the hard ceiling rather than what was still promised, and the first + * oversize file could take the response. + * + * `__tests__/fixtures/displacement-ts/` reproduces it. Four pipeline stages + * compete for one envelope; the first, `ingest.ts`, is a single ~20K function — + * one cluster member far bigger than any reservation it can earn — so it takes + * the bounded overshoot CG-30 left it. The fixture is padded to >500 indexed + * files on purpose: the displacement only exists on the 24K tier, where the + * reservations plus the response preamble genuinely saturate the hard ceiling. + * + * Measured against the pre-fix build (CG-30 landed, CG-31 not): + * + * ingest.ts 9,301 chars emitted on a 6,289 spendable — then dropped whole + * by the final ceiling, so it cost the response and delivered 0 + * types.ts skipped `budget-whole-file` + * sink.ts skipped `budget-whole-file` + * delivered 3 of 6 admitted files, 14,908-char envelope + * + * With the guard: 6 of 6, 22,066-char envelope, and `ingest.ts` bounded to the + * 4,913 that were actually still free. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler } from '../src/mcp/tools'; +import { attributeSourceBytes } from '../src/mcp/explore-diagnostics'; +import type { ExploreDiagnosticReport, ExploreDiagnosticFile } from '../src/mcp/explore-diagnostics'; + +const FIXTURE_SRC = path.join(__dirname, 'fixtures', 'displacement-ts'); + +/** + * Padding modules, written into the temp copy rather than checked in. The + * output tier is chosen by INDEXED FILE COUNT, and the displacement this test + * pins only exists at >=500 files (24K envelope against a 24.4K render ceiling + * that also has to hold the response preamble). Below that the ceiling has + * enough slack to absorb an overshoot and the bug is invisible. + */ +const FILLER_FILES = 520; + +/** A symbol bag spanning all four stages — they compete for one envelope. */ +const QUERY = 'ingestRecords normalizeRecords enrichRecords publishRecords'; +/** One symbol, one file — the concentration case the guard must not flatten. */ +const PRECISE_QUERY = 'ingestRecords'; + +/** The giant: one ~20K function, the file that used to take the response. */ +const GIANT = 'src/pipeline/ingest.ts'; +/** Ranked below the giant and dropped by it pre-fix. */ +const STARVED = ['src/pipeline/types.ts', 'src/pipeline/sink.ts']; + +interface Probe { + response: string; + report: ExploreDiagnosticReport; + bytes: Map; +} + +describe('CG-31 — the cluster path holds back what is still owed below it', () => { + let testDir: string; + let cg: CodeGraph; + let spread: Probe; + let precise: Probe; + + const fileOf = (probe: Probe, p: string): ExploreDiagnosticFile => { + const rec = probe.report.files.find((f) => f.path === p); + if (!rec) throw new Error(`${p} absent from the diagnostic report`); + return rec; + }; + /** Admitted = the allocator reserved bytes for it. */ + const admitted = (probe: Probe): ExploreDiagnosticFile[] => + probe.report.files.filter((f) => (f.allowance ?? 0) > 0); + + beforeAll(async () => { + testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cg31-')); + fs.cpSync(FIXTURE_SRC, testDir, { recursive: true }); + fs.rmSync(path.join(testDir, '.codegraph'), { recursive: true, force: true }); + + const filler = path.join(testDir, 'src', 'generated'); + fs.mkdirSync(filler, { recursive: true }); + for (let i = 0; i < FILLER_FILES; i++) { + // Deterministic, unrelated to the query — these pad the file count, they + // must never rank. + fs.writeFileSync( + path.join(filler, `unit${i}.ts`), + `export const seed${i} = ${i};\n` + + `export function widget${i}(n: number): number {\n return n * ${i + 1} + seed${i};\n}\n`, + ); + } + + cg = CodeGraph.initSync(testDir); + await cg.indexAll(); + + // The per-file bounds are only observable through the diagnostic sidecar. + const sidecar = path.join(testDir, 'explore-diag.jsonl'); + const previous = process.env.CODEGRAPH_EXPLORE_DEBUG; + process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar; + const run = async (handler: ToolHandler, query: string): Promise => { + const result = await handler.execute('codegraph_explore', { query }); + const response = result.content?.[0]?.text ?? ''; + const written = fs.readFileSync(sidecar, 'utf-8').trim().split('\n').filter(Boolean); + return { + response, + report: JSON.parse(written[written.length - 1]!) as ExploreDiagnosticReport, + bytes: attributeSourceBytes(response), + }; + }; + try { + const handler = new ToolHandler(cg); + spread = await run(handler, QUERY); + precise = await run(handler, PRECISE_QUERY); + } finally { + if (previous === undefined) delete process.env.CODEGRAPH_EXPLORE_DEBUG; + else process.env.CODEGRAPH_EXPLORE_DEBUG = previous; + } + }, 180_000); + + afterAll(() => { + if (cg) cg.destroy(); + if (testDir && fs.existsSync(testDir)) fs.rmSync(testDir, { recursive: true, force: true }); + }); + + // ── Fixture shape — if these rot, the gate below means nothing ───────────── + + describe('fixture shape', () => { + it('sits on the 24K tier, where the reservations saturate the ceiling', () => { + expect(cg.getStats().fileCount).toBeGreaterThanOrEqual(500); + expect(spread.report.budget.maxOutputChars).toBe(24000); + }); + + it('admits every stage file, so there is something to displace', () => { + const paths = admitted(spread).map((f) => f.path); + expect(paths).toContain(GIANT); + for (const p of STARVED) expect(paths).toContain(p); + expect(paths.length).toBeGreaterThanOrEqual(5); + }); + + it('renders the giant through the CLUSTER path, over its reservation', () => { + const rec = fileOf(spread, GIANT); + expect(rec.render).toBe('clusters'); + // One member bigger than anything it can earn beside its siblings — the + // shape that makes the bounded overshoot fire at all. + const source = fs.readFileSync(path.join(testDir, GIANT), 'utf-8'); + expect(source.length).toBeGreaterThan((rec.spendable ?? 0) * 2); + // And the guard actually bit — a vacuous pass here would hide a regression. + expect(rec.funded).not.toBeNull(); + expect(rec.funded!).toBeLessThan(rec.spendable!); + }); + }); + + // ── The gate ────────────────────────────────────────────────────────────── + + describe('displacement refusal', () => { + it('CG-31 GATE: no clustered file emits past what was still free to spend', () => { + for (const probe of [spread, precise]) { + const over = probe.report.files + .filter((f) => f.render === 'clusters' && f.funded !== null) + // +1 for the render loop's own rounding on the windowed cut. + .filter((f) => f.emittedChars > f.funded! + 1) + .map((f) => `${f.path}: ${f.emittedChars} of ${f.funded}`); + expect(over).toEqual([]); + } + }); + + it('CG-31 GATE: every admitted file below the top one is delivered', () => { + // Pre-fix: 3 of 6 — `ingest.ts` overshot, was itself cut by the final + // ceiling, and took `types.ts` + `sink.ts` down with it. + for (const rec of admitted(spread)) { + expect(rec.skipped, `${rec.path} skipped`).toBeNull(); + expect(spread.bytes.get(rec.path) ?? 0, `${rec.path} bytes`).toBeGreaterThan(0); + } + for (const p of STARVED) expect(spread.bytes.get(p) ?? 0).toBeGreaterThan(0); + }); + + it('the guard is symmetric — it is about ORDER, not rank', () => { + // Nothing here protects rank #1 specifically: the LAST admitted file, the + // only one with no reservation owed below it, is delivered too. + const files = admitted(spread); + const last = files[files.length - 1]!; + expect(last.skipped).toBeNull(); + expect(spread.bytes.get(last.path) ?? 0).toBeGreaterThan(0); + // And the last file is never itself cut by the guard — nothing is owed + // below it, so `funded` may not sit under its own reservation. + expect(last.funded!).toBeGreaterThanOrEqual(Math.min(last.allowance!, last.emittedChars)); + }); + + it('a kept promise is not a displacement — no file is cut below its reservation', () => { + for (const probe of [spread, precise]) { + for (const rec of admitted(probe)) { + if (rec.funded === null) continue; + expect(rec.funded, rec.path).toBeGreaterThanOrEqual( + Math.min(rec.allowance!, rec.emittedChars)); + } + } + }); + + it('keeps the response inside the hard ceiling', () => { + for (const probe of [spread, precise]) { + expect(probe.report.envelope.chars).toBeLessThanOrEqual(probe.report.budget.hardCeiling); + } + }); + }); + + // ── The thing the guard must NOT become ─────────────────────────────────── + + describe('concentration survives', () => { + it('a precise symbol query still puts the most source in the named file', () => { + const mine = precise.bytes.get(GIANT) ?? 0; + const others = [...precise.bytes.entries()].filter(([p]) => p !== GIANT); + expect(mine).toBeGreaterThan(0); + for (const [p, n] of others) { + expect(mine, `${GIANT} vs ${p}`).toBeGreaterThan(n); + } + // Not a forced even split: the named file takes a clear plurality. + const total = [...precise.bytes.values()].reduce((s, n) => s + n, 0); + expect(mine / total).toBeGreaterThan(1 / precise.bytes.size); + }); + + it('the named file still outspends what it would get from an even split', () => { + const rec = fileOf(precise, GIANT); + const even = precise.report.budget.maxOutputChars / admitted(precise).length; + expect(rec.emittedChars).toBeGreaterThan(even); + }); + }); +}); diff --git a/__tests__/fixtures/displacement-ts/package.json b/__tests__/fixtures/displacement-ts/package.json new file mode 100644 index 0000000..1998004 --- /dev/null +++ b/__tests__/fixtures/displacement-ts/package.json @@ -0,0 +1,6 @@ +{ + "name": "displacement-fixture", + "version": "1.0.0", + "private": true, + "type": "module" +} diff --git a/__tests__/fixtures/displacement-ts/src/index.ts b/__tests__/fixtures/displacement-ts/src/index.ts new file mode 100644 index 0000000..c6eadf0 --- /dev/null +++ b/__tests__/fixtures/displacement-ts/src/index.ts @@ -0,0 +1,10 @@ +import { ingestRecords } from './pipeline/ingest'; +import { normalizeRecords } from './pipeline/normalize'; +import { enrichRecords } from './pipeline/enrich'; +import { publishRecords } from './pipeline/publish'; +import type { PipelineOptions, PipelineRecord, RawRecord } from './pipeline/types'; + +/** Run one batch through every pipeline stage, in order. */ +export function runPipeline(batch: RawRecord[], options: PipelineOptions): PipelineRecord[] { + return publishRecords(enrichRecords(normalizeRecords(ingestRecords(batch, options), options), options), options); +} diff --git a/__tests__/fixtures/displacement-ts/src/pipeline/enrich.ts b/__tests__/fixtures/displacement-ts/src/pipeline/enrich.ts new file mode 100644 index 0000000..ae098aa --- /dev/null +++ b/__tests__/fixtures/displacement-ts/src/pipeline/enrich.ts @@ -0,0 +1,127 @@ +import { writeBatch } from './sink'; +import type { PipelineOptions, PipelineRecord } from './types'; + +/** Enrich every record in a batch. */ +export function enrichRecords(records: PipelineRecord[], options: PipelineOptions): PipelineRecord[] { + const out: PipelineRecord[] = []; + for (const record of records) { + const tags = [...record.tags]; + const warnings = [...record.warnings]; + let value = record.value; + + // 1. segment + { + const hit = tags.find((t) => t.startsWith('segment:')); + if (hit === undefined) { + if (options.strict) warnings.push('segment: missing after enrich'); + } else { + value = weightFacet(value, hit.length); + tags.push('segment.enri'); + } + } + + // 2. referrer + { + const hit = tags.find((t) => t.startsWith('referrer:')); + if (hit === undefined) { + if (options.strict) warnings.push('referrer: missing after enrich'); + } else { + value = blendFacet(value, hit.length); + tags.push('referrer.enri'); + } + } + + // 3. experiment + { + const hit = tags.find((t) => t.startsWith('experiment:')); + if (hit === undefined) { + if (options.strict) warnings.push('experiment: missing after enrich'); + } else { + value = weightFacet(value, hit.length); + tags.push('experiment.enri'); + } + } + + // 4. subscription + { + const hit = tags.find((t) => t.startsWith('subscription:')); + if (hit === undefined) { + if (options.strict) warnings.push('subscription: missing after enrich'); + } else { + value = blendFacet(value, hit.length); + tags.push('subscription.enri'); + } + } + + // 5. entitlement + { + const hit = tags.find((t) => t.startsWith('entitlement:')); + if (hit === undefined) { + if (options.strict) warnings.push('entitlement: missing after enrich'); + } else { + value = weightFacet(value, hit.length); + tags.push('entitlement.enri'); + } + } + + // 6. invoice + { + const hit = tags.find((t) => t.startsWith('invoice:')); + if (hit === undefined) { + if (options.strict) warnings.push('invoice: missing after enrich'); + } else { + value = blendFacet(value, hit.length); + tags.push('invoice.enri'); + } + } + + // 7. refund + { + const hit = tags.find((t) => t.startsWith('refund:')); + if (hit === undefined) { + if (options.strict) warnings.push('refund: missing after enrich'); + } else { + value = weightFacet(value, hit.length); + tags.push('refund.enri'); + } + } + + // 8. dispute + { + const hit = tags.find((t) => t.startsWith('dispute:')); + if (hit === undefined) { + if (options.strict) warnings.push('dispute: missing after enrich'); + } else { + value = blendFacet(value, hit.length); + tags.push('dispute.enri'); + } + } + + // 9. payout + { + const hit = tags.find((t) => t.startsWith('payout:')); + if (hit === undefined) { + if (options.strict) warnings.push('payout: missing after enrich'); + } else { + value = weightFacet(value, hit.length); + tags.push('payout.enri'); + } + } + + out.push({ ...record, value, tags: tags.slice(0, options.maxTags), warnings }); + } + writeBatch('enrichRecords', out); + return out; +} + +/** weightFacet — a small deterministic helper. */ +export function weightFacet(base: number, width: number): number { + const scaled = base + width * 3 - (width % 7); + return scaled < 0 ? 0 : scaled; +} + +/** blendFacet — a small deterministic helper. */ +export function blendFacet(base: number, width: number): number { + const scaled = base + width * 3 - (width % 7); + return scaled < 0 ? 0 : scaled; +} diff --git a/__tests__/fixtures/displacement-ts/src/pipeline/ingest.ts b/__tests__/fixtures/displacement-ts/src/pipeline/ingest.ts new file mode 100644 index 0000000..eccaee7 --- /dev/null +++ b/__tests__/fixtures/displacement-ts/src/pipeline/ingest.ts @@ -0,0 +1,541 @@ +import { scaleFacet, clampFacet } from './normalize'; +import { writeBatch } from './sink'; +import type { PipelineOptions, PipelineRecord, RawRecord } from './types'; + +/** + * Ingest one batch of raw records. + * + * Every facet is unpacked in its own block so an on-call engineer can read the + * ingest end-to-end in one place. The shape is deliberately flat: this single + * function is the whole stage, which is exactly the shape that makes it the + * biggest cluster member in the file. + */ +export function ingestRecords(batch: RawRecord[], options: PipelineOptions): PipelineRecord[] { + const out: PipelineRecord[] = []; + for (const record of batch) { + const tags: string[] = []; + const warnings: string[] = []; + let value = 0; + + // 1. identity — normalise the identity facet of the record. + { + const raw = record.payload['identity']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('identity: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('identity:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('identity: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 2. geography — normalise the geography facet of the record. + { + const raw = record.payload['geography']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('geography: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('geography:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('geography: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 3. currency — normalise the currency facet of the record. + { + const raw = record.payload['currency']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('currency: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('currency:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('currency: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 4. timestamp — normalise the timestamp facet of the record. + { + const raw = record.payload['timestamp']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('timestamp: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('timestamp:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('timestamp: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 5. channel — normalise the channel facet of the record. + { + const raw = record.payload['channel']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('channel: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('channel:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('channel: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 6. campaign — normalise the campaign facet of the record. + { + const raw = record.payload['campaign']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('campaign: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('campaign:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('campaign: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 7. device — normalise the device facet of the record. + { + const raw = record.payload['device']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('device: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('device:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('device: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 8. locale — normalise the locale facet of the record. + { + const raw = record.payload['locale']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('locale: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('locale:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('locale: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 9. consent — normalise the consent facet of the record. + { + const raw = record.payload['consent']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('consent: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('consent:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('consent: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 10. segment — normalise the segment facet of the record. + { + const raw = record.payload['segment']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('segment: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('segment:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('segment: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 11. referrer — normalise the referrer facet of the record. + { + const raw = record.payload['referrer']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('referrer: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('referrer:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('referrer: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 12. experiment — normalise the experiment facet of the record. + { + const raw = record.payload['experiment']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('experiment: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('experiment:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('experiment: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 13. subscription — normalise the subscription facet of the record. + { + const raw = record.payload['subscription']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('subscription: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('subscription:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('subscription: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 14. entitlement — normalise the entitlement facet of the record. + { + const raw = record.payload['entitlement']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('entitlement: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('entitlement:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('entitlement: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 15. invoice — normalise the invoice facet of the record. + { + const raw = record.payload['invoice']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('invoice: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('invoice:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('invoice: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 16. refund — normalise the refund facet of the record. + { + const raw = record.payload['refund']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('refund: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('refund:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('refund: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 17. dispute — normalise the dispute facet of the record. + { + const raw = record.payload['dispute']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('dispute: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('dispute:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('dispute: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 18. payout — normalise the payout facet of the record. + { + const raw = record.payload['payout']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('payout: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('payout:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('payout: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 19. shipment — normalise the shipment facet of the record. + { + const raw = record.payload['shipment']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('shipment: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('shipment:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('shipment: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 20. inventory — normalise the inventory facet of the record. + { + const raw = record.payload['inventory']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('inventory: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('inventory:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('inventory: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 21. warehouse — normalise the warehouse facet of the record. + { + const raw = record.payload['warehouse']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('warehouse: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('warehouse:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('warehouse: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 22. carrier — normalise the carrier facet of the record. + { + const raw = record.payload['carrier']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('carrier: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('carrier:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('carrier: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 23. customs — normalise the customs facet of the record. + { + const raw = record.payload['customs']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('customs: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('customs:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('customs: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 24. tariff — normalise the tariff facet of the record. + { + const raw = record.payload['tariff']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('tariff: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('tariff:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('tariff: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 25. sensor — normalise the sensor facet of the record. + { + const raw = record.payload['sensor']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('sensor: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('sensor:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('sensor: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 26. firmware — normalise the firmware facet of the record. + { + const raw = record.payload['firmware']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('firmware: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('firmware:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('firmware: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 27. telemetry — normalise the telemetry facet of the record. + { + const raw = record.payload['telemetry']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('telemetry: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('telemetry:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('telemetry: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 28. battery — normalise the battery facet of the record. + { + const raw = record.payload['battery']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('battery: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('battery:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('battery: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 29. network — normalise the network facet of the record. + { + const raw = record.payload['network']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('network: empty, dropped'); + } else { + const scaled = scaleFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('network:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('network: not scalable — ' + text.slice(0, 16)); + } + } + } + + // 30. roaming — normalise the roaming facet of the record. + { + const raw = record.payload['roaming']; + const text = typeof raw === 'string' ? raw.trim() : raw === null ? '' : String(raw); + if (text.length === 0 && options.dropEmpty) { + warnings.push('roaming: empty, dropped'); + } else { + const scaled = clampFacet(text.length, options.maxTags); + if (Number.isFinite(scaled) && scaled !== 0) { + tags.push('roaming:' + text.slice(0, 24)); + value += scaled; + } else if (options.strict) { + warnings.push('roaming: not scalable — ' + text.slice(0, 16)); + } + } + } + + out.push({ + id: record.id, + source: record.source, + kind: options.defaultKind, + value, + tags: tags.slice(0, options.maxTags), + warnings, + }); + } + writeBatch('ingest', out); + return out; +} diff --git a/__tests__/fixtures/displacement-ts/src/pipeline/normalize.ts b/__tests__/fixtures/displacement-ts/src/pipeline/normalize.ts new file mode 100644 index 0000000..5a50a15 --- /dev/null +++ b/__tests__/fixtures/displacement-ts/src/pipeline/normalize.ts @@ -0,0 +1,127 @@ +import { writeBatch } from './sink'; +import type { PipelineOptions, PipelineRecord } from './types'; + +/** Normalize every record in a batch. */ +export function normalizeRecords(records: PipelineRecord[], options: PipelineOptions): PipelineRecord[] { + const out: PipelineRecord[] = []; + for (const record of records) { + const tags = [...record.tags]; + const warnings = [...record.warnings]; + let value = record.value; + + // 1. identity + { + const hit = tags.find((t) => t.startsWith('identity:')); + if (hit === undefined) { + if (options.strict) warnings.push('identity: missing after normalize'); + } else { + value = scaleFacet(value, hit.length); + tags.push('identity.norm'); + } + } + + // 2. geography + { + const hit = tags.find((t) => t.startsWith('geography:')); + if (hit === undefined) { + if (options.strict) warnings.push('geography: missing after normalize'); + } else { + value = clampFacet(value, hit.length); + tags.push('geography.norm'); + } + } + + // 3. currency + { + const hit = tags.find((t) => t.startsWith('currency:')); + if (hit === undefined) { + if (options.strict) warnings.push('currency: missing after normalize'); + } else { + value = scaleFacet(value, hit.length); + tags.push('currency.norm'); + } + } + + // 4. timestamp + { + const hit = tags.find((t) => t.startsWith('timestamp:')); + if (hit === undefined) { + if (options.strict) warnings.push('timestamp: missing after normalize'); + } else { + value = clampFacet(value, hit.length); + tags.push('timestamp.norm'); + } + } + + // 5. channel + { + const hit = tags.find((t) => t.startsWith('channel:')); + if (hit === undefined) { + if (options.strict) warnings.push('channel: missing after normalize'); + } else { + value = scaleFacet(value, hit.length); + tags.push('channel.norm'); + } + } + + // 6. campaign + { + const hit = tags.find((t) => t.startsWith('campaign:')); + if (hit === undefined) { + if (options.strict) warnings.push('campaign: missing after normalize'); + } else { + value = clampFacet(value, hit.length); + tags.push('campaign.norm'); + } + } + + // 7. device + { + const hit = tags.find((t) => t.startsWith('device:')); + if (hit === undefined) { + if (options.strict) warnings.push('device: missing after normalize'); + } else { + value = scaleFacet(value, hit.length); + tags.push('device.norm'); + } + } + + // 8. locale + { + const hit = tags.find((t) => t.startsWith('locale:')); + if (hit === undefined) { + if (options.strict) warnings.push('locale: missing after normalize'); + } else { + value = clampFacet(value, hit.length); + tags.push('locale.norm'); + } + } + + // 9. consent + { + const hit = tags.find((t) => t.startsWith('consent:')); + if (hit === undefined) { + if (options.strict) warnings.push('consent: missing after normalize'); + } else { + value = scaleFacet(value, hit.length); + tags.push('consent.norm'); + } + } + + out.push({ ...record, value, tags: tags.slice(0, options.maxTags), warnings }); + } + writeBatch('normalizeRecords', out); + return out; +} + +/** scaleFacet — a small deterministic helper. */ +export function scaleFacet(base: number, width: number): number { + const scaled = base + width * 3 - (width % 7); + return scaled < 0 ? 0 : scaled; +} + +/** clampFacet — a small deterministic helper. */ +export function clampFacet(base: number, width: number): number { + const scaled = base + width * 3 - (width % 7); + return scaled < 0 ? 0 : scaled; +} diff --git a/__tests__/fixtures/displacement-ts/src/pipeline/publish.ts b/__tests__/fixtures/displacement-ts/src/pipeline/publish.ts new file mode 100644 index 0000000..405f11b --- /dev/null +++ b/__tests__/fixtures/displacement-ts/src/pipeline/publish.ts @@ -0,0 +1,127 @@ +import { writeBatch } from './sink'; +import type { PipelineOptions, PipelineRecord } from './types'; + +/** Publish every record in a batch. */ +export function publishRecords(records: PipelineRecord[], options: PipelineOptions): PipelineRecord[] { + const out: PipelineRecord[] = []; + for (const record of records) { + const tags = [...record.tags]; + const warnings = [...record.warnings]; + let value = record.value; + + // 1. shipment + { + const hit = tags.find((t) => t.startsWith('shipment:')); + if (hit === undefined) { + if (options.strict) warnings.push('shipment: missing after publish'); + } else { + value = rankFacet(value, hit.length); + tags.push('shipment.publ'); + } + } + + // 2. inventory + { + const hit = tags.find((t) => t.startsWith('inventory:')); + if (hit === undefined) { + if (options.strict) warnings.push('inventory: missing after publish'); + } else { + value = sealFacet(value, hit.length); + tags.push('inventory.publ'); + } + } + + // 3. warehouse + { + const hit = tags.find((t) => t.startsWith('warehouse:')); + if (hit === undefined) { + if (options.strict) warnings.push('warehouse: missing after publish'); + } else { + value = rankFacet(value, hit.length); + tags.push('warehouse.publ'); + } + } + + // 4. carrier + { + const hit = tags.find((t) => t.startsWith('carrier:')); + if (hit === undefined) { + if (options.strict) warnings.push('carrier: missing after publish'); + } else { + value = sealFacet(value, hit.length); + tags.push('carrier.publ'); + } + } + + // 5. customs + { + const hit = tags.find((t) => t.startsWith('customs:')); + if (hit === undefined) { + if (options.strict) warnings.push('customs: missing after publish'); + } else { + value = rankFacet(value, hit.length); + tags.push('customs.publ'); + } + } + + // 6. tariff + { + const hit = tags.find((t) => t.startsWith('tariff:')); + if (hit === undefined) { + if (options.strict) warnings.push('tariff: missing after publish'); + } else { + value = sealFacet(value, hit.length); + tags.push('tariff.publ'); + } + } + + // 7. sensor + { + const hit = tags.find((t) => t.startsWith('sensor:')); + if (hit === undefined) { + if (options.strict) warnings.push('sensor: missing after publish'); + } else { + value = rankFacet(value, hit.length); + tags.push('sensor.publ'); + } + } + + // 8. firmware + { + const hit = tags.find((t) => t.startsWith('firmware:')); + if (hit === undefined) { + if (options.strict) warnings.push('firmware: missing after publish'); + } else { + value = sealFacet(value, hit.length); + tags.push('firmware.publ'); + } + } + + // 9. telemetry + { + const hit = tags.find((t) => t.startsWith('telemetry:')); + if (hit === undefined) { + if (options.strict) warnings.push('telemetry: missing after publish'); + } else { + value = rankFacet(value, hit.length); + tags.push('telemetry.publ'); + } + } + + out.push({ ...record, value, tags: tags.slice(0, options.maxTags), warnings }); + } + writeBatch('publishRecords', out); + return out; +} + +/** rankFacet — a small deterministic helper. */ +export function rankFacet(base: number, width: number): number { + const scaled = base + width * 3 - (width % 7); + return scaled < 0 ? 0 : scaled; +} + +/** sealFacet — a small deterministic helper. */ +export function sealFacet(base: number, width: number): number { + const scaled = base + width * 3 - (width % 7); + return scaled < 0 ? 0 : scaled; +} diff --git a/__tests__/fixtures/displacement-ts/src/pipeline/sink.ts b/__tests__/fixtures/displacement-ts/src/pipeline/sink.ts new file mode 100644 index 0000000..2f91720 --- /dev/null +++ b/__tests__/fixtures/displacement-ts/src/pipeline/sink.ts @@ -0,0 +1,18 @@ +import type { PipelineRecord } from './types'; + +const sink = new Map(); + +/** Hand a finished batch to the downstream sink. */ +export function writeBatch(batchId: string, records: PipelineRecord[]): void { + sink.set(batchId, records); +} + +/** Read a batch back out of the sink. */ +export function readBatch(batchId: string): PipelineRecord[] { + return sink.get(batchId) ?? []; +} + +/** Forget a batch. */ +export function dropBatch(batchId: string): void { + sink.delete(batchId); +} diff --git a/__tests__/fixtures/displacement-ts/src/pipeline/types.ts b/__tests__/fixtures/displacement-ts/src/pipeline/types.ts new file mode 100644 index 0000000..32bf283 --- /dev/null +++ b/__tests__/fixtures/displacement-ts/src/pipeline/types.ts @@ -0,0 +1,25 @@ +/** One raw record as it arrives from the upstream feed. */ +export interface RawRecord { + id: string; + source: string; + payload: Record; + receivedAt: number; +} + +/** A record after the pipeline has cleaned and annotated it. */ +export interface PipelineRecord { + id: string; + source: string; + kind: string; + value: number; + tags: string[]; + warnings: string[]; +} + +/** Per-run knobs shared by every pipeline stage. */ +export interface PipelineOptions { + strict: boolean; + dropEmpty: boolean; + defaultKind: string; + maxTags: number; +} diff --git a/scripts/agent-eval/allocation-fixtures.json b/scripts/agent-eval/allocation-fixtures.json index e8b7077..a99d12e 100644 --- a/scripts/agent-eval/allocation-fixtures.json +++ b/scripts/agent-eval/allocation-fixtures.json @@ -4,11 +4,12 @@ "explore budget allocation. Run them with `node scripts/agent-eval/probe-allocation.mjs`", "against a built dist/.", "", - "STATUS: payroll-go PASSES. self-query's delivered-share gates FAIL as of CG-30 —", - "see its `afterCG30` block: the allocated shares are unchanged, but bounding the", - "oversize-member overshoot stopped the hard ceiling from truncating away the", - "incidental file that had been over-RESERVED all along. The over-reservation is", - "epic CG-24's subject (a low-scoring file taking a top-file share), not CG-30's.", + "STATUS: BOTH FIXTURES PASS again as of CG-31. self-query's delivered-share gates", + "failed on the CG-30-only build (`afterCG30`) and CG-31 restored them (`afterCG31`):", + "the incidental file was not over-RESERVED at all — it was over-SPENDING, drawing on", + "the reservations of files the render loop had not reached yet. Bounding that put the", + "response back inside the envelope, so nothing truncates and every admitted file", + "delivers. Read the two blocks together; the CG-30 verdict's diagnosis was wrong.", "", "CG-10 (relevance scoring) closed the RANKING half —", "nothing incidental reaches the envelope any more — and CG-12 (score-proportional", @@ -169,7 +170,18 @@ "src/mcp/explore-session-state.ts": 0.147, "src/resolution/memory-budget.ts": 0.0 }, - "verdict": "THREE GATES FAIL — and the cause is not the CG-30 bound. Allocation is unchanged between arms (parse-run.mjs 32.3% here vs 33.9% on main); what changed is that it now DELIVERS. On main its whole 8,548-char section was cut by the hard-ceiling truncation, so the incidental group scored 0.0% by luck, not by design, and the fixture passed on that. Bounding the oversize-member overshoot freed enough headroom that the response no longer truncates the same section away. Every file obeys the new bound on this repo (max ratio 1.40x of spendable, against the 1.5x ceiling). What the failure exposes is real and pre-existing: parse-run.mjs scores 18 against tools.ts's 58 yet is reserved a comparable slice — a low-scoring file taking a top-file share, which is epic CG-24's subject. Fix it there; do not tune the CG-30 bound to restore a pass that depended on truncation." + "verdict": "THREE GATES FAIL — and the cause is not the CG-30 bound. Allocation is unchanged between arms (parse-run.mjs 32.3% here vs 33.9% on main); what changed is that it now DELIVERS. On main its whole 8,548-char section was cut by the hard-ceiling truncation, so the incidental group scored 0.0% by luck, not by design, and the fixture passed on that. Bounding the oversize-member overshoot freed enough headroom that the response no longer truncates the same section away. Every file obeys the new bound on this repo (max ratio 1.40x of spendable, against the 1.5x ceiling). What the failure exposes is real and pre-existing: parse-run.mjs scores 18 against tools.ts's 58 yet is reserved a comparable slice — a low-scoring file taking a top-file share, which is epic CG-24's subject. Fix it there; do not tune the CG-30 bound to restore a pass that depended on truncation. SUPERSEDED by afterCG31 — the diagnosis above is wrong on one load-bearing point, see there." + }, + "afterCG31": { + "measuredOn": "2026-08-06", + "note": "23,083 delivered of 23,080 allocated — inside the envelope, nothing truncated. Both arms measured on the SAME clean FULL REBUILD of this repo's index (CG-33: an incrementally-synced index diverges and shifts ranking). CG-30-only arm on that index: 23,692 delivered of 26,410 allocated, TRUNCATED.", + "delivered": { + "src/mcp/tools.ts": 0.359, + "scripts/agent-eval/parse-run.mjs": 0.187, + "src/mcp/explore-session-state.ts": 0.151, + "src/resolution/lru-cache.ts": 0.087 + }, + "verdict": "ALL FOUR GATES PASS. The afterCG30 verdict called parse-run.mjs over-RESERVED; it was not — its reservation is 4,314 in both arms. It was over-SPENDING: 8,548 chars, drawing on reservations belonging to files the render loop had not reached yet, which is the CG-31 defect. With the displacement guard it renders 4,314, tools.ts's identical 8,282 chars go from 35.0% to 35.9% of a response that no longer overruns, and lru-cache.ts (dropped as memory-budget.ts was on the CG-30 arm) delivers. Note what did NOT change: allocation. This fixture moved because the render loop stopped spending other files' bytes, not because anything was re-ranked." } } ] diff --git a/src/mcp/explore-diagnostics.ts b/src/mcp/explore-diagnostics.ts index f428f82..758442a 100644 --- a/src/mcp/explore-diagnostics.ts +++ b/src/mcp/explore-diagnostics.ts @@ -103,6 +103,15 @@ interface FileRecord extends ExploreCandidateMeta { * a file spending over its reservation. */ spendable: number | null; + /** + * The DISPLACEMENT-GUARDED bound (CG-31): how much this file may render + * without spending a reservation still owed to a file the loop has not + * reached. `spendable` is what the file was promised, this is what is + * actually still there to pay it with — when it sits below `spendable`, the + * difference is the overshoot the guard refused, and the files below this one + * in the table are the reason. `null` until the render loop reaches the file. + */ + funded: number | null; render?: ExploreRenderMode; /** * Source chars this call did NOT re-send because an earlier call in the @@ -152,6 +161,8 @@ export interface ExploreDiagnosticFile extends ExploreCandidateMeta { allowance: number | null; /** Reservation + inherited slack — the bound the render paths actually use. */ spendable: number | null; + /** Same bound after holding back what is still owed to unreached files. */ + funded: number | null; render: ExploreRenderMode | null; skipped: ExploreSkipReason | null; clipped: boolean; @@ -375,7 +386,7 @@ export class ExploreDiagnostics { /** Record one ranked candidate's scoring inputs, in final sort order. */ noteCandidate(path: string, meta: ExploreCandidateMeta): void { this.files.set(path, { - path, ...meta, allowance: null, spendable: null, + path, ...meta, allowance: null, spendable: null, funded: null, dedupSavedChars: 0, dedupCovered: [], emittedChars: 0, finalChars: 0, share: 0, allocatedShare: 0, clipped: false, }); @@ -413,6 +424,15 @@ export class ExploreDiagnostics { if (rec) rec.spendable = chars; } + /** + * What the render loop will let this file spend once the reservations still + * owed BELOW it are held back (CG-31). Called alongside `recordSpendable`. + */ + recordFunded(path: string, chars: number): void { + const rec = this.files.get(path); + if (rec) rec.funded = chars; + } + /** A candidate rendered source into the response. */ recordRender(path: string, render: ExploreRenderMode, sourceChars: number, clipped: boolean): void { const rec = this.files.get(path); @@ -561,6 +581,7 @@ export class ExploreDiagnostics { kinds: r.kinds, allowance: r.allowance, spendable: r.spendable, + funded: r.funded, render: r.render ?? null, skipped: r.skipped ?? null, clipped: r.clipped, @@ -732,6 +753,11 @@ export function renderTable(report: ExploreDiagnosticReport): string { if (f.spendable !== null && f.allowance !== null && f.spendable !== f.allowance) { out.push(` spendable: ${num(f.spendable)} (reservation + inherited slack)`); } + // Only when the displacement guard actually bit: the gap is what this file + // was refused so the files below it could still be paid. + if (f.funded !== null && f.spendable !== null && f.funded < f.spendable) { + out.push(` funded: ${num(f.funded)} (capped — ${num(f.spendable - f.funded)} held back for files not yet rendered)`); + } if (f.dedupSavedChars > 0) { const spans = f.dedupCovered.slice(0, 6).map(([a, b]) => (a === b ? `${a}` : `${a}-${b}`)).join(','); const more = f.dedupCovered.length > 6 ? `,+${f.dedupCovered.length - 6}` : ''; diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 2655827..869e2eb 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -4048,6 +4048,11 @@ export class ToolHandler { // and no file is ever cut BELOW the reservation it was promised. let reservedSoFar = 0; let sourceSpent = 0; + // How many admitted files the loop has already drawn a reservation for. + // Pairs with `reservedSoFar` to say how many reservations are still owed + // BELOW the current file — the render-space overhead of those pending + // sections has to be held back too, not just their source (CG-31). + let admittedSoFar = 0; // Funding line for the whole-file BUY rule: the response's SOURCE may reach // everything the allocator promised plus one bounded overshoot, and no more. // Measured against the promise rather than `renderCeiling` on purpose — the @@ -4055,6 +4060,7 @@ export class ToolHandler { // what, so funding a buy from it just moves the shortfall to whichever file // the loop reaches last. See WHOLE_FILE_BUY_OVERSHOOT_FRACTION. const reservedTotal = [...allocation.allowances.values()].reduce((sum, n) => sum + n, 0); + const admittedTotal = allocation.allowances.size; const sourceCeiling = reservedTotal + Math.round( budget.maxOutputChars * EXPLORE_ALLOCATION.WHOLE_FILE_BUY_OVERSHOOT_FRACTION, ); @@ -4093,7 +4099,39 @@ export class ToolHandler { Math.max(reserved, Math.round(budget.maxOutputChars * EXPLORE_ALLOCATION.MAX_SHARE)), ); reservedSoFar += reserved; + admittedSoFar++; diag?.recordSpendable(filePath, allowance); + // DISPLACEMENT GUARD, in render space (CG-31). `allowance` says what this + // file MAY spend; it does not say the bytes are still there to spend. The + // hard ceiling is shared with every file the loop has not reached yet, and + // their reservations are promises the allocator already made — so what is + // left before the ceiling is not all ours: `owedRenderBelow` of it is + // spoken for. Subtracting it is the same inequality the whole-file BUY arm + // enforces with `owedBelow` (see below), moved into the units the cluster + // path actually spends in — source PLUS the per-section overhead each + // pending file will charge. + // + // Floored at this file's OWN reservation, never below: a kept promise is + // not a displacement, and cutting a file under what it earned is the + // failure this whole allocation layer exists to prevent. When the + // reservations genuinely cannot all fit under the ceiling (the response + // preamble is charged to the same ceiling but not to the allocator's + // envelope), the floor means the shortfall lands on the LAST file rather + // than being taken out of the top one — same as before this guard. + // + // Slack still reaches the file: a file above that under-spends leaves + // `totalChars` lower, which raises `headroom` one-for-one, so the + // carry-forward the `allowance` line grants is exactly the carry-forward + // this bound funds. + const owedBelow = Math.max(0, reservedTotal - reservedSoFar); + const owedRenderBelow = owedBelow + + EXPLORE_ALLOCATION.FILE_OVERHEAD * Math.max(0, admittedTotal - admittedSoFar); + const headroom = Math.max(0, renderCeiling - totalChars - EXPLORE_ALLOCATION.FILE_OVERHEAD); + const fundedHeadroom = Math.max( + Math.min(reserved, headroom), + headroom - owedRenderBelow, + ); + diag?.recordFunded(filePath, Math.min(allowance, fundedHeadroom)); const absPath = validatePathWithinRoot(projectRoot, filePath); if (!absPath || !existsSync(absPath)) { diag?.recordSkip(filePath, 'unreadable'); @@ -4306,7 +4344,13 @@ export class ToolHandler { // response and starve the co-flow file (harness.rs's poll). The native agent // windows such a file too (~190 lines at a time), so this mimics, not // truncates. Always emit ≥1 (never an empty section). - const bodyCap = allowance; + // + // Held to `fundedHeadroom` as well (CG-31) so this path cannot spend a + // reservation still owed below it either. It never exceeds `allowance` + // today, so the bound only bites once the ceiling is genuinely tight — + // but "every render path" has to mean every one, or the guard is just a + // detour the next god-file takes. + const bodyCap = Math.min(allowance, fundedHeadroom); const bodyIds = new Set(); let bodyChars = 0; for (const n of syms.filter(n => prio(n) < 99 && n.endLine >= n.startLine).sort((a, b) => prio(a) - prio(b))) { @@ -4434,8 +4478,10 @@ export class ToolHandler { // rather than a size cap — a buy that fits the line only by spending a // lower-ranked file's reservation is the trade that dropped // `payslip_builder.go`, and it is refused here. Self-limiting: each buy - // grows `sourceSpent`, so the pool cannot be spent twice. - const owedBelow = Math.max(0, reservedTotal - reservedSoFar); + // grows `sourceSpent`, so the pool cannot be spent twice. (`owedBelow` is + // computed once at the top of the iteration — the cluster path below + // enforces the same inequality in render space; see `fundedHeadroom`.) + // // Third condition on the BUY arm only: it must also FIT. A whole render // that overruns `renderCeiling` is skipped ENTIRELY a few lines below (the // branch refuses to slice a file mid-method), so attempting a buy that @@ -4940,13 +4986,20 @@ export class ToolHandler { // top-scoring file at the same 3,800 as the weakest one, while the whole-file // branch above handed a small file 3x that. The reservation is the whole point // of CG-12 — bytes follow relevance, not file size. - const headroom = Math.max(0, renderCeiling - totalChars - 200); - const fileBudget = Math.min(allowance, headroom); + // + // `fundedHeadroom`, not `headroom` (CG-31): what is left before the hard + // ceiling includes every unreached file's reservation, and spending that + // is how one clustered file zeroed five admitted peers. It is ≤ `headroom` + // by construction, so it is the only bound these three lines need. + const fileBudget = Math.min(allowance, fundedHeadroom); // Spine ceiling: a flow-path cluster may exceed the reservation (the call path // IS the answer and clipping it forces the Read), but bounded — 1.5x the // reservation and never past the ceiling — so a pathological long in-file - // spine can't run away or starve co-flow files entirely. - const SPINE_CEILING = Math.min(Math.round(allowance * 1.5), headroom); + // spine can't run away or starve co-flow files entirely. The 1.5x is drawn + // from the shared envelope, so it is exactly the overshoot the displacement + // guard has to fund: past `fundedHeadroom` the extra half-reservation is + // another file's, not spare room. + const SPINE_CEILING = Math.min(Math.round(allowance * 1.5), fundedHeadroom); const chosenIndices = new Set(); // Final renders (deduped, shrunk where oversize) by cluster index. Computed // during selection and reused at emission so the two never disagree.