test(explore): #1500 regression fixtures for budget allocation (CG-6)
Two permanent fixtures pinning the failure mode from issue #1500 — explore spending its byte envelope on files that merely name-collide with the query. BOTH FAIL TODAY, by design: they document the bug and become the pass gate for CG-10 (scoring) + CG-12 (proportional allocation). __tests__/fixtures/payroll-go/ — a synthetic Go service mirroring the reporter's shape: generated FKIT CRUD beside a hand-written payroll use-case, entered from an HTTP route. Half the generated tree carries ORDINARY names detectable only by their `// Code generated ... DO NOT EDIT.` header (the #1500 case, and end-to-end cover for CG-5); `payrollpb/*.pb.go` covers the path-detectable channel. BuildPayslip, Upsert and Store each exist twice, generated and hand-written. cycle.go sits above the whole-file window so it clips; the generated files sit below it so they ship whole. Asking "how does payroll cycle create and calculate payslips?" — naming none of the answering symbols — the generated CRUD delivers 57.4% of the envelope against the hand-written layer's 25.6%, all of the latter domain types. cycle.go is allocated the single largest slice (30.6%) and delivers ZERO: the hard ceiling drops its whole section. runPayrollCycleAll, the hand-written BuildPayslip and the real Upsert never reach the agent. The second fixture is this repo, "how does explore allocate its output budget across files", where scripts/agent-eval/*.mjs take 71.8% against tools.ts's 18.5% despite scoring 4.6x lower. It reads the live index, so its assertions are relative rather than fixed percentages. - scripts/agent-eval/probe-allocation.mjs — per-file budget-share probe, driving the CG-4 diagnostic through a JSONL sidecar so it measures the shipping allocator. Fixture entries are hermetic (copy + re-index per run, verified byte-identical across runs); exits 1 while any assertion fails. - scripts/agent-eval/allocation-fixtures.json — both fixtures declared, with the 2026-08-03 baselines. - __tests__/explore-allocation-1500.test.ts — fixture-shape assertions green today; the allocation assertions held as `it.fails` so the suite stays green while the bug is open and goes RED the moment it is fixed. Also documented and deliberately left unfixed: runPayrollCycleAll's `s.store.Upsert` edge resolves to the GENERATED Store.Upsert, not the hand-written one — same-name method resolution across two packages picks the wrong receiver. It is upstream of the allocation bug, so it belongs with CG-10's scoring work. Refs #1500 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
co-authored by
Claude Opus 5
Claude Opus 5
parent
16e17495f4
commit
bd86ad2061
@@ -0,0 +1,112 @@
|
||||
{
|
||||
"$comment": [
|
||||
"Regression fixtures for GitHub issue #1500 / epic CG-1 — relevance-proportional",
|
||||
"explore budget allocation. Run them with `node scripts/agent-eval/probe-allocation.mjs`",
|
||||
"against a built dist/. BOTH FIXTURES FAIL TODAY: that is the point — they document",
|
||||
"the bug and become the pass gate for the allocation change (CG-10 + CG-12).",
|
||||
"",
|
||||
"`groups` partitions the files explore rendered into `answer` (what the query is",
|
||||
"actually about) and `incidental` (what wins the envelope today on name collisions).",
|
||||
"Assertions are on the DELIVERED envelope unless suffixed `Allocated`; delivered is",
|
||||
"what the agent got, allocated is what the render loop chose before the hard ceiling.",
|
||||
"Shares are fractions of the whole response, meta-text included, so they never sum to 1."
|
||||
],
|
||||
"fixtures": [
|
||||
{
|
||||
"id": "payroll-go",
|
||||
"title": "#1500 — generated Go CRUD beside a hand-written payroll workflow",
|
||||
"kind": "fixture",
|
||||
"path": "__tests__/fixtures/payroll-go",
|
||||
"query": "how does payroll cycle create and calculate payslips?",
|
||||
"rationale": [
|
||||
"The reporter's repo shape: a Go service whose generated FKIT CRUD layer sits",
|
||||
"beside the hand-written use-case that does the real work. The query deliberately",
|
||||
"does NOT name runPayrollCycleAll / BuildPayslip / Upsert — an architecture",
|
||||
"question phrased the way a newcomer would phrase it. The generated layer",
|
||||
"name-collides on every query term (CreatePayslip, PayrollCycleCreateRequest,",
|
||||
"CalculatePayrollCycleTotals, a second BuildPayslip, a second Upsert), so a",
|
||||
"scorer that rewards incidental name matches surfaces the CRUD path.",
|
||||
"Half the generated files carry ORDINARY names and are only detectable by their",
|
||||
"`// Code generated ... DO NOT EDIT.` header (CG-5), which is what makes this",
|
||||
"the #1500 case rather than a .pb.go case."
|
||||
],
|
||||
"groups": {
|
||||
"answer": [
|
||||
"internal/usecase/**",
|
||||
"internal/store/**",
|
||||
"internal/transport/**",
|
||||
"internal/domain/**",
|
||||
"cmd/**"
|
||||
],
|
||||
"incidental": ["internal/gen/**"]
|
||||
},
|
||||
"assert": {
|
||||
"answerShareAtLeast": 0.55,
|
||||
"incidentalShareAtMost": 0.25,
|
||||
"topFileGroup": "answer",
|
||||
"mustDeliverBytes": [
|
||||
"internal/usecase/payroll/cycle.go",
|
||||
"internal/usecase/payroll/payslip_builder.go"
|
||||
],
|
||||
"$mustContainComment": "Needles are chosen to match the HAND-WRITTEN chain only — a bare `BuildPayslip`/`Upsert` also matches the generated collisions, which is the whole point of the fixture.",
|
||||
"mustContain": [
|
||||
"runPayrollCycleAll",
|
||||
"func (s *Service) BuildPayslip",
|
||||
"s.store.Upsert(ctx, slip)"
|
||||
]
|
||||
},
|
||||
"baseline": {
|
||||
"measuredOn": "2026-08-03",
|
||||
"note": "19 files → very-tiny tier (13,000 budget); 23,020 chars allocated against it, cut to 16,011 by the 19,500 hard ceiling.",
|
||||
"delivered": {
|
||||
"internal/gen/fkit/payroll/payslip.go": 0.307,
|
||||
"internal/gen/fkit/payroll/payroll_cycle.go": 0.266,
|
||||
"internal/domain/payroll/payslip.go": 0.256,
|
||||
"internal/usecase/payroll/cycle.go": 0.0
|
||||
},
|
||||
"verdict": "The workflow file is allocated the single largest slice (30.6%) and delivers ZERO — the hard ceiling drops its whole section. Generated CRUD takes 57.4% of what the agent actually receives; runPayrollCycleAll, BuildPayslip and the real Upsert never reach the response."
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "self-query",
|
||||
"title": "This repo — incidental `explore`/`BUDGET` matches in the agent-eval scripts",
|
||||
"kind": "self",
|
||||
"path": ".",
|
||||
"query": "how does explore allocate its output budget across files",
|
||||
"rationale": [
|
||||
"The same failure mode with no generated code in sight. `scripts/agent-eval/*.mjs`",
|
||||
"mention `explore` and `BUDGET` incidentally — they are eval harnesses, not the",
|
||||
"allocator — and they are small enough to ship WHOLE, while src/mcp/tools.ts (which",
|
||||
"carries getExploreOutputBudget and the render loop, and scores 4x higher on every",
|
||||
"signal) is large enough to be clipped at maxCharsPerFile. Allocation follows file",
|
||||
"size, not relevance.",
|
||||
"",
|
||||
"Unlike payroll-go this fixture reads THIS repo's live index, so its exact numbers",
|
||||
"move as the repo changes (indexed file count crossing 500 flips the budget tier).",
|
||||
"The assertions are therefore relative — answer-vs-incidental, not fixed percentages."
|
||||
],
|
||||
"groups": {
|
||||
"answer": ["src/mcp/**"],
|
||||
"incidental": ["scripts/**"]
|
||||
},
|
||||
"assert": {
|
||||
"answerShareAtLeast": 0.5,
|
||||
"incidentalShareAtMost": 0.25,
|
||||
"topFileGroup": "answer",
|
||||
"mustDeliverBytes": ["src/mcp/tools.ts"]
|
||||
},
|
||||
"baseline": {
|
||||
"measuredOn": "2026-08-03",
|
||||
"note": "493 files → small tier (18,000 budget); 27,518 chars allocated against it, cut to 19,749 by the 25,000 hard ceiling.",
|
||||
"delivered": {
|
||||
"scripts/agent-eval/offload-eval-hook.mjs": 0.25,
|
||||
"scripts/agent-eval/offload-eval-metrics.mjs": 0.236,
|
||||
"scripts/agent-eval/parse-session.mjs": 0.232,
|
||||
"src/mcp/tools.ts": 0.185,
|
||||
"scripts/agent-eval/offload-eval-cost.mjs": 0.0
|
||||
},
|
||||
"verdict": "The script corpus takes 71.8% of the delivered envelope (79.4% of what was allocated) against tools.ts's 18.5%, despite tools.ts scoring 46 vs 10, carrying 2.3x the graph mass and 3x the distinct term hits."
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
Executable
+296
@@ -0,0 +1,296 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Deterministic per-file budget-share probe for `codegraph_explore` (CG-6).
|
||||
*
|
||||
* `probe-explore.mjs` prints what explore returned. This prints how the response
|
||||
* was DIVIDED — which files won the byte envelope and in what proportion — and
|
||||
* checks that division against a declared expectation. It is the regression gate
|
||||
* for GitHub issue #1500 / epic CG-1: an architecture question that doesn't name
|
||||
* the exact use-case must concentrate the budget on the code that answers it, not
|
||||
* on a generated CRUD layer (or an eval script) that merely name-collides.
|
||||
*
|
||||
* The numbers come from the CG-4 diagnostic (`CODEGRAPH_EXPLORE_DEBUG`), read back
|
||||
* from a JSONL sidecar, so the probe measures the shipping allocator rather than
|
||||
* re-deriving shares from the markdown.
|
||||
*
|
||||
* Fixtures are declared in `allocation-fixtures.json`. A `kind: "fixture"` entry is
|
||||
* hermetic — the fixture tree is copied to a fresh temp dir and indexed per run, so
|
||||
* two runs on one build give identical numbers. A `kind: "self"` entry reads this
|
||||
* repo's live index and therefore moves as the repo changes; its assertions are
|
||||
* relative for that reason.
|
||||
*
|
||||
* Usage (needs a current `npm run build`):
|
||||
* node scripts/agent-eval/probe-allocation.mjs # every fixture
|
||||
* node scripts/agent-eval/probe-allocation.mjs payroll-go # one fixture
|
||||
* node scripts/agent-eval/probe-allocation.mjs --json # machine-readable
|
||||
* node scripts/agent-eval/probe-allocation.mjs --keep # keep the temp index
|
||||
*
|
||||
* Exit code: 0 if every assertion holds, 1 if any fails, 2 on a setup error.
|
||||
* BOTH FIXTURES ARE EXPECTED TO FAIL until CG-10/CG-12 land — that failure is the
|
||||
* documented bug. Use --expect-fail to invert the exit code while it is the state
|
||||
* of the world (0 = still broken, 1 = fixed, go flip the gate).
|
||||
*/
|
||||
import { cpSync, mkdtempSync, mkdirSync, readFileSync, rmSync, existsSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const REPO_ROOT = resolve(HERE, '../..');
|
||||
const SPEC_PATH = join(HERE, 'allocation-fixtures.json');
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
const flags = new Set(argv.filter((a) => a.startsWith('--')));
|
||||
const wanted = argv.filter((a) => !a.startsWith('--'));
|
||||
const asJson = flags.has('--json');
|
||||
const keepTemp = flags.has('--keep');
|
||||
const expectFail = flags.has('--expect-fail');
|
||||
|
||||
const say = (line = '') => { if (!asJson) console.log(line); };
|
||||
const pct = (f) => `${(f * 100).toFixed(1)}%`;
|
||||
const num = (n) => Math.round(n).toLocaleString('en-US');
|
||||
|
||||
/** Load the built dist — the probe measures the shipping allocator, not src. */
|
||||
async function loadDist() {
|
||||
const distIndex = join(REPO_ROOT, 'dist/index.js');
|
||||
if (!existsSync(distIndex)) {
|
||||
console.error('dist/ not built — run `npm run build` first.');
|
||||
process.exit(2);
|
||||
}
|
||||
const idx = await import(pathToFileURL(distIndex).href);
|
||||
const tools = await import(pathToFileURL(join(REPO_ROOT, 'dist/mcp/tools.js')).href);
|
||||
// esModuleInterop: dynamic import of CJS yields { default: module.exports, ...named }
|
||||
const CodeGraph = idx.default?.default ?? idx.default ?? idx.CodeGraph;
|
||||
const ToolHandler = tools.ToolHandler ?? tools.default?.ToolHandler;
|
||||
if (typeof CodeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') {
|
||||
console.error('could not resolve CodeGraph/ToolHandler from dist/');
|
||||
process.exit(2);
|
||||
}
|
||||
return { CodeGraph, ToolHandler };
|
||||
}
|
||||
|
||||
/** `internal/gen/**` → /^internal\/gen\/.*$/ . Supports `**`, `*` and literals. */
|
||||
function globToRegExp(glob) {
|
||||
// Park `**` on a sentinel no path can contain, so the `*` pass cannot eat it.
|
||||
// Written as an escape, not a literal byte — a raw NUL makes git treat this
|
||||
// whole script as binary, which costs every future diff of it.
|
||||
const DOUBLE_STAR = '\u0000';
|
||||
const escaped = glob.replace(/[.+^${}()|[\]\\]/g, '\\$&');
|
||||
const body = escaped
|
||||
.replace(/\*\*/g, DOUBLE_STAR)
|
||||
.replace(/\*/g, '[^/]*')
|
||||
.replaceAll(DOUBLE_STAR, '.*');
|
||||
return new RegExp(`^${body}$`);
|
||||
}
|
||||
|
||||
const groupOf = (path, groups) => {
|
||||
for (const [name, globs] of Object.entries(groups)) {
|
||||
if (globs.some((g) => globToRegExp(g).test(path))) return name;
|
||||
}
|
||||
return 'other';
|
||||
};
|
||||
|
||||
/**
|
||||
* Run one explore call with the diagnostic pointed at a sidecar, and return the
|
||||
* report plus the response text.
|
||||
*/
|
||||
async function runExplore({ CodeGraph, ToolHandler }, repoPath, query, sidecar) {
|
||||
const cg = CodeGraph.openSync(repoPath);
|
||||
const prior = process.env.CODEGRAPH_EXPLORE_DEBUG;
|
||||
process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar;
|
||||
try {
|
||||
const res = await new ToolHandler(cg).execute('codegraph_explore', { query });
|
||||
const text = res.content?.[0]?.text ?? '';
|
||||
const lines = readFileSync(sidecar, 'utf-8').trim().split('\n').filter(Boolean);
|
||||
if (lines.length === 0) throw new Error('diagnostic produced no report');
|
||||
return { report: JSON.parse(lines[lines.length - 1]), text };
|
||||
} finally {
|
||||
if (prior === undefined) delete process.env.CODEGRAPH_EXPLORE_DEBUG;
|
||||
else process.env.CODEGRAPH_EXPLORE_DEBUG = prior;
|
||||
try { cg.close?.(); } catch {}
|
||||
}
|
||||
}
|
||||
|
||||
/** Copy a fixture tree to a fresh temp dir and index it — hermetic per run. */
|
||||
async function materializeFixture({ CodeGraph }, fixturePath) {
|
||||
const src = resolve(REPO_ROOT, fixturePath);
|
||||
if (!existsSync(src)) throw new Error(`fixture tree not found: ${src}`);
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cg-alloc-'));
|
||||
cpSync(src, dir, { recursive: true });
|
||||
// A stray index inside the checked-in tree would be copied in and reused.
|
||||
rmSync(join(dir, '.codegraph'), { recursive: true, force: true });
|
||||
const cg = CodeGraph.initSync(dir);
|
||||
await cg.indexAll();
|
||||
cg.close?.();
|
||||
return dir;
|
||||
}
|
||||
|
||||
/** Evaluate one fixture's assertions against its report. Returns check rows. */
|
||||
function evaluate(fixture, report, text) {
|
||||
const { groups, assert: want } = fixture;
|
||||
const delivered = new Map();
|
||||
const allocated = new Map();
|
||||
for (const f of report.files) {
|
||||
const g = groupOf(f.path, groups);
|
||||
delivered.set(g, (delivered.get(g) ?? 0) + f.share);
|
||||
allocated.set(g, (allocated.get(g) ?? 0) + f.allocatedShare);
|
||||
}
|
||||
const share = (g) => delivered.get(g) ?? 0;
|
||||
const top = report.files
|
||||
.filter((f) => f.finalChars > 0)
|
||||
.sort((a, b) => b.finalChars - a.finalChars)[0];
|
||||
|
||||
const checks = [];
|
||||
const add = (name, pass, detail) => checks.push({ name, pass, detail });
|
||||
|
||||
if (want.answerShareAtLeast !== undefined) {
|
||||
add(
|
||||
`answer group takes >= ${pct(want.answerShareAtLeast)} of the envelope`,
|
||||
share('answer') >= want.answerShareAtLeast,
|
||||
`answer ${pct(share('answer'))} delivered (${pct(allocated.get('answer') ?? 0)} allocated)`,
|
||||
);
|
||||
}
|
||||
if (want.incidentalShareAtMost !== undefined) {
|
||||
add(
|
||||
`incidental group takes <= ${pct(want.incidentalShareAtMost)} of the envelope`,
|
||||
share('incidental') <= want.incidentalShareAtMost,
|
||||
`incidental ${pct(share('incidental'))} delivered (${pct(allocated.get('incidental') ?? 0)} allocated)`,
|
||||
);
|
||||
}
|
||||
if (want.topFileGroup) {
|
||||
const actual = top ? groupOf(top.path, groups) : '(nothing delivered)';
|
||||
add(
|
||||
`largest delivered file is in "${want.topFileGroup}"`,
|
||||
actual === want.topFileGroup,
|
||||
top ? `${top.path} (${pct(top.share)}, group "${actual}")` : 'no file delivered any source',
|
||||
);
|
||||
}
|
||||
for (const path of want.mustDeliverBytes ?? []) {
|
||||
const rec = report.files.find((f) => f.path === path);
|
||||
add(
|
||||
`${path} delivers source`,
|
||||
!!rec && rec.finalChars > 0,
|
||||
rec
|
||||
? `${num(rec.finalChars)} delivered of ${num(rec.emittedChars)} allocated` +
|
||||
(rec.finalChars === 0 && rec.emittedChars > 0 ? ' — hard ceiling dropped the whole section' : '') +
|
||||
(rec.emittedChars === 0 ? ` — never rendered (${rec.skipped ?? 'not reached'}, rank #${rec.rank})` : '')
|
||||
: 'not among the ranked candidates',
|
||||
);
|
||||
}
|
||||
for (const needle of want.mustContain ?? []) {
|
||||
add(`response contains "${needle}"`, text.includes(needle), text.includes(needle) ? 'present' : 'absent');
|
||||
}
|
||||
return { checks, delivered, allocated, top };
|
||||
}
|
||||
|
||||
function printReport(fixture, report, evaluated) {
|
||||
const { checks, delivered, allocated } = evaluated;
|
||||
const env = report.envelope;
|
||||
say('');
|
||||
say(`── ${fixture.id} — ${fixture.title}`);
|
||||
say(` query "${report.query}"`);
|
||||
say(` project ${report.projectRoot} · ${num(report.indexedFileCount)} files indexed`);
|
||||
say(
|
||||
` envelope ${num(env.chars)} delivered · ${num(env.allocatedChars)} allocated` +
|
||||
` of ${num(report.budget.maxOutputChars)} budget (hard ceiling ${num(report.budget.hardCeiling)})` +
|
||||
`${env.overBudget ? ' [over budget]' : ''}${env.truncated ? ' [TRUNCATED]' : ''}`,
|
||||
);
|
||||
say('');
|
||||
say(' group alloc% deliv%');
|
||||
for (const g of ['answer', 'incidental', 'other']) {
|
||||
if (!delivered.has(g) && !allocated.has(g)) continue;
|
||||
say(` ${g.padEnd(12)} ${pct(allocated.get(g) ?? 0).padStart(6)} ${pct(delivered.get(g) ?? 0).padStart(6)}`);
|
||||
}
|
||||
say('');
|
||||
say(' # alloc% deliv% bytes score graph hits gen render file');
|
||||
for (const f of report.files.filter((f) => f.emittedChars > 0 || f.finalChars > 0)) {
|
||||
say(
|
||||
' ' + String(f.rank).padStart(2) + ' ' +
|
||||
pct(f.allocatedShare).padStart(6) + ' ' +
|
||||
pct(f.share).padStart(6) + ' ' +
|
||||
num(f.emittedChars).padStart(7) + ' ' +
|
||||
String(f.score).padStart(5) + ' ' +
|
||||
f.graphScore.toFixed(5).padStart(7) + ' ' +
|
||||
String(f.termHits).padStart(4) + ' ' +
|
||||
(f.generated ? ' ✓ ' : ' ') + ' ' +
|
||||
((f.render ?? '-') + (f.clipped ? '*' : '')).padEnd(9) + ' ' +
|
||||
f.path,
|
||||
);
|
||||
}
|
||||
say('');
|
||||
for (const c of checks) say(` ${c.pass ? 'PASS' : 'FAIL'} ${c.name}\n ${c.detail}`);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const spec = JSON.parse(readFileSync(SPEC_PATH, 'utf-8'));
|
||||
const fixtures = spec.fixtures.filter((f) => wanted.length === 0 || wanted.includes(f.id));
|
||||
if (fixtures.length === 0) {
|
||||
console.error(`no fixture matched ${JSON.stringify(wanted)}; known: ${spec.fixtures.map((f) => f.id).join(', ')}`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const dist = await loadDist();
|
||||
const sidecarDir = mkdtempSync(join(tmpdir(), 'cg-alloc-diag-'));
|
||||
const results = [];
|
||||
const temps = [];
|
||||
|
||||
for (const fixture of fixtures) {
|
||||
let repoPath;
|
||||
if (fixture.kind === 'fixture') {
|
||||
repoPath = await materializeFixture(dist, fixture.path);
|
||||
temps.push(repoPath);
|
||||
} else {
|
||||
repoPath = resolve(REPO_ROOT, fixture.path);
|
||||
if (!existsSync(join(repoPath, '.codegraph'))) {
|
||||
console.error(`${fixture.id}: ${repoPath} has no .codegraph index — run \`codegraph init\` there first.`);
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
const sidecar = join(sidecarDir, `${fixture.id}.jsonl`);
|
||||
mkdirSync(dirname(sidecar), { recursive: true });
|
||||
const { report, text } = await runExplore(dist, repoPath, fixture.query, sidecar);
|
||||
const evaluated = evaluate(fixture, report, text);
|
||||
printReport(fixture, report, evaluated);
|
||||
results.push({
|
||||
id: fixture.id,
|
||||
kind: fixture.kind,
|
||||
query: fixture.query,
|
||||
passed: evaluated.checks.every((c) => c.pass),
|
||||
checks: evaluated.checks,
|
||||
shares: {
|
||||
delivered: Object.fromEntries(evaluated.delivered),
|
||||
allocated: Object.fromEntries(evaluated.allocated),
|
||||
},
|
||||
envelope: report.envelope,
|
||||
files: report.files.filter((f) => f.emittedChars > 0 || f.finalChars > 0),
|
||||
});
|
||||
}
|
||||
|
||||
if (!keepTemp) {
|
||||
for (const dir of temps) rmSync(dir, { recursive: true, force: true });
|
||||
rmSync(sidecarDir, { recursive: true, force: true });
|
||||
} else {
|
||||
say('');
|
||||
say(` kept: ${[...temps, sidecarDir].join(' ')}`);
|
||||
}
|
||||
|
||||
const allPassed = results.every((r) => r.passed);
|
||||
if (asJson) {
|
||||
console.log(JSON.stringify({ passed: allPassed, fixtures: results }, null, 2));
|
||||
} else {
|
||||
say('');
|
||||
for (const r of results) say(`${r.passed ? 'PASS' : 'FAIL'} ${r.id}`);
|
||||
if (!allPassed) {
|
||||
say('');
|
||||
say('Failures here are the DOCUMENTED #1500 bug — the budget goes to files that merely');
|
||||
say('name-collide with the query. They become the pass gate once CG-10/CG-12 land.');
|
||||
}
|
||||
}
|
||||
process.exit(expectFail ? (allPassed ? 1 : 0) : (allPassed ? 0 : 1));
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(err?.stack ?? String(err));
|
||||
process.exit(2);
|
||||
});
|
||||
Reference in New Issue
Block a user