fix(db): loop-append dense unresolved-ref result rows; make stripped-salvage visible (#1558) (#1576)
Real-world validation of #1575 on indexes damaged by the released v1.5.0 binary surfaced both of these. getUnresolvedReferencesByFiles chunked its INPUT under SQLite's parameter limit but appended each chunk's RESULT rows with a spread — every row becomes a call argument, so a dense recovery sync (the #1541 self-heal re-indexing 919 files produced 234,440 rows) exceeded V8's argument limit and killed resolution mid-sync with "Maximum call stack size exceeded", leaving the graph 226k edges short until another sync resumed the orphans (and that sweep resolves measurably worse than the batched path — see the follow-up issue). The failed-ref retry loader had the identical pattern on unbounded result rows. Both append with a loop now (#1558). The #1575 stripped-salvage warning also never rendered: init's summary prints only index_partial warnings and counts only hard errors, so a run with salvaged files still read as fully clean — and with no hard errors the detail wasn't written to errors.log either. Salvage entries now carry code 'salvaged_stripped', the summary prints a visible warning naming the files, and errors.log is written for salvage-only runs. Validated on real corpora with full-graph dumps: healthy-path inits stay byte-identical to the pre-#1575 baseline (cpython Lib, Alamofire, with a determinism control); a realistically-damaged index (41 wiped + 5 missing files, damage generated by the released binary) heals in one plain sync to identical per-file counts and an edge set within the normal incremental residual; pathological mass damage (52% of the repo) completes without crashing. New regression test reproduces the RangeError on the old code with 200k pending refs. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
26045b3159
commit
d8f2eeaddf
@@ -0,0 +1,74 @@
|
||||
/**
|
||||
* getUnresolvedReferencesByFiles must survive dense result sets (#1558).
|
||||
*
|
||||
* The input file-path list is chunked under SQLite's parameter limit, but the
|
||||
* ROWS a chunk returns are unbounded — and appending them with
|
||||
* `rows.push(...chunkRows)` passes every row as a call argument, so a dense
|
||||
* chunk (a recovery sync re-indexing many files at once, e.g. the #1541
|
||||
* self-heal) exceeded V8's argument limit and killed the whole sync with
|
||||
* "Maximum call stack size exceeded" after the store phase, leaving every
|
||||
* re-indexed file's references unresolved. Reproduced for real on a
|
||||
* cpython-stdlib-sized heal (919 files, 234k refs). The append is now a loop;
|
||||
* this pins it with a result set well past V8's argument ceiling (~124k).
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import * as os from 'node:os';
|
||||
import { CodeGraph } from '../src';
|
||||
import type { UnresolvedReference } from '../src/types';
|
||||
|
||||
describe('unresolved-ref loads with dense result sets (#1558)', () => {
|
||||
let dir: string;
|
||||
let cg: CodeGraph;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'refs-spread-'));
|
||||
fs.writeFileSync(path.join(dir, 'anchor.py'), 'def anchor():\n return 1\n');
|
||||
cg = await CodeGraph.init(dir);
|
||||
await cg.indexAll();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
cg.destroy();
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('returns 200k pending refs from few files without exhausting the call stack', () => {
|
||||
const queries = (cg as unknown as {
|
||||
queries: {
|
||||
insertUnresolvedRefsBatch(refs: UnresolvedReference[]): void;
|
||||
getUnresolvedReferencesByFiles(paths: string[]): UnresolvedReference[];
|
||||
};
|
||||
}).queries;
|
||||
|
||||
const FILES = 200;
|
||||
const TOTAL = 200_000;
|
||||
const paths: string[] = Array.from({ length: FILES }, (_, i) => `src/f${i}.py`);
|
||||
// unresolved_refs.from_node_id is FK-constrained — anchor on a real node.
|
||||
const anchorId = cg.getNodesInFile('anchor.py')[0]!.id;
|
||||
|
||||
const batch: UnresolvedReference[] = [];
|
||||
for (let i = 0; i < TOTAL; i++) {
|
||||
batch.push({
|
||||
fromNodeId: anchorId,
|
||||
referenceName: `ref_${i}`,
|
||||
referenceKind: 'call',
|
||||
line: (i % 1000) + 1,
|
||||
column: 0,
|
||||
filePath: paths[i % FILES]!,
|
||||
language: 'python',
|
||||
});
|
||||
if (batch.length === 20_000) {
|
||||
queries.insertUnresolvedRefsBatch(batch);
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
if (batch.length > 0) queries.insertUnresolvedRefsBatch(batch);
|
||||
|
||||
// All 200 paths fit in ONE SQLite parameter chunk, so a single query
|
||||
// returns all 200k rows — the exact shape that blew the argument limit.
|
||||
const rows = queries.getUnresolvedReferencesByFiles(paths);
|
||||
expect(rows.length).toBe(TOTAL);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user