Large-codebase indexing died at the end of "Resolving refs" two ways: watchdog kills of healthy work (24k-file Java on Windows, #1212 — third iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale, where v1.3.0 could not complete at any watchdog setting). Root causes: ~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no yield points, several materialized whole-graph snapshots (kotlin expect/actual opened with getAllNodes() — 2M nodes in one array; the C fn-pointer pass retained every C file's contents twice plus every function node), and the post-index WAL checkpoint ran minutes of synchronous IO on the main thread, killing even a successful index at the finish line. The pipeline tail now follows the same discipline as the rest: never hold O(graph) in the heap, yield everywhere. - All synthesis passes stream node-kind scans (cursors, not arrays) and yield on time-budgeted checkpoints; language gates skip passes whose filters a project's file languages provably can't satisfy. - kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded, units stream one file at a time, and the all-functions array + write-only id map are gone; spring reads each .java once, not twice. - runMaintenance moved to a worker thread (own SQLite connection); per-file store commits chunk with yields behind a serialized flush chain (preserving #1015 file-order determinism); resolver warm-up streams the DISTINCT name set; resolution batch-tail and merged-edge inserts run in bounded sub-transactions. - Daemon: fixed a socket-handoff race that could leave a fresh MCP session permanently silent (client-hello tail unshifted into a flowing stream with zero listeners — the long-standing #662 test flake was this real bug); first tool call no longer queues behind the query pool's cold start (pool.ready gate). Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully indexes in 27m8s on a 2-core/6GB container at default heap + default watchdog; llvm-project (180k files) completes under 1GB RSS including kill-and-sync recovery; synthesized-edge and full-graph parity are byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky daemon test passed 25/25 under load. Env-gated diagnostics kept: CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop tracing. Design record: docs/design/main-thread-stall-followup.md. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
104 lines
3.5 KiB
TypeScript
104 lines
3.5 KiB
TypeScript
/**
|
|
* Synthesis-tail scaling regressions (#1212).
|
|
*
|
|
* On a 2M-node graph (Linux kernel) the dynamic-edge synthesis tail OOM'd
|
|
* Node's default heap and/or starved the #850 liveness watchdog: the kotlin
|
|
* expect/actual pass opened with `getAllNodes()` (hydrating the entire node
|
|
* table into one array), and most passes ran start-to-finish with no yield
|
|
* points. The fix streams every whole-kind scan, filters the kotlin pass
|
|
* SQL-side, and language-gates passes off the files table.
|
|
*
|
|
* These tests pin the query-level building blocks and the end-to-end kotlin
|
|
* bridge so the memory fix can't silently change what gets synthesized.
|
|
*/
|
|
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
|
import * as fs from 'node:fs';
|
|
import * as path from 'node:path';
|
|
import * as os from 'node:os';
|
|
import { CodeGraph } from '../src';
|
|
|
|
describe('synthesis-tail scaling (#1212)', () => {
|
|
let dir: string;
|
|
beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'synth-scaling-')); });
|
|
afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); });
|
|
|
|
it('kotlin expect/actual still bridges through the streamed decorator query', async () => {
|
|
fs.writeFileSync(
|
|
path.join(dir, 'Platform.kt'),
|
|
`package com.example.shared
|
|
|
|
expect fun platformName(): String
|
|
`
|
|
);
|
|
fs.writeFileSync(
|
|
path.join(dir, 'Platform.jvm.kt'),
|
|
`package com.example.shared
|
|
|
|
actual fun platformName(): String = "JVM"
|
|
`
|
|
);
|
|
const cg = await CodeGraph.init(dir);
|
|
await cg.indexAll();
|
|
|
|
const db = (cg as any).db.db;
|
|
const edges = db
|
|
.prepare(
|
|
`SELECT e.source, e.target FROM edges e
|
|
WHERE json_extract(e.metadata,'$.synthesizedBy') = 'kotlin-expect-actual'`
|
|
)
|
|
.all();
|
|
expect(edges.length).toBeGreaterThanOrEqual(1);
|
|
cg.close();
|
|
});
|
|
|
|
it('iterateNodesByLanguageWithDecorator matches getAllNodes().filter exactly', async () => {
|
|
fs.writeFileSync(
|
|
path.join(dir, 'A.kt'),
|
|
`package p
|
|
|
|
actual fun realActual(): Int = 1
|
|
`
|
|
);
|
|
// TypeScript decorator whose name CONTAINS "actual" — the SQL LIKE
|
|
// pre-filter must not surface it as a kotlin actual.
|
|
fs.writeFileSync(
|
|
path.join(dir, 'b.ts'),
|
|
`function actual(target: object): void {}
|
|
class C {
|
|
m(): number { return 1; }
|
|
}
|
|
`
|
|
);
|
|
const cg = await CodeGraph.init(dir);
|
|
await cg.indexAll();
|
|
|
|
const queries = (cg as unknown as { queries: import('../src/db/queries').QueryBuilder }).queries;
|
|
const streamed = [...queries.iterateNodesByLanguageWithDecorator('kotlin', 'actual')]
|
|
.filter((n) => n.decorators?.includes('actual'))
|
|
.map((n) => n.id)
|
|
.sort();
|
|
const reference = queries
|
|
.getAllNodes()
|
|
.filter((n) => n.language === 'kotlin' && !!n.decorators?.includes('actual'))
|
|
.map((n) => n.id)
|
|
.sort();
|
|
expect(streamed).toEqual(reference);
|
|
expect(reference.length).toBeGreaterThanOrEqual(1); // the fixture really has one
|
|
cg.close();
|
|
});
|
|
|
|
it('getDistinctFileLanguages reports exactly the languages present', async () => {
|
|
fs.writeFileSync(path.join(dir, 'x.ts'), 'export const a = 1;\n');
|
|
fs.writeFileSync(path.join(dir, 'y.py'), 'def f():\n return 1\n');
|
|
const cg = await CodeGraph.init(dir);
|
|
await cg.indexAll();
|
|
|
|
const queries = (cg as unknown as { queries: import('../src/db/queries').QueryBuilder }).queries;
|
|
const langs = queries.getDistinctFileLanguages();
|
|
expect(langs.has('typescript')).toBe(true);
|
|
expect(langs.has('python')).toBe(true);
|
|
expect(langs.has('kotlin')).toBe(false);
|
|
cg.close();
|
|
});
|
|
});
|