Files
codegraph/__tests__/synthesis-tail-scaling.test.ts
T
a3f90089e8 fix(indexing): bounded-memory yielding pipeline tail + daemon session fixes (#1212) (#1226)
Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.

The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.

- All synthesis passes stream node-kind scans (cursors, not arrays) and
  yield on time-budgeted checkpoints; language gates skip passes whose
  filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
  units stream one file at a time, and the all-functions array +
  write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
  per-file store commits chunk with yields behind a serialized flush
  chain (preserving #1015 file-order determinism); resolver warm-up
  streams the DISTINCT name set; resolution batch-tail and merged-edge
  inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
  session permanently silent (client-hello tail unshifted into a
  flowing stream with zero listeners — the long-standing #662 test
  flake was this real bug); first tool call no longer queues behind
  the query pool's cold start (pool.ready gate).

Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 23:18:23 -05:00

104 lines
3.5 KiB
TypeScript

/**
* Synthesis-tail scaling regressions (#1212).
*
* On a 2M-node graph (Linux kernel) the dynamic-edge synthesis tail OOM'd
* Node's default heap and/or starved the #850 liveness watchdog: the kotlin
* expect/actual pass opened with `getAllNodes()` (hydrating the entire node
* table into one array), and most passes ran start-to-finish with no yield
* points. The fix streams every whole-kind scan, filters the kotlin pass
* SQL-side, and language-gates passes off the files table.
*
* These tests pin the query-level building blocks and the end-to-end kotlin
* bridge so the memory fix can't silently change what gets synthesized.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
import { CodeGraph } from '../src';
describe('synthesis-tail scaling (#1212)', () => {
let dir: string;
beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'synth-scaling-')); });
afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); });
it('kotlin expect/actual still bridges through the streamed decorator query', async () => {
fs.writeFileSync(
path.join(dir, 'Platform.kt'),
`package com.example.shared
expect fun platformName(): String
`
);
fs.writeFileSync(
path.join(dir, 'Platform.jvm.kt'),
`package com.example.shared
actual fun platformName(): String = "JVM"
`
);
const cg = await CodeGraph.init(dir);
await cg.indexAll();
const db = (cg as any).db.db;
const edges = db
.prepare(
`SELECT e.source, e.target FROM edges e
WHERE json_extract(e.metadata,'$.synthesizedBy') = 'kotlin-expect-actual'`
)
.all();
expect(edges.length).toBeGreaterThanOrEqual(1);
cg.close();
});
it('iterateNodesByLanguageWithDecorator matches getAllNodes().filter exactly', async () => {
fs.writeFileSync(
path.join(dir, 'A.kt'),
`package p
actual fun realActual(): Int = 1
`
);
// TypeScript decorator whose name CONTAINS "actual" — the SQL LIKE
// pre-filter must not surface it as a kotlin actual.
fs.writeFileSync(
path.join(dir, 'b.ts'),
`function actual(target: object): void {}
class C {
m(): number { return 1; }
}
`
);
const cg = await CodeGraph.init(dir);
await cg.indexAll();
const queries = (cg as unknown as { queries: import('../src/db/queries').QueryBuilder }).queries;
const streamed = [...queries.iterateNodesByLanguageWithDecorator('kotlin', 'actual')]
.filter((n) => n.decorators?.includes('actual'))
.map((n) => n.id)
.sort();
const reference = queries
.getAllNodes()
.filter((n) => n.language === 'kotlin' && !!n.decorators?.includes('actual'))
.map((n) => n.id)
.sort();
expect(streamed).toEqual(reference);
expect(reference.length).toBeGreaterThanOrEqual(1); // the fixture really has one
cg.close();
});
it('getDistinctFileLanguages reports exactly the languages present', async () => {
fs.writeFileSync(path.join(dir, 'x.ts'), 'export const a = 1;\n');
fs.writeFileSync(path.join(dir, 'y.py'), 'def f():\n return 1\n');
const cg = await CodeGraph.init(dir);
await cg.indexAll();
const queries = (cg as unknown as { queries: import('../src/db/queries').QueryBuilder }).queries;
const langs = queries.getDistinctFileLanguages();
expect(langs.has('typescript')).toBe(true);
expect(langs.has('python')).toBe(true);
expect(langs.has('kotlin')).toBe(false);
cg.close();
});
});