perf(index): faster fresh indexing + parallel reference resolution, byte-identical graphs (#1305)
* perf(index): ~34% faster fresh indexing, byte-identical graphs Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed the main thread as the critical path: per-row SQLite statement calls, repeated import-resolution walks, and per-row FTS trigger firings, with the parse workers ~75% idle behind it. This lands the semantics-preserving tranche of fixes: - Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments) behind cached per-batch-size prepared statements; row order preserved, so rowid-based resolution determinism (#1015) is unchanged. - storeFileBundle: one transaction per file instead of four; nested transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no caller depended on nested rollback). - Dedicated store-writer thread for the fresh-DB bulk path (bundles applied in file order on a single writer connection; main thread does no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1. - Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load, rebuild once at the end; crash inside the window self-heals on the next open. - Per-context memos for resolveImportPath/findExportedSymbol + a per-file exported-symbol index, invalidated exactly where clearCaches() already resets the resolver's own caches. - Fast-init on completely fresh DBs (journal in memory, no fsync until the index completes; interrupted init re-runs from scratch). Kill switch: CODEGRAPH_NO_FAST_INIT=1. - MaybeYield returns undefined on the not-due path so per-ref yield checks stop paying a promise + microtask hop each. - Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and worker entry points. Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series). Graph dumps byte-identical across init, re-index, and sync paths; full suite green (2403 passed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(resolution): parallel reference resolution with canonical admission Fan resolution batches across a pool of read-only worker threads, each hosting a full ReferenceResolver over its own SQLite connection; results are admitted on the main thread in chunk order, so edge insertion order, row cleanup, failure parking, and deferred post-pass queues are exactly the sequence the single-threaded loop produces. Per-ref inputs match the baseline because the sequential path already resolves each batch against the state committed BEFORE that batch. Validated byte-identical on excalidraw (pool forced on) and apache/dubbo (4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical graph dumps (91,495 nodes / 223,953 edges). The pool only engages when total pending refs clear a threshold (default 150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1 to disable): measured on a ~58k-ref repo the workers' boot CPU contends with resolution on the same cores and makes indexing slower, so small repos keep the sequential path. When fast-init left the DB in memory-journal mode, WAL is restored before resolution only when the pool will run (readers + rollback-journal writers don't mix). Also: sqlite adapter readOnly open support. TreeCursor spine rewrite of the body walker was built, measured neutral on real repos and equal in a 20k-child microbench (web-tree-sitter's namedChild(i) is not quadratic in this binding), and rejected — per-node JS<->WASM marshaling is the floor, which a traversal swap cannot remove. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Fable 5
Claude Fable 5
parent
246aee8373
commit
5736e24bb6
@@ -24,8 +24,16 @@
|
||||
* stop killing work that is demonstrably making progress.
|
||||
*/
|
||||
|
||||
/** Yield when more than `budgetMs` of wall-clock has passed since the last yield. */
|
||||
export type MaybeYield = () => Promise<void>;
|
||||
/**
|
||||
* Yield when more than `budgetMs` of wall-clock has passed since the last
|
||||
* yield. Returns `undefined` on the (overwhelmingly common) not-due path so a
|
||||
* hot loop can skip the await entirely — `await`ing an async no-op costs a
|
||||
* promise allocation + microtask hop, which at hundreds of thousands of calls
|
||||
* per index is real time. Callers may either `await maybeYield()` (works for
|
||||
* both return shapes) or use the fast form:
|
||||
* `const y = maybeYield(); if (y) await y;`
|
||||
*/
|
||||
export type MaybeYield = () => Promise<void> | undefined;
|
||||
|
||||
/** Default budget: well under the watchdog's minimum heartbeat cadence (~1s), so
|
||||
* a heartbeat byte always has a chance to land between yields. */
|
||||
@@ -33,9 +41,13 @@ export const DEFAULT_YIELD_BUDGET_MS = 250;
|
||||
|
||||
export function createYielder(budgetMs: number = DEFAULT_YIELD_BUDGET_MS): MaybeYield {
|
||||
let last = Date.now();
|
||||
return async function maybeYield(): Promise<void> {
|
||||
if (Date.now() - last < budgetMs) return;
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
last = Date.now();
|
||||
return function maybeYield(): Promise<void> | undefined {
|
||||
if (Date.now() - last < budgetMs) return undefined;
|
||||
return new Promise<void>((resolve) =>
|
||||
setImmediate(() => {
|
||||
last = Date.now();
|
||||
resolve();
|
||||
})
|
||||
);
|
||||
};
|
||||
}
|
||||
|
||||
@@ -55,11 +55,78 @@ export function isNixPathImportRef(ref: UnresolvedRef): boolean {
|
||||
/**
|
||||
* Resolve an import path to an actual file
|
||||
*/
|
||||
// Per-context memos for the two hottest pure lookups on the resolution path:
|
||||
// import-specifier → file resolution and exported-symbol lookup. Both are pure
|
||||
// given a stable file set + node table, which is exactly the window between
|
||||
// ReferenceResolver.clearCaches() calls — clearImportResolverMemos() is invoked
|
||||
// there, so the staleness discipline matches the resolver's own caches.
|
||||
const importPathMemos = new WeakMap<ResolutionContext, Map<string, string | null>>();
|
||||
const exportedSymbolMemos = new WeakMap<ResolutionContext, Map<string, Node | undefined>>();
|
||||
|
||||
/**
|
||||
* Per-file index of exported symbols, replacing repeated linear `.find`s over
|
||||
* `getNodesInFile` arrays (a barrel-heavy repo scans its biggest files once
|
||||
* per referencing symbol otherwise). First-wins insertion preserves exactly
|
||||
* the array-order semantics of the `.find` calls it replaces.
|
||||
*/
|
||||
interface FileExportIndex {
|
||||
byName: Map<string, Node>;
|
||||
defaultComponent: Node | undefined;
|
||||
defaultFnClass: Node | undefined;
|
||||
}
|
||||
const fileExportIndexes = new WeakMap<ResolutionContext, Map<string, FileExportIndex>>();
|
||||
|
||||
function getFileExportIndex(filePath: string, context: ResolutionContext): FileExportIndex {
|
||||
let perFile = fileExportIndexes.get(context);
|
||||
if (!perFile) {
|
||||
perFile = new Map();
|
||||
fileExportIndexes.set(context, perFile);
|
||||
}
|
||||
let idx = perFile.get(filePath);
|
||||
if (!idx) {
|
||||
idx = { byName: new Map(), defaultComponent: undefined, defaultFnClass: undefined };
|
||||
for (const n of context.getNodesInFile(filePath)) {
|
||||
if (!n.isExported) continue;
|
||||
if (!idx.byName.has(n.name)) idx.byName.set(n.name, n);
|
||||
if (idx.defaultComponent === undefined && n.kind === 'component') idx.defaultComponent = n;
|
||||
if (idx.defaultFnClass === undefined && (n.kind === 'function' || n.kind === 'class')) idx.defaultFnClass = n;
|
||||
}
|
||||
perFile.set(filePath, idx);
|
||||
}
|
||||
return idx;
|
||||
}
|
||||
|
||||
/** Drop the per-context memo tables (see ReferenceResolver.clearCaches). */
|
||||
export function clearImportResolverMemos(context: ResolutionContext): void {
|
||||
importPathMemos.delete(context);
|
||||
exportedSymbolMemos.delete(context);
|
||||
fileExportIndexes.delete(context);
|
||||
}
|
||||
|
||||
export function resolveImportPath(
|
||||
importPath: string,
|
||||
fromFile: string,
|
||||
language: Language,
|
||||
context: ResolutionContext
|
||||
): string | null {
|
||||
let memo = importPathMemos.get(context);
|
||||
if (!memo) {
|
||||
memo = new Map();
|
||||
importPathMemos.set(context, memo);
|
||||
}
|
||||
const key = `${language}\0${fromFile}\0${importPath}`;
|
||||
const hit = memo.get(key);
|
||||
if (hit !== undefined || memo.has(key)) return hit ?? null;
|
||||
const resolved = resolveImportPathUncached(importPath, fromFile, language, context);
|
||||
memo.set(key, resolved);
|
||||
return resolved;
|
||||
}
|
||||
|
||||
function resolveImportPathUncached(
|
||||
importPath: string,
|
||||
fromFile: string,
|
||||
language: Language,
|
||||
context: ResolutionContext
|
||||
): string | null {
|
||||
// COBOL COPY/EXEC SQL INCLUDE names a copybook member, not a path — the
|
||||
// compiler searches a library, so we match against indexed file basenames.
|
||||
@@ -1972,12 +2039,45 @@ function findExportedSymbol(
|
||||
context: ResolutionContext,
|
||||
visited: Set<string>,
|
||||
depth = 0
|
||||
): Node | undefined {
|
||||
// Memoize fresh (top-level) lookups only: recursive re-export steps carry a
|
||||
// populated `visited` set, whose contents change the reachable answer.
|
||||
// Every ref to the same imported symbol repeats this exact walk, so the
|
||||
// top-level memo removes the re-export chase + per-file linear scans from
|
||||
// all but the first occurrence.
|
||||
if (depth === 0 && visited.size === 0) {
|
||||
let memo = exportedSymbolMemos.get(context);
|
||||
if (!memo) {
|
||||
memo = new Map();
|
||||
exportedSymbolMemos.set(context, memo);
|
||||
}
|
||||
const key = `${filePath}\0${want.isDefault ? 1 : 0}${want.isNamespace ? 1 : 0}\0${want.exportedName}\0${want.memberName ?? ''}\0${language}`;
|
||||
if (memo.has(key)) return memo.get(key);
|
||||
const result = findExportedSymbolWalk(filePath, want, language, context, visited, depth);
|
||||
memo.set(key, result);
|
||||
return result;
|
||||
}
|
||||
return findExportedSymbolWalk(filePath, want, language, context, visited, depth);
|
||||
}
|
||||
|
||||
function findExportedSymbolWalk(
|
||||
filePath: string,
|
||||
want: {
|
||||
isDefault: boolean;
|
||||
isNamespace: boolean;
|
||||
exportedName: string;
|
||||
memberName: string | null;
|
||||
},
|
||||
language: Language,
|
||||
context: ResolutionContext,
|
||||
visited: Set<string>,
|
||||
depth: number
|
||||
): Node | undefined {
|
||||
if (depth > REEXPORT_MAX_DEPTH) return undefined;
|
||||
if (visited.has(filePath)) return undefined;
|
||||
visited.add(filePath);
|
||||
|
||||
const nodesInFile = context.getNodesInFile(filePath);
|
||||
const exportIndex = getFileExportIndex(filePath, context);
|
||||
|
||||
// 1. Direct hit: the symbol is declared in this file.
|
||||
if (want.isDefault) {
|
||||
@@ -1987,21 +2087,13 @@ function findExportedSymbol(
|
||||
// `.ts`/`.tsx` `export default fn`/`class` case. Without the component
|
||||
// branch, an `export { default as X } from './X.svelte'` barrel never
|
||||
// resolves and the component shows a false 0 callers (#629).
|
||||
const direct =
|
||||
nodesInFile.find((n) => n.isExported && n.kind === 'component') ??
|
||||
nodesInFile.find(
|
||||
(n) => n.isExported && (n.kind === 'function' || n.kind === 'class')
|
||||
);
|
||||
const direct = exportIndex.defaultComponent ?? exportIndex.defaultFnClass;
|
||||
if (direct) return direct;
|
||||
} else if (want.isNamespace && want.memberName) {
|
||||
const direct = nodesInFile.find(
|
||||
(n) => n.name === want.memberName && n.isExported
|
||||
);
|
||||
const direct = exportIndex.byName.get(want.memberName);
|
||||
if (direct) return direct;
|
||||
} else {
|
||||
const direct = nodesInFile.find(
|
||||
(n) => n.name === want.exportedName && n.isExported
|
||||
);
|
||||
const direct = exportIndex.byName.get(want.exportedName);
|
||||
if (direct) return direct;
|
||||
}
|
||||
|
||||
|
||||
+120
-4
@@ -17,7 +17,8 @@ import {
|
||||
ImportMapping,
|
||||
} from './types';
|
||||
import { matchReference, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesKnownFamily } from './name-matcher';
|
||||
import { resolveViaImport, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef } from './import-resolver';
|
||||
import { resolveViaImport, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, clearImportResolverMemos } from './import-resolver';
|
||||
import { ResolverPool, minRefsForPool } from './resolver-pool';
|
||||
import { detectFrameworks } from './frameworks';
|
||||
import { synthesizeCallbackEdges } from './callback-synthesizer';
|
||||
import { createYielder, type MaybeYield } from './cooperative-yield';
|
||||
@@ -372,6 +373,9 @@ export class ReferenceResolver {
|
||||
this.knownNames = null;
|
||||
this.knownFiles = null;
|
||||
this.cachesWarmed = false;
|
||||
// The import-resolver's per-context memos assume the same stable window
|
||||
// as the caches above — drop them together.
|
||||
if (this.context) clearImportResolverMemos(this.context);
|
||||
}
|
||||
|
||||
/** `readFile` through the LRU content cache (null = read failed, also cached). */
|
||||
@@ -1236,7 +1240,10 @@ export class ReferenceResolver {
|
||||
} else {
|
||||
unresolved.push(ref);
|
||||
}
|
||||
await maybeYield();
|
||||
// Fast-path the per-ref yield check: awaiting the async no-op costs a
|
||||
// microtask hop per ref, which dominates at ~10⁵ refs (see MaybeYield).
|
||||
const y = maybeYield();
|
||||
if (y) await y;
|
||||
}
|
||||
|
||||
return {
|
||||
@@ -1251,6 +1258,64 @@ export class ReferenceResolver {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a list of refs and return everything the ADMISSION side needs to
|
||||
* persist the outcome: resolutions, failures, the deferred post-pass refs
|
||||
* this run produced (drained, so the caller owns routing them), and stats.
|
||||
* This is the resolver-worker entry point — it runs the exact per-ref loop
|
||||
* of resolveBatchYielding, minus the main-thread yields (worker threads have
|
||||
* no watchdog heartbeat to starve). Results are in input order.
|
||||
*/
|
||||
resolveListForAdmission(refs: UnresolvedReference[]): {
|
||||
resolved: ResolvedRef[];
|
||||
unresolved: UnresolvedRef[];
|
||||
deferredChain: UnresolvedRef[];
|
||||
deferredThisMember: UnresolvedRef[];
|
||||
byMethod: Record<string, number>;
|
||||
} {
|
||||
this.warmCaches();
|
||||
const resolved: ResolvedRef[] = [];
|
||||
const unresolved: UnresolvedRef[] = [];
|
||||
const byMethod: Record<string, number> = {};
|
||||
for (const raw of refs) {
|
||||
const ref: UnresolvedRef = {
|
||||
fromNodeId: raw.fromNodeId,
|
||||
referenceName: raw.referenceName,
|
||||
referenceKind: raw.referenceKind,
|
||||
line: raw.line,
|
||||
column: raw.column,
|
||||
filePath: raw.filePath || this.getFilePathFromNodeId(raw.fromNodeId),
|
||||
language: raw.language || this.getLanguageFromNodeId(raw.fromNodeId),
|
||||
rowId: raw.rowId,
|
||||
};
|
||||
const result = this.resolveOne(ref);
|
||||
if (result) {
|
||||
resolved.push(result);
|
||||
byMethod[result.resolvedBy] = (byMethod[result.resolvedBy] || 0) + 1;
|
||||
} else {
|
||||
unresolved.push(ref);
|
||||
}
|
||||
}
|
||||
return {
|
||||
resolved,
|
||||
unresolved,
|
||||
deferredChain: this.deferredChainRefs.splice(0),
|
||||
deferredThisMember: this.deferredThisMemberRefs.splice(0),
|
||||
byMethod,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-queue deferred post-pass refs produced by resolver workers, preserving
|
||||
* their admission order so resolveChainedCallsViaConformance /
|
||||
* resolveDeferredThisMemberRefs process them exactly as the sequential path
|
||||
* would have.
|
||||
*/
|
||||
appendDeferredFromWorkers(deferredChain: UnresolvedRef[], deferredThisMember: UnresolvedRef[]): void {
|
||||
this.deferredChainRefs.push(...deferredChain);
|
||||
this.deferredThisMemberRefs.push(...deferredThisMember);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve and persist in batches to keep memory bounded.
|
||||
* Processes unresolved references in chunks, persisting edges and cleaning
|
||||
@@ -1259,7 +1324,12 @@ export class ReferenceResolver {
|
||||
async resolveAndPersistBatched(
|
||||
onProgress?: (current: number, total: number) => void,
|
||||
batchSize: number = 5000,
|
||||
onSynthesisProgress?: (done: number, total: number) => void
|
||||
onSynthesisProgress?: (done: number, total: number) => void,
|
||||
// When provided, big batches fan out across a read-only resolver-worker
|
||||
// pool with results admitted in canonical order (see resolver-pool.ts).
|
||||
// Sequential fallback on any pool failure. CODEGRAPH_NO_PARALLEL_RESOLVE=1
|
||||
// disables entirely.
|
||||
parallel?: { dbPath: string }
|
||||
): Promise<ResolutionResult> {
|
||||
// Resolution runs on the indexer's MAIN thread, and the #850 liveness
|
||||
// watchdog SIGKILLs a process whose event loop stalls past its window (60s
|
||||
@@ -1280,16 +1350,59 @@ export class ReferenceResolver {
|
||||
byMethod: {} as Record<string, number>,
|
||||
};
|
||||
|
||||
// Parallel pool, started immediately but never awaited up front: early
|
||||
// batches run sequentially while the workers boot (module load + readonly
|
||||
// DB open + framework detect + cache warm ≈ hundreds of ms), and the loop
|
||||
// switches to fan-out the moment the pool reports ready — so pool boot
|
||||
// costs zero wall-clock. Any failure downgrades to sequential permanently.
|
||||
let pool: ResolverPool | null = null;
|
||||
let poolReady = false;
|
||||
if (parallel && total >= minRefsForPool()) {
|
||||
pool = ResolverPool.tryCreate(parallel.dbPath, this.projectRoot);
|
||||
pool?.ready().then(
|
||||
() => { poolReady = true; },
|
||||
() => { void pool?.destroy().catch(() => undefined); pool = null; }
|
||||
);
|
||||
}
|
||||
|
||||
// Process in batches. We always read from offset 0 because every ref the
|
||||
// batch processed leaves the pending set (resolved rows are deleted,
|
||||
// unresolvable ones flip to status='failed'), shifting the remaining
|
||||
// pending rows forward.
|
||||
let prevRemaining = Number.POSITIVE_INFINITY;
|
||||
try {
|
||||
while (true) {
|
||||
const batch = this.queries.getUnresolvedReferencesBatch(0, batchSize);
|
||||
if (batch.length === 0) break;
|
||||
|
||||
const result = await this.resolveBatchYielding(batch, maybeYield);
|
||||
let result: ResolutionResult;
|
||||
if (pool && poolReady && ResolverPool.worthParallel(batch.length)) {
|
||||
try {
|
||||
const out = await pool.resolveBatch(batch);
|
||||
// Deferred post-pass refs ride back from the workers; re-queue them
|
||||
// in admission order so the post-passes see the sequential order.
|
||||
this.appendDeferredFromWorkers(out.deferredChain, out.deferredThisMember);
|
||||
result = {
|
||||
resolved: out.resolved,
|
||||
unresolved: out.unresolved,
|
||||
stats: {
|
||||
total: batch.length,
|
||||
resolved: out.resolved.length,
|
||||
unresolved: out.unresolved.length,
|
||||
byMethod: out.byMethod,
|
||||
},
|
||||
};
|
||||
} catch (err) {
|
||||
logDebug('Parallel resolution failed; falling back to sequential', {
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
await pool.destroy().catch(() => undefined);
|
||||
pool = null;
|
||||
result = await this.resolveBatchYielding(batch, maybeYield);
|
||||
}
|
||||
} else {
|
||||
result = await this.resolveBatchYielding(batch, maybeYield);
|
||||
}
|
||||
|
||||
// Persist in bounded sub-transactions with yields between: a whole
|
||||
// batch's edge insert / keyed deletes are otherwise one solid
|
||||
@@ -1370,6 +1483,9 @@ export class ReferenceResolver {
|
||||
if (remaining >= prevRemaining) break;
|
||||
prevRemaining = remaining;
|
||||
}
|
||||
} finally {
|
||||
if (pool) await pool.destroy().catch(() => undefined);
|
||||
}
|
||||
|
||||
// Dynamic-edge synthesis: now that all base `calls` edges are persisted,
|
||||
// synthesize observer/callback dispatch edges (dispatcher → registered
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
/**
|
||||
* ResolverPool — main-thread client for the parallel-resolution workers.
|
||||
*
|
||||
* resolveBatch() splits a rowid-ordered batch into ordered chunks, fans the
|
||||
* chunks across the pool, and reassembles the results IN CHUNK ORDER, so the
|
||||
* caller's admission (edge inserts, row cleanup, failure parking, deferred
|
||||
* post-pass queues) is byte-for-byte the sequence the single-threaded loop
|
||||
* would have produced. Any worker failure fails the batch — the caller falls
|
||||
* back to the sequential path. Kill switch: CODEGRAPH_NO_PARALLEL_RESOLVE=1.
|
||||
*/
|
||||
|
||||
import { Worker } from 'worker_threads';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import type { UnresolvedReference } from '../types';
|
||||
import type { ResolvedRef, UnresolvedRef } from './types';
|
||||
|
||||
export interface ChunkResult {
|
||||
resolved: ResolvedRef[];
|
||||
unresolved: UnresolvedRef[];
|
||||
deferredChain: UnresolvedRef[];
|
||||
deferredThisMember: UnresolvedRef[];
|
||||
byMethod: Record<string, number>;
|
||||
}
|
||||
|
||||
interface PoolWorker {
|
||||
worker: Worker;
|
||||
ready: Promise<void>;
|
||||
busy: number;
|
||||
}
|
||||
|
||||
const MIN_PARALLEL_BATCH = 1000;
|
||||
const CHUNK_SIZE = 500;
|
||||
|
||||
/**
|
||||
* Minimum TOTAL pending refs before the pool is created at all. Pool boot
|
||||
* (module load + readonly DB open + framework detect + cache warm, times N
|
||||
* workers) costs real CPU that CONTENDS with sequential resolution on the
|
||||
* same cores — measured on a medium repo (~40k refs, ~1.2s of resolution)
|
||||
* the pool made indexing slower. It pays off when resolution runs for tens
|
||||
* of seconds to minutes (large JVM/Spring-class repos). Override:
|
||||
* CODEGRAPH_PARALLEL_RESOLVE_MIN=<refs> (0 forces the pool on).
|
||||
*/
|
||||
export function minRefsForPool(): number {
|
||||
const raw = process.env.CODEGRAPH_PARALLEL_RESOLVE_MIN;
|
||||
if (raw !== undefined) {
|
||||
const parsed = Number.parseInt(raw, 10);
|
||||
if (Number.isFinite(parsed) && parsed >= 0) return parsed;
|
||||
}
|
||||
return 150_000;
|
||||
}
|
||||
|
||||
export class ResolverPool {
|
||||
private workers: PoolWorker[] = [];
|
||||
private nextId = 0;
|
||||
private waiters = new Map<number, { resolve: (r: ChunkResult) => void; reject: (e: Error) => void }>();
|
||||
private failed: Error | null = null;
|
||||
|
||||
/**
|
||||
* Create a pool when the compiled worker exists (absent when running from
|
||||
* source in tests → callers use the sequential path), the kill switch is
|
||||
* off, and the machine has cores to spare. Returns null otherwise.
|
||||
*/
|
||||
static tryCreate(dbPath: string, projectRoot: string): ResolverPool | null {
|
||||
if (process.env.CODEGRAPH_NO_PARALLEL_RESOLVE === '1') return null;
|
||||
const workerScript = path.join(__dirname, 'resolver-worker.js');
|
||||
if (!fs.existsSync(workerScript)) return null;
|
||||
const size = Math.max(1, Math.min(os.cpus().length - 2, 6));
|
||||
if (size < 2) return null;
|
||||
try {
|
||||
return new ResolverPool(workerScript, dbPath, projectRoot, size);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private constructor(workerScript: string, dbPath: string, projectRoot: string, size: number) {
|
||||
for (let i = 0; i < size; i++) {
|
||||
const worker = new Worker(workerScript);
|
||||
let readyResolve!: () => void;
|
||||
let readyReject!: (e: Error) => void;
|
||||
const ready = new Promise<void>((resolve, reject) => {
|
||||
readyResolve = resolve;
|
||||
readyReject = reject;
|
||||
});
|
||||
const pw: PoolWorker = { worker, ready, busy: 0 };
|
||||
worker.on('message', (msg: { type: string; id?: number; message?: string } & Partial<ChunkResult>) => {
|
||||
if (msg.type === 'ready') {
|
||||
readyResolve();
|
||||
} else if (msg.type === 'result' && msg.id !== undefined) {
|
||||
pw.busy--;
|
||||
const waiter = this.waiters.get(msg.id);
|
||||
this.waiters.delete(msg.id);
|
||||
waiter?.resolve({
|
||||
resolved: msg.resolved!,
|
||||
unresolved: msg.unresolved!,
|
||||
deferredChain: msg.deferredChain!,
|
||||
deferredThisMember: msg.deferredThisMember!,
|
||||
byMethod: msg.byMethod!,
|
||||
});
|
||||
} else if (msg.type === 'error') {
|
||||
pw.busy--;
|
||||
const err = new Error(`resolver worker: ${msg.message}`);
|
||||
if (msg.id !== undefined && this.waiters.has(msg.id)) {
|
||||
const waiter = this.waiters.get(msg.id)!;
|
||||
this.waiters.delete(msg.id);
|
||||
waiter.reject(err);
|
||||
} else {
|
||||
this.fail(err);
|
||||
}
|
||||
}
|
||||
});
|
||||
worker.on('error', (err) => {
|
||||
this.fail(err instanceof Error ? err : new Error(String(err)));
|
||||
readyReject(this.failed!);
|
||||
});
|
||||
worker.on('exit', (code) => {
|
||||
if (code !== 0) {
|
||||
this.fail(new Error(`resolver worker exited with code ${code}`));
|
||||
readyReject(this.failed!);
|
||||
}
|
||||
});
|
||||
worker.postMessage({ type: 'open', dbPath, projectRoot });
|
||||
this.workers.push(pw);
|
||||
}
|
||||
}
|
||||
|
||||
private fail(err: Error): void {
|
||||
if (!this.failed) this.failed = err;
|
||||
for (const [, waiter] of this.waiters) waiter.reject(this.failed);
|
||||
this.waiters.clear();
|
||||
}
|
||||
|
||||
/** Whether this batch is worth fanning out. */
|
||||
static worthParallel(batchLength: number): boolean {
|
||||
return batchLength >= MIN_PARALLEL_BATCH;
|
||||
}
|
||||
|
||||
async ready(): Promise<void> {
|
||||
await Promise.all(this.workers.map((w) => w.ready));
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve `refs` across the pool. Chunks preserve input order; the returned
|
||||
* arrays are the in-order concatenation of the chunk results.
|
||||
*/
|
||||
async resolveBatch(refs: UnresolvedReference[]): Promise<ChunkResult> {
|
||||
if (this.failed) throw this.failed;
|
||||
const chunkPromises: Promise<ChunkResult>[] = [];
|
||||
for (let i = 0; i < refs.length; i += CHUNK_SIZE) {
|
||||
const chunk = refs.slice(i, i + CHUNK_SIZE);
|
||||
const id = this.nextId++;
|
||||
// Least-busy dispatch keeps workers evenly loaded regardless of chunk
|
||||
// cost variance; result order is fixed by the promise array, not by
|
||||
// completion order.
|
||||
const pw = this.workers.reduce((a, b) => (b.busy < a.busy ? b : a));
|
||||
pw.busy++;
|
||||
chunkPromises.push(
|
||||
new Promise<ChunkResult>((resolve, reject) => {
|
||||
this.waiters.set(id, { resolve, reject });
|
||||
pw.worker.postMessage({ type: 'resolve', id, refs: chunk });
|
||||
})
|
||||
);
|
||||
}
|
||||
const chunks = await Promise.all(chunkPromises);
|
||||
const out: ChunkResult = { resolved: [], unresolved: [], deferredChain: [], deferredThisMember: [], byMethod: {} };
|
||||
for (const c of chunks) {
|
||||
out.resolved.push(...c.resolved);
|
||||
out.unresolved.push(...c.unresolved);
|
||||
out.deferredChain.push(...c.deferredChain);
|
||||
out.deferredThisMember.push(...c.deferredThisMember);
|
||||
for (const [k, v] of Object.entries(c.byMethod)) out.byMethod[k] = (out.byMethod[k] || 0) + v;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async destroy(): Promise<void> {
|
||||
await Promise.all(
|
||||
this.workers.map(
|
||||
(pw) =>
|
||||
new Promise<void>((resolve) => {
|
||||
const t = setTimeout(() => {
|
||||
void pw.worker.terminate().then(() => resolve());
|
||||
}, 5000);
|
||||
pw.worker.once('exit', () => {
|
||||
clearTimeout(t);
|
||||
resolve();
|
||||
});
|
||||
pw.worker.postMessage({ type: 'close' });
|
||||
})
|
||||
)
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
/**
|
||||
* Resolver worker — one member of the parallel-resolution pool.
|
||||
*
|
||||
* Opens the project database READ-ONLY on its own connection and hosts a full
|
||||
* ReferenceResolver over it. The main thread partitions each resolution batch
|
||||
* into ordered chunks, fans them across the pool, and ADMITS the results
|
||||
* sequentially in chunk order — so edge insertion order (and every cleanup /
|
||||
* parking side effect) is identical to the single-threaded loop. Workers only
|
||||
* ever read; all writes stay on the main thread.
|
||||
*
|
||||
* Visibility note: the sequential baseline resolves every ref of a batch
|
||||
* against the DB state committed BEFORE that batch (edges persist after the
|
||||
* whole batch resolves). Workers read exactly that same committed state, so
|
||||
* per-ref inputs match the baseline ref-for-ref.
|
||||
*/
|
||||
|
||||
// Compile cache FIRST — same worker-boot rationale as parse-worker.ts.
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||
(require('node:module') as { enableCompileCache?: () => void }).enableCompileCache?.();
|
||||
} catch { /* cache is best-effort */ }
|
||||
|
||||
import { parentPort } from 'worker_threads';
|
||||
import { createDatabase, SqliteDatabase } from '../db/sqlite-adapter';
|
||||
import { QueryBuilder } from '../db/queries';
|
||||
import { ReferenceResolver } from './index';
|
||||
import type { UnresolvedReference } from '../types';
|
||||
|
||||
if (!parentPort) {
|
||||
throw new Error('resolver-worker must be run as a worker thread');
|
||||
}
|
||||
const port = parentPort;
|
||||
|
||||
let db: SqliteDatabase | null = null;
|
||||
let resolver: ReferenceResolver | null = null;
|
||||
|
||||
type InMessage =
|
||||
| { type: 'open'; dbPath: string; projectRoot: string }
|
||||
| { type: 'resolve'; id: number; refs: UnresolvedReference[] }
|
||||
| { type: 'close' };
|
||||
|
||||
port.on('message', (msg: InMessage) => {
|
||||
try {
|
||||
switch (msg.type) {
|
||||
case 'open': {
|
||||
const created = createDatabase(msg.dbPath, { readOnly: true });
|
||||
db = created.db;
|
||||
db.pragma('busy_timeout = 5000');
|
||||
db.pragma('cache_size = -32000');
|
||||
const queries = new QueryBuilder(db);
|
||||
resolver = new ReferenceResolver(msg.projectRoot, queries);
|
||||
resolver.initialize();
|
||||
port.postMessage({ type: 'ready' });
|
||||
break;
|
||||
}
|
||||
case 'resolve': {
|
||||
if (!resolver) throw new Error('resolver-worker: resolve before open');
|
||||
const out = resolver.resolveListForAdmission(msg.refs);
|
||||
port.postMessage({ type: 'result', id: msg.id, ...out });
|
||||
break;
|
||||
}
|
||||
case 'close': {
|
||||
try {
|
||||
db?.close();
|
||||
} catch {
|
||||
/* already closed */
|
||||
}
|
||||
process.exit(0);
|
||||
break;
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
port.postMessage({
|
||||
type: 'error',
|
||||
id: (msg as { id?: number }).id,
|
||||
message: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user