perf(index): faster fresh indexing + parallel reference resolution, byte-identical graphs (#1305)
* perf(index): ~34% faster fresh indexing, byte-identical graphs Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed the main thread as the critical path: per-row SQLite statement calls, repeated import-resolution walks, and per-row FTS trigger firings, with the parse workers ~75% idle behind it. This lands the semantics-preserving tranche of fixes: - Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments) behind cached per-batch-size prepared statements; row order preserved, so rowid-based resolution determinism (#1015) is unchanged. - storeFileBundle: one transaction per file instead of four; nested transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no caller depended on nested rollback). - Dedicated store-writer thread for the fresh-DB bulk path (bundles applied in file order on a single writer connection; main thread does no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1. - Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load, rebuild once at the end; crash inside the window self-heals on the next open. - Per-context memos for resolveImportPath/findExportedSymbol + a per-file exported-symbol index, invalidated exactly where clearCaches() already resets the resolver's own caches. - Fast-init on completely fresh DBs (journal in memory, no fsync until the index completes; interrupted init re-runs from scratch). Kill switch: CODEGRAPH_NO_FAST_INIT=1. - MaybeYield returns undefined on the not-due path so per-ref yield checks stop paying a promise + microtask hop each. - Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and worker entry points. Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series). Graph dumps byte-identical across init, re-index, and sync paths; full suite green (2403 passed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(resolution): parallel reference resolution with canonical admission Fan resolution batches across a pool of read-only worker threads, each hosting a full ReferenceResolver over its own SQLite connection; results are admitted on the main thread in chunk order, so edge insertion order, row cleanup, failure parking, and deferred post-pass queues are exactly the sequence the single-threaded loop produces. Per-ref inputs match the baseline because the sequential path already resolves each batch against the state committed BEFORE that batch. Validated byte-identical on excalidraw (pool forced on) and apache/dubbo (4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical graph dumps (91,495 nodes / 223,953 edges). The pool only engages when total pending refs clear a threshold (default 150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1 to disable): measured on a ~58k-ref repo the workers' boot CPU contends with resolution on the same cores and makes indexing slower, so small repos keep the sequential path. When fast-init left the DB in memory-journal mode, WAL is restored before resolution only when the pool will run (readers + rollback-journal writers don't mix). Also: sqlite adapter readOnly open support. TreeCursor spine rewrite of the body walker was built, measured neutral on real repos and equal in a 20k-child microbench (web-tree-sitter's namedChild(i) is not quadratic in this binding), and rejected — per-node JS<->WASM marshaling is the floor, which a traversal swap cannot remove. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Fable 5
Claude Fable 5
parent
246aee8373
commit
5736e24bb6
@@ -55,11 +55,78 @@ export function isNixPathImportRef(ref: UnresolvedRef): boolean {
|
||||
/**
|
||||
* Resolve an import path to an actual file
|
||||
*/
|
||||
// Per-context memos for the two hottest pure lookups on the resolution path:
|
||||
// import-specifier → file resolution and exported-symbol lookup. Both are pure
|
||||
// given a stable file set + node table, which is exactly the window between
|
||||
// ReferenceResolver.clearCaches() calls — clearImportResolverMemos() is invoked
|
||||
// there, so the staleness discipline matches the resolver's own caches.
|
||||
const importPathMemos = new WeakMap<ResolutionContext, Map<string, string | null>>();
|
||||
const exportedSymbolMemos = new WeakMap<ResolutionContext, Map<string, Node | undefined>>();
|
||||
|
||||
/**
|
||||
* Per-file index of exported symbols, replacing repeated linear `.find`s over
|
||||
* `getNodesInFile` arrays (a barrel-heavy repo scans its biggest files once
|
||||
* per referencing symbol otherwise). First-wins insertion preserves exactly
|
||||
* the array-order semantics of the `.find` calls it replaces.
|
||||
*/
|
||||
interface FileExportIndex {
|
||||
byName: Map<string, Node>;
|
||||
defaultComponent: Node | undefined;
|
||||
defaultFnClass: Node | undefined;
|
||||
}
|
||||
const fileExportIndexes = new WeakMap<ResolutionContext, Map<string, FileExportIndex>>();
|
||||
|
||||
function getFileExportIndex(filePath: string, context: ResolutionContext): FileExportIndex {
|
||||
let perFile = fileExportIndexes.get(context);
|
||||
if (!perFile) {
|
||||
perFile = new Map();
|
||||
fileExportIndexes.set(context, perFile);
|
||||
}
|
||||
let idx = perFile.get(filePath);
|
||||
if (!idx) {
|
||||
idx = { byName: new Map(), defaultComponent: undefined, defaultFnClass: undefined };
|
||||
for (const n of context.getNodesInFile(filePath)) {
|
||||
if (!n.isExported) continue;
|
||||
if (!idx.byName.has(n.name)) idx.byName.set(n.name, n);
|
||||
if (idx.defaultComponent === undefined && n.kind === 'component') idx.defaultComponent = n;
|
||||
if (idx.defaultFnClass === undefined && (n.kind === 'function' || n.kind === 'class')) idx.defaultFnClass = n;
|
||||
}
|
||||
perFile.set(filePath, idx);
|
||||
}
|
||||
return idx;
|
||||
}
|
||||
|
||||
/** Drop the per-context memo tables (see ReferenceResolver.clearCaches). */
|
||||
export function clearImportResolverMemos(context: ResolutionContext): void {
|
||||
importPathMemos.delete(context);
|
||||
exportedSymbolMemos.delete(context);
|
||||
fileExportIndexes.delete(context);
|
||||
}
|
||||
|
||||
export function resolveImportPath(
|
||||
importPath: string,
|
||||
fromFile: string,
|
||||
language: Language,
|
||||
context: ResolutionContext
|
||||
): string | null {
|
||||
let memo = importPathMemos.get(context);
|
||||
if (!memo) {
|
||||
memo = new Map();
|
||||
importPathMemos.set(context, memo);
|
||||
}
|
||||
const key = `${language}\0${fromFile}\0${importPath}`;
|
||||
const hit = memo.get(key);
|
||||
if (hit !== undefined || memo.has(key)) return hit ?? null;
|
||||
const resolved = resolveImportPathUncached(importPath, fromFile, language, context);
|
||||
memo.set(key, resolved);
|
||||
return resolved;
|
||||
}
|
||||
|
||||
function resolveImportPathUncached(
|
||||
importPath: string,
|
||||
fromFile: string,
|
||||
language: Language,
|
||||
context: ResolutionContext
|
||||
): string | null {
|
||||
// COBOL COPY/EXEC SQL INCLUDE names a copybook member, not a path — the
|
||||
// compiler searches a library, so we match against indexed file basenames.
|
||||
@@ -1972,12 +2039,45 @@ function findExportedSymbol(
|
||||
context: ResolutionContext,
|
||||
visited: Set<string>,
|
||||
depth = 0
|
||||
): Node | undefined {
|
||||
// Memoize fresh (top-level) lookups only: recursive re-export steps carry a
|
||||
// populated `visited` set, whose contents change the reachable answer.
|
||||
// Every ref to the same imported symbol repeats this exact walk, so the
|
||||
// top-level memo removes the re-export chase + per-file linear scans from
|
||||
// all but the first occurrence.
|
||||
if (depth === 0 && visited.size === 0) {
|
||||
let memo = exportedSymbolMemos.get(context);
|
||||
if (!memo) {
|
||||
memo = new Map();
|
||||
exportedSymbolMemos.set(context, memo);
|
||||
}
|
||||
const key = `${filePath}\0${want.isDefault ? 1 : 0}${want.isNamespace ? 1 : 0}\0${want.exportedName}\0${want.memberName ?? ''}\0${language}`;
|
||||
if (memo.has(key)) return memo.get(key);
|
||||
const result = findExportedSymbolWalk(filePath, want, language, context, visited, depth);
|
||||
memo.set(key, result);
|
||||
return result;
|
||||
}
|
||||
return findExportedSymbolWalk(filePath, want, language, context, visited, depth);
|
||||
}
|
||||
|
||||
function findExportedSymbolWalk(
|
||||
filePath: string,
|
||||
want: {
|
||||
isDefault: boolean;
|
||||
isNamespace: boolean;
|
||||
exportedName: string;
|
||||
memberName: string | null;
|
||||
},
|
||||
language: Language,
|
||||
context: ResolutionContext,
|
||||
visited: Set<string>,
|
||||
depth: number
|
||||
): Node | undefined {
|
||||
if (depth > REEXPORT_MAX_DEPTH) return undefined;
|
||||
if (visited.has(filePath)) return undefined;
|
||||
visited.add(filePath);
|
||||
|
||||
const nodesInFile = context.getNodesInFile(filePath);
|
||||
const exportIndex = getFileExportIndex(filePath, context);
|
||||
|
||||
// 1. Direct hit: the symbol is declared in this file.
|
||||
if (want.isDefault) {
|
||||
@@ -1987,21 +2087,13 @@ function findExportedSymbol(
|
||||
// `.ts`/`.tsx` `export default fn`/`class` case. Without the component
|
||||
// branch, an `export { default as X } from './X.svelte'` barrel never
|
||||
// resolves and the component shows a false 0 callers (#629).
|
||||
const direct =
|
||||
nodesInFile.find((n) => n.isExported && n.kind === 'component') ??
|
||||
nodesInFile.find(
|
||||
(n) => n.isExported && (n.kind === 'function' || n.kind === 'class')
|
||||
);
|
||||
const direct = exportIndex.defaultComponent ?? exportIndex.defaultFnClass;
|
||||
if (direct) return direct;
|
||||
} else if (want.isNamespace && want.memberName) {
|
||||
const direct = nodesInFile.find(
|
||||
(n) => n.name === want.memberName && n.isExported
|
||||
);
|
||||
const direct = exportIndex.byName.get(want.memberName);
|
||||
if (direct) return direct;
|
||||
} else {
|
||||
const direct = nodesInFile.find(
|
||||
(n) => n.name === want.exportedName && n.isExported
|
||||
);
|
||||
const direct = exportIndex.byName.get(want.exportedName);
|
||||
if (direct) return direct;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user