perf(index): faster fresh indexing + parallel reference resolution, byte-identical graphs (#1305)
* perf(index): ~34% faster fresh indexing, byte-identical graphs Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed the main thread as the critical path: per-row SQLite statement calls, repeated import-resolution walks, and per-row FTS trigger firings, with the parse workers ~75% idle behind it. This lands the semantics-preserving tranche of fixes: - Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments) behind cached per-batch-size prepared statements; row order preserved, so rowid-based resolution determinism (#1015) is unchanged. - storeFileBundle: one transaction per file instead of four; nested transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no caller depended on nested rollback). - Dedicated store-writer thread for the fresh-DB bulk path (bundles applied in file order on a single writer connection; main thread does no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1. - Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load, rebuild once at the end; crash inside the window self-heals on the next open. - Per-context memos for resolveImportPath/findExportedSymbol + a per-file exported-symbol index, invalidated exactly where clearCaches() already resets the resolver's own caches. - Fast-init on completely fresh DBs (journal in memory, no fsync until the index completes; interrupted init re-runs from scratch). Kill switch: CODEGRAPH_NO_FAST_INIT=1. - MaybeYield returns undefined on the not-due path so per-ref yield checks stop paying a promise + microtask hop each. - Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and worker entry points. Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series). Graph dumps byte-identical across init, re-index, and sync paths; full suite green (2403 passed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(resolution): parallel reference resolution with canonical admission Fan resolution batches across a pool of read-only worker threads, each hosting a full ReferenceResolver over its own SQLite connection; results are admitted on the main thread in chunk order, so edge insertion order, row cleanup, failure parking, and deferred post-pass queues are exactly the sequence the single-threaded loop produces. Per-ref inputs match the baseline because the sequential path already resolves each batch against the state committed BEFORE that batch. Validated byte-identical on excalidraw (pool forced on) and apache/dubbo (4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical graph dumps (91,495 nodes / 223,953 edges). The pool only engages when total pending refs clear a threshold (default 150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1 to disable): measured on a ~58k-ref repo the workers' boot CPU contends with resolution on the same cores and makes indexing slower, so small repos keep the sequential path. When fast-init left the DB in memory-journal mode, WAL is restored before resolution only when the pool will run (readers + rollback-journal writers don't mix). Also: sqlite adapter readOnly open support. TreeCursor spine rewrite of the body walker was built, measured neutral on real repos and equal in a 20k-child microbench (web-tree-sitter's namedChild(i) is not quadratic in this binding), and rejected — per-node JS<->WASM marshaling is the floor, which a traversal swap cannot remove. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Fable 5
Claude Fable 5
parent
246aee8373
commit
5736e24bb6
+59
-8
@@ -55,6 +55,7 @@ import { deriveProjectNameTokens } from './search/query-utils';
|
||||
import { CodeGraphPackageVersion } from './mcp/version';
|
||||
import { segmentLookupVariants, splitIdentifierSegments } from './search/identifier-segments';
|
||||
import { createYielder } from './resolution/cooperative-yield';
|
||||
import { minRefsForPool } from './resolution/resolver-pool';
|
||||
|
||||
// Re-export types for consumers
|
||||
export * from './types';
|
||||
@@ -445,7 +446,21 @@ export class CodeGraph {
|
||||
// the final fold-up before the interval is restored in the finally.
|
||||
// Kill switch: CODEGRAPH_NO_WAL_DEFER=1. Non-WAL journal modes (some
|
||||
// network filesystems) have no WAL to defer — skip.
|
||||
const deferWal = process.env.CODEGRAPH_NO_WAL_DEFER !== '1' && this.db.getJournalMode() === 'wal';
|
||||
// Fast-init: on a COMPLETELY fresh DB, trade crash-durability for speed
|
||||
// during the bulk build (journal in memory, no fsync). Safe because the
|
||||
// DB is disposable until the index completes — index_state stays
|
||||
// 'indexing' and a crashed init is re-run from scratch; existing DBs
|
||||
// (re-index/sync) never take this path. Kill switch:
|
||||
// CODEGRAPH_NO_FAST_INIT=1 (same pattern as CODEGRAPH_NO_WAL_DEFER).
|
||||
const freshDb = this.queries.getNodeAndEdgeCount().nodes === 0;
|
||||
const fastInit = process.env.CODEGRAPH_NO_FAST_INIT !== '1' && freshDb;
|
||||
if (fastInit) {
|
||||
try {
|
||||
this.db.getDb().pragma('journal_mode = MEMORY');
|
||||
this.db.getDb().pragma('synchronous = OFF');
|
||||
} catch { /* keep WAL */ }
|
||||
}
|
||||
const deferWal = !fastInit && process.env.CODEGRAPH_NO_WAL_DEFER !== '1' && this.db.getJournalMode() === 'wal';
|
||||
let walValve: WalCheckpointValve | null = null;
|
||||
let priorAutocheckpoint = 1000;
|
||||
if (deferWal) {
|
||||
@@ -470,12 +485,25 @@ export class CodeGraph {
|
||||
// path as every file (re-)indexes below — so a full index is also the
|
||||
// orphan-cleanup pass for names deleted since the last one.
|
||||
try { this.queries.clearNameSegmentVocab(); } catch { /* vocab is advisory — never fail an index over it */ }
|
||||
const result = await this.orchestrator.indexAll(
|
||||
options.onProgress,
|
||||
options.signal,
|
||||
options.verbose,
|
||||
walValve ? () => walValve!.backpressure() : undefined
|
||||
);
|
||||
// Bulk FTS mode for the mass-insert phase: drop the per-row FTS sync
|
||||
// triggers, rebuild nodes_fts once from the nodes table afterwards.
|
||||
// Crash inside the window is healed on the next DatabaseConnection.open.
|
||||
this.db.beginBulkNodeLoad();
|
||||
let result: IndexResult;
|
||||
try {
|
||||
result = await this.orchestrator.indexAll(
|
||||
options.onProgress,
|
||||
options.signal,
|
||||
options.verbose,
|
||||
walValve ? () => walValve!.backpressure() : undefined,
|
||||
// Store-writer offload is fresh-DB-only: with any pre-existing
|
||||
// data the store path must read (existing-file checks, cross-file
|
||||
// edge snapshots) and delete, which belongs on one thread.
|
||||
freshDb ? { dbPath: this.db.getPath(), fastInit } : null
|
||||
);
|
||||
} finally {
|
||||
this.db.endBulkNodeLoad();
|
||||
}
|
||||
|
||||
// Fold the parse phase's WAL BEFORE the first post-parse reads
|
||||
// (resolver re-init and resolution both read on the main thread):
|
||||
@@ -503,6 +531,19 @@ export class CodeGraph {
|
||||
// Get count without loading all refs into memory
|
||||
const unresolvedCount = this.queries.getUnresolvedReferencesCount();
|
||||
|
||||
// Fast-init leaves the DB in memory-journal (rollback) mode, where
|
||||
// the parallel resolver pool's read connections would contend with
|
||||
// the main writer's exclusive commits. When the pool will actually
|
||||
// run (enough pending refs), restore WAL BEFORE resolution so
|
||||
// readers never block the writer; otherwise stay in the fast mode
|
||||
// until the finally — sequential resolution has no readers.
|
||||
if (fastInit && unresolvedCount >= minRefsForPool()) {
|
||||
try {
|
||||
this.db.getDb().pragma('synchronous = NORMAL');
|
||||
this.db.getDb().pragma('journal_mode = WAL');
|
||||
} catch { /* keep current mode; resolution still works sequentially */ }
|
||||
}
|
||||
|
||||
options.onProgress?.({
|
||||
phase: 'resolving',
|
||||
current: 0,
|
||||
@@ -619,6 +660,14 @@ export class CodeGraph {
|
||||
if (deferWal) {
|
||||
try { this.db.setWalAutocheckpoint(priorAutocheckpoint); } catch { /* connection may be closing */ }
|
||||
}
|
||||
if (fastInit) {
|
||||
// Back to the durable defaults; journal_mode=WAL folds the MEMORY
|
||||
// journal state into a normal WAL-mode database file.
|
||||
try {
|
||||
this.db.getDb().pragma('synchronous = NORMAL');
|
||||
this.db.getDb().pragma('journal_mode = WAL');
|
||||
} catch { /* connection may be closing */ }
|
||||
}
|
||||
this.fileLock.release();
|
||||
}
|
||||
});
|
||||
@@ -1032,7 +1081,9 @@ export class CodeGraph {
|
||||
onProgress?: (current: number, total: number) => void,
|
||||
onSynthesisProgress?: (done: number, total: number) => void
|
||||
): Promise<ResolutionResult> {
|
||||
return this.resolver.resolveAndPersistBatched(onProgress, undefined, onSynthesisProgress);
|
||||
return this.resolver.resolveAndPersistBatched(onProgress, undefined, onSynthesisProgress, {
|
||||
dbPath: this.db.getPath(),
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user