perf(index): faster fresh indexing + parallel reference resolution, byte-identical graphs (#1305)
* perf(index): ~34% faster fresh indexing, byte-identical graphs Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed the main thread as the critical path: per-row SQLite statement calls, repeated import-resolution walks, and per-row FTS trigger firings, with the parse workers ~75% idle behind it. This lands the semantics-preserving tranche of fixes: - Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments) behind cached per-batch-size prepared statements; row order preserved, so rowid-based resolution determinism (#1015) is unchanged. - storeFileBundle: one transaction per file instead of four; nested transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no caller depended on nested rollback). - Dedicated store-writer thread for the fresh-DB bulk path (bundles applied in file order on a single writer connection; main thread does no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1. - Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load, rebuild once at the end; crash inside the window self-heals on the next open. - Per-context memos for resolveImportPath/findExportedSymbol + a per-file exported-symbol index, invalidated exactly where clearCaches() already resets the resolver's own caches. - Fast-init on completely fresh DBs (journal in memory, no fsync until the index completes; interrupted init re-runs from scratch). Kill switch: CODEGRAPH_NO_FAST_INIT=1. - MaybeYield returns undefined on the not-due path so per-ref yield checks stop paying a promise + microtask hop each. - Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and worker entry points. Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series). Graph dumps byte-identical across init, re-index, and sync paths; full suite green (2403 passed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(resolution): parallel reference resolution with canonical admission Fan resolution batches across a pool of read-only worker threads, each hosting a full ReferenceResolver over its own SQLite connection; results are admitted on the main thread in chunk order, so edge insertion order, row cleanup, failure parking, and deferred post-pass queues are exactly the sequence the single-threaded loop produces. Per-ref inputs match the baseline because the sequential path already resolves each batch against the state committed BEFORE that batch. Validated byte-identical on excalidraw (pool forced on) and apache/dubbo (4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical graph dumps (91,495 nodes / 223,953 edges). The pool only engages when total pending refs clear a threshold (default 150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1 to disable): measured on a ~58k-ref repo the workers' boot CPU contends with resolution on the same cores and makes indexing slower, so small repos keep the sequential path. When fast-init left the DB in memory-journal mode, WAL is restored before resolution only when the pool will run (readers + rollback-journal writers don't mix). Also: sqlite adapter readOnly open support. TreeCursor spine rewrite of the body walker was built, measured neutral on real repos and equal in a 20k-child microbench (web-tree-sitter's namedChild(i) is not quadratic in this binding), and rejected — per-node JS<->WASM marshaling is the floor, which a traversal swap cannot remove. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
246aee8373
commit
5736e24bb6
@@ -0,0 +1,79 @@
|
||||
/**
|
||||
* Resolver worker — one member of the parallel-resolution pool.
|
||||
*
|
||||
* Opens the project database READ-ONLY on its own connection and hosts a full
|
||||
* ReferenceResolver over it. The main thread partitions each resolution batch
|
||||
* into ordered chunks, fans them across the pool, and ADMITS the results
|
||||
* sequentially in chunk order — so edge insertion order (and every cleanup /
|
||||
* parking side effect) is identical to the single-threaded loop. Workers only
|
||||
* ever read; all writes stay on the main thread.
|
||||
*
|
||||
* Visibility note: the sequential baseline resolves every ref of a batch
|
||||
* against the DB state committed BEFORE that batch (edges persist after the
|
||||
* whole batch resolves). Workers read exactly that same committed state, so
|
||||
* per-ref inputs match the baseline ref-for-ref.
|
||||
*/
|
||||
|
||||
// Compile cache FIRST — same worker-boot rationale as parse-worker.ts.
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||
(require('node:module') as { enableCompileCache?: () => void }).enableCompileCache?.();
|
||||
} catch { /* cache is best-effort */ }
|
||||
|
||||
import { parentPort } from 'worker_threads';
|
||||
import { createDatabase, SqliteDatabase } from '../db/sqlite-adapter';
|
||||
import { QueryBuilder } from '../db/queries';
|
||||
import { ReferenceResolver } from './index';
|
||||
import type { UnresolvedReference } from '../types';
|
||||
|
||||
if (!parentPort) {
|
||||
throw new Error('resolver-worker must be run as a worker thread');
|
||||
}
|
||||
const port = parentPort;
|
||||
|
||||
let db: SqliteDatabase | null = null;
|
||||
let resolver: ReferenceResolver | null = null;
|
||||
|
||||
type InMessage =
|
||||
| { type: 'open'; dbPath: string; projectRoot: string }
|
||||
| { type: 'resolve'; id: number; refs: UnresolvedReference[] }
|
||||
| { type: 'close' };
|
||||
|
||||
port.on('message', (msg: InMessage) => {
|
||||
try {
|
||||
switch (msg.type) {
|
||||
case 'open': {
|
||||
const created = createDatabase(msg.dbPath, { readOnly: true });
|
||||
db = created.db;
|
||||
db.pragma('busy_timeout = 5000');
|
||||
db.pragma('cache_size = -32000');
|
||||
const queries = new QueryBuilder(db);
|
||||
resolver = new ReferenceResolver(msg.projectRoot, queries);
|
||||
resolver.initialize();
|
||||
port.postMessage({ type: 'ready' });
|
||||
break;
|
||||
}
|
||||
case 'resolve': {
|
||||
if (!resolver) throw new Error('resolver-worker: resolve before open');
|
||||
const out = resolver.resolveListForAdmission(msg.refs);
|
||||
port.postMessage({ type: 'result', id: msg.id, ...out });
|
||||
break;
|
||||
}
|
||||
case 'close': {
|
||||
try {
|
||||
db?.close();
|
||||
} catch {
|
||||
/* already closed */
|
||||
}
|
||||
process.exit(0);
|
||||
break;
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
port.postMessage({
|
||||
type: 'error',
|
||||
id: (msg as { id?: number }).id,
|
||||
message: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user