* perf(index): ~34% faster fresh indexing, byte-identical graphs Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed the main thread as the critical path: per-row SQLite statement calls, repeated import-resolution walks, and per-row FTS trigger firings, with the parse workers ~75% idle behind it. This lands the semantics-preserving tranche of fixes: - Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments) behind cached per-batch-size prepared statements; row order preserved, so rowid-based resolution determinism (#1015) is unchanged. - storeFileBundle: one transaction per file instead of four; nested transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no caller depended on nested rollback). - Dedicated store-writer thread for the fresh-DB bulk path (bundles applied in file order on a single writer connection; main thread does no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1. - Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load, rebuild once at the end; crash inside the window self-heals on the next open. - Per-context memos for resolveImportPath/findExportedSymbol + a per-file exported-symbol index, invalidated exactly where clearCaches() already resets the resolver's own caches. - Fast-init on completely fresh DBs (journal in memory, no fsync until the index completes; interrupted init re-runs from scratch). Kill switch: CODEGRAPH_NO_FAST_INIT=1. - MaybeYield returns undefined on the not-due path so per-ref yield checks stop paying a promise + microtask hop each. - Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and worker entry points. Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series). Graph dumps byte-identical across init, re-index, and sync paths; full suite green (2403 passed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(resolution): parallel reference resolution with canonical admission Fan resolution batches across a pool of read-only worker threads, each hosting a full ReferenceResolver over its own SQLite connection; results are admitted on the main thread in chunk order, so edge insertion order, row cleanup, failure parking, and deferred post-pass queues are exactly the sequence the single-threaded loop produces. Per-ref inputs match the baseline because the sequential path already resolves each batch against the state committed BEFORE that batch. Validated byte-identical on excalidraw (pool forced on) and apache/dubbo (4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical graph dumps (91,495 nodes / 223,953 edges). The pool only engages when total pending refs clear a threshold (default 150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1 to disable): measured on a ~58k-ref repo the workers' boot CPU contends with resolution on the same cores and makes indexing slower, so small repos keep the sequential path. When fast-init left the DB in memory-journal mode, WAL is restored before resolution only when the pool will run (readers + rollback-journal writers don't mix). Also: sqlite adapter readOnly open support. TreeCursor spine rewrite of the body walker was built, measured neutral on real repos and equal in a 20k-child microbench (web-tree-sitter's namedChild(i) is not quadratic in this binding), and rejected — per-node JS<->WASM marshaling is the floor, which a traversal swap cannot remove. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
54 lines
2.5 KiB
TypeScript
54 lines
2.5 KiB
TypeScript
/**
|
|
* Cooperative yielding for long synchronous resolution spans.
|
|
*
|
|
* Reference resolution and callback-edge synthesis run on the indexer's MAIN
|
|
* thread — unlike parsing, which is off-thread in the parse worker. The #850
|
|
* liveness watchdog (armed on `index`/`init` since #999) SIGKILLs the process
|
|
* when that thread doesn't turn its event loop for the timeout window (default
|
|
* 60s), because its heartbeat is a `setInterval` on that same thread. On a large
|
|
* repo, resolving refs + synthesizing dynamic-dispatch edges legitimately runs
|
|
* for minutes, so a span that never yields starves the heartbeat and the
|
|
* watchdog kills a VALID, in-progress index — the exact symptom of #1091 (the
|
|
* progress bar freezes at wherever it last rendered — 88% / 100% — then the
|
|
* process is killed).
|
|
*
|
|
* `createYielder` returns a `maybeYield()` that yields (via `setImmediate`) only
|
|
* once more than `budgetMs` of wall-clock has elapsed since the last yield, so
|
|
* fast repos pay essentially nothing while slow ones give the heartbeat a
|
|
* regular window to fire. Call it at natural boundaries in a long loop (between
|
|
* batches, between synthesis passes).
|
|
*
|
|
* This does NOT weaken the watchdog. A genuinely wedged loop — an infinite or
|
|
* non-terminating span, the case the watchdog exists to catch — never reaches a
|
|
* yield point, so the heartbeat still stops and the SIGKILL still fires. We only
|
|
* stop killing work that is demonstrably making progress.
|
|
*/
|
|
|
|
/**
|
|
* Yield when more than `budgetMs` of wall-clock has passed since the last
|
|
* yield. Returns `undefined` on the (overwhelmingly common) not-due path so a
|
|
* hot loop can skip the await entirely — `await`ing an async no-op costs a
|
|
* promise allocation + microtask hop, which at hundreds of thousands of calls
|
|
* per index is real time. Callers may either `await maybeYield()` (works for
|
|
* both return shapes) or use the fast form:
|
|
* `const y = maybeYield(); if (y) await y;`
|
|
*/
|
|
export type MaybeYield = () => Promise<void> | undefined;
|
|
|
|
/** Default budget: well under the watchdog's minimum heartbeat cadence (~1s), so
|
|
* a heartbeat byte always has a chance to land between yields. */
|
|
export const DEFAULT_YIELD_BUDGET_MS = 250;
|
|
|
|
export function createYielder(budgetMs: number = DEFAULT_YIELD_BUDGET_MS): MaybeYield {
|
|
let last = Date.now();
|
|
return function maybeYield(): Promise<void> | undefined {
|
|
if (Date.now() - last < budgetMs) return undefined;
|
|
return new Promise<void>((resolve) =>
|
|
setImmediate(() => {
|
|
last = Date.now();
|
|
resolve();
|
|
})
|
|
);
|
|
};
|
|
}
|