* perf(index): ~34% faster fresh indexing, byte-identical graphs Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed the main thread as the critical path: per-row SQLite statement calls, repeated import-resolution walks, and per-row FTS trigger firings, with the parse workers ~75% idle behind it. This lands the semantics-preserving tranche of fixes: - Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments) behind cached per-batch-size prepared statements; row order preserved, so rowid-based resolution determinism (#1015) is unchanged. - storeFileBundle: one transaction per file instead of four; nested transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no caller depended on nested rollback). - Dedicated store-writer thread for the fresh-DB bulk path (bundles applied in file order on a single writer connection; main thread does no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1. - Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load, rebuild once at the end; crash inside the window self-heals on the next open. - Per-context memos for resolveImportPath/findExportedSymbol + a per-file exported-symbol index, invalidated exactly where clearCaches() already resets the resolver's own caches. - Fast-init on completely fresh DBs (journal in memory, no fsync until the index completes; interrupted init re-runs from scratch). Kill switch: CODEGRAPH_NO_FAST_INIT=1. - MaybeYield returns undefined on the not-due path so per-ref yield checks stop paying a promise + microtask hop each. - Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and worker entry points. Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series). Graph dumps byte-identical across init, re-index, and sync paths; full suite green (2403 passed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(resolution): parallel reference resolution with canonical admission Fan resolution batches across a pool of read-only worker threads, each hosting a full ReferenceResolver over its own SQLite connection; results are admitted on the main thread in chunk order, so edge insertion order, row cleanup, failure parking, and deferred post-pass queues are exactly the sequence the single-threaded loop produces. Per-ref inputs match the baseline because the sequential path already resolves each batch against the state committed BEFORE that batch. Validated byte-identical on excalidraw (pool forced on) and apache/dubbo (4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical graph dumps (91,495 nodes / 223,953 edges). The pool only engages when total pending refs clear a threshold (default 150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1 to disable): measured on a ~58k-ref repo the workers' boot CPU contends with resolution on the same cores and makes indexing slower, so small repos keep the sequential path. When fast-init left the DB in memory-journal mode, WAL is restored before resolution only when the pool will run (readers + rollback-journal writers don't mix). Also: sqlite adapter readOnly open support. TreeCursor spine rewrite of the body walker was built, measured neutral on real repos and equal in a 20k-child microbench (web-tree-sitter's namedChild(i) is not quadratic in this binding), and rejected — per-node JS<->WASM marshaling is the floor, which a traversal swap cannot remove. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
167 lines
5.7 KiB
TypeScript
167 lines
5.7 KiB
TypeScript
/**
|
|
* SQLite Adapter
|
|
*
|
|
* Thin wrapper over Node's built-in `node:sqlite` (`DatabaseSync`), exposed
|
|
* through a small better-sqlite3-shaped interface so the rest of the codebase
|
|
* is storage-agnostic.
|
|
*
|
|
* CodeGraph ships with a bundled Node runtime, so `node:sqlite` (real SQLite,
|
|
* with WAL + FTS5) is always available — there is no native build step and no
|
|
* wasm fallback. When run from source instead, it requires Node >= 22.5.
|
|
*/
|
|
|
|
export interface SqliteStatement {
|
|
run(...params: any[]): { changes: number; lastInsertRowid: number | bigint };
|
|
get(...params: any[]): any;
|
|
all(...params: any[]): any[];
|
|
/**
|
|
* Lazily yield result rows one at a time instead of materializing the whole
|
|
* set with `all()`. Use for unbounded scans (e.g. every function/method node)
|
|
* so memory stays O(1) in the row count rather than O(rows) — see #610, where
|
|
* `all()`-ing every symbol on a dense project spiked the heap into an OOM.
|
|
*/
|
|
iterate(...params: any[]): IterableIterator<any>;
|
|
}
|
|
|
|
export interface SqliteDatabase {
|
|
prepare(sql: string): SqliteStatement;
|
|
exec(sql: string): void;
|
|
pragma(str: string, options?: { simple?: boolean }): any;
|
|
transaction<T>(fn: (...args: any[]) => T): (...args: any[]) => T;
|
|
close(): void;
|
|
readonly open: boolean;
|
|
}
|
|
|
|
/**
|
|
* The active SQLite backend. Only one now (`node:sqlite`); kept as a named type
|
|
* so `codegraph status` and the per-instance reporting have a stable shape.
|
|
*/
|
|
export type SqliteBackend = 'node-sqlite';
|
|
|
|
/**
|
|
* Wraps Node's built-in `node:sqlite` (`DatabaseSync`) to match the
|
|
* better-sqlite3 interface the rest of the code expects.
|
|
*
|
|
* node:sqlite is real SQLite compiled into Node, so it supports WAL, FTS5,
|
|
* mmap, and `@named` params natively — the only shims needed are the
|
|
* better-sqlite3 conveniences node:sqlite omits: a `.pragma()` helper, a
|
|
* `.transaction()` helper, and `open` (node:sqlite exposes `isOpen`).
|
|
*/
|
|
class NodeSqliteAdapter implements SqliteDatabase {
|
|
private _db: any;
|
|
private _txDepth = 0;
|
|
|
|
constructor(dbPath: string, opts?: { readOnly?: boolean }) {
|
|
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
|
const { DatabaseSync } = require('node:sqlite');
|
|
this._db = opts?.readOnly ? new DatabaseSync(dbPath, { readOnly: true }) : new DatabaseSync(dbPath);
|
|
}
|
|
|
|
get open(): boolean {
|
|
return this._db.isOpen;
|
|
}
|
|
|
|
prepare(sql: string): SqliteStatement {
|
|
// node:sqlite matches better-sqlite3's calling convention (variadic
|
|
// positional args, or a single object for @named params), so params forward
|
|
// through unchanged.
|
|
const stmt = this._db.prepare(sql);
|
|
return {
|
|
run(...params: any[]) {
|
|
const r = stmt.run(...params);
|
|
return {
|
|
changes: Number(r?.changes ?? 0),
|
|
lastInsertRowid: r?.lastInsertRowid ?? 0,
|
|
};
|
|
},
|
|
get(...params: any[]) {
|
|
return stmt.get(...params);
|
|
},
|
|
all(...params: any[]) {
|
|
return stmt.all(...params);
|
|
},
|
|
iterate(...params: any[]) {
|
|
return stmt.iterate(...params);
|
|
},
|
|
};
|
|
}
|
|
|
|
exec(sql: string): void {
|
|
this._db.exec(sql);
|
|
}
|
|
|
|
pragma(str: string, options?: { simple?: boolean }): any {
|
|
const trimmed = str.trim();
|
|
// Write pragma ("key = value"): node:sqlite is real SQLite, so every pragma
|
|
// (WAL, mmap, synchronous, …) applies as-is.
|
|
if (trimmed.includes('=')) {
|
|
this._db.exec(`PRAGMA ${trimmed}`);
|
|
return;
|
|
}
|
|
// Read pragma. Default: the row object (e.g. { journal_mode: 'wal' }).
|
|
// `{ simple: true }` returns just the single column value, like better-sqlite3.
|
|
const row = this._db.prepare(`PRAGMA ${trimmed}`).get();
|
|
if (options?.simple) {
|
|
return row && typeof row === 'object' ? Object.values(row)[0] : row;
|
|
}
|
|
return row;
|
|
}
|
|
|
|
transaction<T>(fn: (...args: any[]) => T): (...args: any[]) => T {
|
|
return (...args: any[]) => {
|
|
// Nested call (a transaction()-wrapped helper invoked from inside another
|
|
// transaction): run the body directly inside the enclosing transaction.
|
|
// BEGIN would throw "cannot start a transaction within a transaction",
|
|
// so no existing caller ever relied on nested rollback granularity —
|
|
// flattening is behavior-preserving and free.
|
|
if (this._txDepth > 0) {
|
|
this._txDepth++;
|
|
try {
|
|
return fn(...args);
|
|
} finally {
|
|
this._txDepth--;
|
|
}
|
|
}
|
|
this._db.exec('BEGIN');
|
|
this._txDepth = 1;
|
|
try {
|
|
const result = fn(...args);
|
|
this._db.exec('COMMIT');
|
|
this._txDepth = 0;
|
|
return result;
|
|
} catch (error) {
|
|
this._db.exec('ROLLBACK');
|
|
this._txDepth = 0;
|
|
throw error;
|
|
}
|
|
};
|
|
}
|
|
|
|
close(): void {
|
|
// node:sqlite's DatabaseSync.close() throws if already closed; make it
|
|
// idempotent to match better-sqlite3 (callers may close more than once).
|
|
if (this._db.isOpen) this._db.close();
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Create a database connection backed by `node:sqlite`.
|
|
*
|
|
* Returns the active backend alongside the db so each `DatabaseConnection` can
|
|
* report it per-instance — MCP can open multiple project DBs in one process, so
|
|
* a process-global would race.
|
|
*/
|
|
export function createDatabase(dbPath: string, opts?: { readOnly?: boolean }): { db: SqliteDatabase; backend: SqliteBackend } {
|
|
try {
|
|
return { db: new NodeSqliteAdapter(dbPath, opts), backend: 'node-sqlite' };
|
|
} catch (error) {
|
|
const msg = error instanceof Error ? error.message : String(error);
|
|
throw new Error(
|
|
'Failed to open SQLite via the built-in node:sqlite module.\n' +
|
|
'CodeGraph requires node:sqlite (Node.js 22.5+). Install the self-contained\n' +
|
|
'CodeGraph release (it bundles a compatible Node), or run on Node 22.5+.\n' +
|
|
`Underlying error: ${msg}`
|
|
);
|
|
}
|
|
}
|