perf(synthesis): fan dynamic-dispatch passes across the resolver pool, byte-identical graphs (#1321)

The ~36 independent synthesis passes (callback/event/framework wiring) ran
sequentially on the indexer's main thread — 2.0s of a 4,402-file Java repo's
index, and the stage where kernel-class repos die (#1212). They now live in
an explicit registry (SYNTH_PASSES) and, when the resolver pool is alive
(>=150k-ref repos), fan out across its read-only workers: dubbo synthesis
2,024ms -> ~900ms (-55%), total fresh init 13.5s -> 11.9s. Graphs verified
byte-for-byte identical on both the pool path (dubbo) and the sequential
path (excalidraw).

Why this is safe: no pass's edges persist until the ordered merge, so every
pass sees the same committed post-resolution DB state in either mode, and
results merge in registry order regardless of completion order — the
first-seen dedup is unchanged. The pool now survives through synthesis
(destroy moved after it) instead of being torn down moments before the one
stage that could reuse it.

Robustness: a pass that fails on a worker (crash, OOM) is retried on the
main thread — a synthesizer blow-up now costs one worker instead of the
whole index, which is half the #1212 story on very large repos.

Also: ref-row cleanup deletes now run as one transaction with a cached
statement instead of one implicit commit per 500-row chunk (mechanically
fewer WAL commits; matters most on HDD-class storage). A set-based rewrite
of failed-ref parking was tried, measured ~zero on NVMe, and dropped — the
remaining persist cost is edge-index B-tree maintenance, not statement
dispatch.

SYNTH_PROGRESS_STEPS now derives from the registry (passes + fixed marks);
the pin test counts registry entries plus literal __mark sites.

Suite green (2444). Sequential-path timing unchanged on excalidraw.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-16 18:07:51 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent a2f3c31a97
commit cf38ef65af
7 changed files with 268 additions and 99 deletions
+23 -5
View File
@@ -230,6 +230,7 @@ export class QueryBuilder {
getNodesByLowerName?: SqliteStatement;
getUnresolvedCount?: SqliteStatement;
getUnresolvedBatch?: SqliteStatement;
deleteRefsByRowIdsFull?: SqliteStatement;
getAllFilePaths?: SqliteStatement;
getAllNodeNames?: SqliteStatement;
getDominantFile?: SqliteStatement;
@@ -2204,11 +2205,28 @@ export class QueryBuilder {
*/
deleteReferencesByRowIds(rowIds: number[]): void {
if (rowIds.length === 0) return;
for (let i = 0; i < rowIds.length; i += SQLITE_PARAM_CHUNK_SIZE) {
const chunk = rowIds.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
const placeholders = chunk.map(() => '?').join(',');
this.db.prepare(`DELETE FROM unresolved_refs WHERE id IN (${placeholders})`).run(...chunk);
}
// One transaction for all chunks (each chunk was previously its own
// implicit transaction = its own WAL commit — measurable on 100k+-ref
// resolution persists), and the full-size chunk statement is cached so
// repeat calls skip the re-prepare; only the final partial chunk (if any)
// prepares ad hoc.
this.db.transaction(() => {
for (let i = 0; i < rowIds.length; i += SQLITE_PARAM_CHUNK_SIZE) {
const chunk = rowIds.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
if (chunk.length === SQLITE_PARAM_CHUNK_SIZE) {
if (!this.stmts.deleteRefsByRowIdsFull) {
const placeholders = new Array(SQLITE_PARAM_CHUNK_SIZE).fill('?').join(',');
this.stmts.deleteRefsByRowIdsFull = this.db.prepare(
`DELETE FROM unresolved_refs WHERE id IN (${placeholders})`
);
}
this.stmts.deleteRefsByRowIdsFull.run(...chunk);
} else {
const placeholders = chunk.map(() => '?').join(',');
this.db.prepare(`DELETE FROM unresolved_refs WHERE id IN (${placeholders})`).run(...chunk);
}
}
})();
}
/**