From cf38ef65afe942f5d8e1122b92b40017f0b9e67c Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 16 Jul 2026 18:07:51 -0500 Subject: [PATCH] perf(synthesis): fan dynamic-dispatch passes across the resolver pool, byte-identical graphs (#1321) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ~36 independent synthesis passes (callback/event/framework wiring) ran sequentially on the indexer's main thread — 2.0s of a 4,402-file Java repo's index, and the stage where kernel-class repos die (#1212). They now live in an explicit registry (SYNTH_PASSES) and, when the resolver pool is alive (>=150k-ref repos), fan out across its read-only workers: dubbo synthesis 2,024ms -> ~900ms (-55%), total fresh init 13.5s -> 11.9s. Graphs verified byte-for-byte identical on both the pool path (dubbo) and the sequential path (excalidraw). Why this is safe: no pass's edges persist until the ordered merge, so every pass sees the same committed post-resolution DB state in either mode, and results merge in registry order regardless of completion order — the first-seen dedup is unchanged. The pool now survives through synthesis (destroy moved after it) instead of being torn down moments before the one stage that could reuse it. Robustness: a pass that fails on a worker (crash, OOM) is retried on the main thread — a synthesizer blow-up now costs one worker instead of the whole index, which is half the #1212 story on very large repos. Also: ref-row cleanup deletes now run as one transaction with a cached statement instead of one implicit commit per 500-row chunk (mechanically fewer WAL commits; matters most on HDD-class storage). A set-based rewrite of failed-ref parking was tried, measured ~zero on NVMe, and dropped — the remaining persist cost is edge-index B-tree maintenance, not statement dispatch. SYNTH_PROGRESS_STEPS now derives from the registry (passes + fixed marks); the pin test counts registry entries plus literal __mark sites. Suite green (2444). Sequential-path timing unchanged on excalidraw. Co-authored-by: Claude Fable 5 --- CHANGELOG.md | 1 + __tests__/synthesis-progress.test.ts | 15 +- src/db/queries.ts | 28 ++- src/resolution/callback-synthesizer.ts | 230 ++++++++++++++++--------- src/resolution/index.ts | 22 ++- src/resolution/resolver-pool.ts | 39 ++++- src/resolution/resolver-worker.ts | 32 +++- 7 files changed, 268 insertions(+), 99 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 378a101..3bcbabf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=` to tune when it engages. - Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle. +- The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical. - Indexing is significantly faster — a fresh `codegraph init` on a medium TypeScript project takes about a third less wall-clock time, with the same graph produced byte-for-byte. The gains come from batching database writes, storing files on a dedicated writer thread, memoizing repeated import-resolution lookups, skipping per-row search-index maintenance during the bulk build (rebuilt once at the end), and — on completely fresh databases only — deferring disk durability until the index completes, since an interrupted first index is simply re-run. Set `CODEGRAPH_NO_FAST_INIT=1` to keep full crash-durability during the initial build, or `CODEGRAPH_NO_STORE_WORKER=1` to store on the main thread. - `codegraph install` and `codegraph upgrade` now offer CodeGraph Pro beta access after finishing — answer yes, type your email, and you join the same waitlist as the getcodegraph.com homepage form. Strictly opt-in and asked at most once per machine total: nothing is sent unless you say yes and enter an email, either answer is remembered so no later install or upgrade ever re-asks, and non-interactive runs (`--yes`, scripts, CI) never see the question. - Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify -R colbymchenry/codegraph`. diff --git a/__tests__/synthesis-progress.test.ts b/__tests__/synthesis-progress.test.ts index f747880..8d90223 100644 --- a/__tests__/synthesis-progress.test.ts +++ b/__tests__/synthesis-progress.test.ts @@ -14,19 +14,22 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import { CodeGraph, IndexProgress } from '../src/index'; -import { SYNTH_PROGRESS_STEPS } from '../src/resolution/callback-synthesizer'; +import { SYNTH_PASSES, SYNTH_PROGRESS_STEPS } from '../src/resolution/callback-synthesizer'; describe('synthesis progress ("Linking dynamic dispatch" phase)', () => { - it('SYNTH_PROGRESS_STEPS matches the synthesizer’s actual __mark() step count', () => { + it('SYNTH_PROGRESS_STEPS matches the synthesizer’s actual step count', () => { // The constant is cosmetic (progress denominator), but drift makes the bar - // end early or jump to 100% — adding a pass must bump it. Every step site - // calls __mark('