perf(resolution): drop non-unique edge indexes during the bulk resolution window, byte-identical graphs (#1322)
The resolution persist's measured cost is B-tree maintenance on the edges table's five indexes (offline replay of a 224k-edge resolution set: 2.8s with all indexes, 1.1s with only the unique identity index, +0.3s to recreate the rest). On big runs (same >=150k-ref gate as the resolver pool) the four non-unique edge indexes are now dropped for the batch loop and recreated in one pass each before synthesis. Why this is safe: - idx_edges_identity stays: INSERT OR IGNORE's dedup conflicts on it (#1034), and its leftmost column is `source`, so the only mid-window edge reads — resolution's supertype walks (implements/extends by source) — keep an index via its prefix (verified with EXPLAIN QUERY PLAN). - The window closes BEFORE synthesis, whose passes read kind-keyed, and on every error path (finally). - A crash inside the window heals on the next DatabaseConnection open — schema.sql re-applies CREATE INDEX IF NOT EXISTS, same recovery as the FTS bulk-load pattern this mirrors. - Concurrent readers (a daemon serving the project mid-index) stay correct; target/kind-keyed reads degrade to scans only for the window's duration. dubbo (4,402 files): persists 4.0s -> 3.0s, fresh init 11.9s -> ~11.1s, graph byte-identical. excalidraw (below the gate): untouched, byte-identical. Recreation cost ~250ms, logged under CODEGRAPH_SYNTH_TIMINGS as edge-index-recreate. Suite green (2444). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
cf38ef65af
commit
567b4ad4be
@@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
- Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=<count>` to tune when it engages.
|
||||
- Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle.
|
||||
- The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical.
|
||||
- On large projects, indexing writes its relationship data noticeably faster: secondary database indexes are set aside during the bulk of reference resolution and rebuilt once at the end, instead of being maintained row by row. Graphs remain byte-for-byte identical, and small projects are unaffected.
|
||||
- Indexing is significantly faster — a fresh `codegraph init` on a medium TypeScript project takes about a third less wall-clock time, with the same graph produced byte-for-byte. The gains come from batching database writes, storing files on a dedicated writer thread, memoizing repeated import-resolution lookups, skipping per-row search-index maintenance during the bulk build (rebuilt once at the end), and — on completely fresh databases only — deferring disk durability until the index completes, since an interrupted first index is simply re-run. Set `CODEGRAPH_NO_FAST_INIT=1` to keep full crash-durability during the initial build, or `CODEGRAPH_NO_STORE_WORKER=1` to store on the main thread.
|
||||
- `codegraph install` and `codegraph upgrade` now offer CodeGraph Pro beta access after finishing — answer yes, type your email, and you join the same waitlist as the getcodegraph.com homepage form. Strictly opt-in and asked at most once per machine total: nothing is sent unless you say yes and enter an email, either answer is remembered so no later install or upgrade ever re-asks, and non-interactive runs (`--yes`, scripts, CI) never see the question.
|
||||
- Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify <file> -R colbymchenry/codegraph`.
|
||||
|
||||
Reference in New Issue
Block a user