perf(store): resolution ref-index window — kernel-scale resolution 423→276s, 8c envelope ≈11min (§4d round 2) (#1369)

Store-architecture arc round 2. The batched resolution loop reads
unresolved_refs ONLY through the status index + the PK keyset pager;
the other five ref indexes (from_node, name, file_path, from_name,
failed_tail) serve sync-time paths — yet every per-batch DELETE of
resolved refs maintained all of them, the biggest single main-thread
stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase)
and 50-81s at kernel scale.

beginBulkRefLoad/endBulkRefLoad on DatabaseConnection, threaded as
refIndexLoad hooks next to the existing bulkEdgeLoad pair with the
same minRefsForPool gate (small syncs never pay): drop the five for
the loop, rebuild each in one scan at the end — where the table holds
only the surviving FAILED refs (resolved rows are deleted by then),
so the recreate is near-free. Crash inside the window heals on the
next open (schema.sql re-applies CREATE INDEX IF NOT EXISTS).

Measured:
- dubbo: deletes 1.2 → 0.2s, marks 0.6 → 0.3s, recreate 219ms; wall
  ~8.5s flat — the freed main-lane time shifts into settle (the worker
  lane now binds the double-buffer at medium scale).
- Linux kernel 8c: resolution 423.4 → 275.9s (deletes 50-81 → 3.2s,
  backpressure 16.8 → 7.4s — fewer index writes mean less WAL and
  cheaper folds), ref recreate 10.3s. Envelope ≈ 11.0min, from the
  14.8min pre-arc best; <10min-on-8c now needs ~1 more minute.

Gates: dubbo/gson dumps byte-identical; linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-19 23:24:14 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent f6d8e8fdab
commit ce0ae30e09
5 changed files with 92 additions and 1 deletions
+1 -1
View File
@@ -22,7 +22,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify <file> -R colbymchenry/codegraph`.
- Indexing inside CPU- or memory-limited containers (Docker, CI runners) now sizes its worker pools from the container's actual allowance instead of the host machine's, and giant codebases no longer balloon temporary database files during indexing (previously tens of GB of transient disk on Linux-kernel-scale projects). Together these prevent out-of-memory and out-of-disk failures on constrained machines; set `CODEGRAPH_RESOLVE_WORKERS` to override the resolution worker count explicitly.
- Indexing very large projects on multi-core machines got faster again: the parallel-resolution workers now periodically refresh their read-only database connections, which lets database housekeeping advance instead of silently building up a backlog behind long-lived readers — a backlog that was taxing the indexer's own writes. Graphs remain byte-for-byte identical; the win is largest at Linux-kernel scale on many-core machines.
- Fresh indexing got a sizeable across-the-board speedup: during the initial build, the database's secondary lookup indexes are set aside and rebuilt once after parsing instead of being maintained row by row — the same proven trick the later linking phase already used, now applied to the whole parse lane. A medium Java project's parse phase runs about 58% faster and its full fresh index about 19% faster end-to-end; on Linux-kernel-scale projects the whole index drops several percent further because the rebuilt indexes are more compact and every later phase reads them faster. Graphs remain byte-for-byte identical, and incremental syncs are unaffected.
- Fresh indexing got a sizeable across-the-board speedup: during the initial build, the database's secondary lookup indexes are set aside and rebuilt once after parsing instead of being maintained row by row — the same proven trick the later linking phase already used, now applied to the whole parse lane — and the reference-resolution loop likewise stops maintaining lookup indexes it never reads, rebuilding them at the end when almost nothing is left in the table. A medium Java project's parse phase runs about 58% faster and its full fresh index about 19% faster end-to-end; a Linux-kernel-scale index that took ~15 minutes on an 8-core machine now completes in about 11, with the resolution phase alone dropping by a third. Graphs remain byte-for-byte identical, and incremental syncs are unaffected.
- Indexing large C and C++ codebases spends much less time in the function-pointer dispatch analysis (the pass that connects handler tables like a command table or an ops struct to their call sites): each source file is now read and prepared once instead of four times, files that can't contribute any dispatch wiring are skipped outright in the later linking steps, and on platforms with the native engine the per-file scanning itself now runs natively too. On a Linux-kernel-scale tree the pass runs about a third faster end-to-end, with graphs byte-for-byte identical; platforms without a native binary keep the same results on the previous path.
### Fixes