perf(store): resolution ref-index window — kernel-scale resolution 423→276s, 8c envelope ≈11min (§4d round 2) (#1369)

Store-architecture arc round 2. The batched resolution loop reads
unresolved_refs ONLY through the status index + the PK keyset pager;
the other five ref indexes (from_node, name, file_path, from_name,
failed_tail) serve sync-time paths — yet every per-batch DELETE of
resolved refs maintained all of them, the biggest single main-thread
stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase)
and 50-81s at kernel scale.

beginBulkRefLoad/endBulkRefLoad on DatabaseConnection, threaded as
refIndexLoad hooks next to the existing bulkEdgeLoad pair with the
same minRefsForPool gate (small syncs never pay): drop the five for
the loop, rebuild each in one scan at the end — where the table holds
only the surviving FAILED refs (resolved rows are deleted by then),
so the recreate is near-free. Crash inside the window heals on the
next open (schema.sql re-applies CREATE INDEX IF NOT EXISTS).

Measured:
- dubbo: deletes 1.2 → 0.2s, marks 0.6 → 0.3s, recreate 219ms; wall
  ~8.5s flat — the freed main-lane time shifts into settle (the worker
  lane now binds the double-buffer at medium scale).
- Linux kernel 8c: resolution 423.4 → 275.9s (deletes 50-81 → 3.2s,
  backpressure 16.8 → 7.4s — fewer index writes mean less WAL and
  cheaper folds), ref recreate 10.3s. Envelope ≈ 11.0min, from the
  14.8min pre-arc best; <10min-on-8c now needs ~1 more minute.

Gates: dubbo/gson dumps byte-identical; linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-19 23:24:14 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent f6d8e8fdab
commit ce0ae30e09
5 changed files with 92 additions and 1 deletions
+24
View File
@@ -543,6 +543,30 @@ gap. Multi-file write transactions are likely ~zero on the fastInit path
batching); buffer→bind remains a CPU-side option if the writer re-emerges as
the wall.
**Store-arc round 2 SHIPPED (2026-07-19): resolution ref-index window.** The
batched resolution loop reads unresolved_refs ONLY through the status index
+ PK keyset pager; the other five ref indexes (from_node, name, file_path,
from_name, failed_tail) serve sync-time paths — yet every per-batch DELETE
maintained all of them. `beginBulkRefLoad`/`endBulkRefLoad` (same
`minRefsForPool` gate as the edge window) drop the five for the loop and
rebuild at the end, where the table holds only the surviving failed refs
(resolved rows are deleted by then), making the recreate near-free.
- **dubbo**: deletes 1.2 → 0.2s, marks 0.6 → 0.3s (recreate 219ms) — but the
wall stayed ~8.5s: the freed main-lane time moved into `settle` (the
worker lane now binds the double-buffer). The honest read: at medium
scale, resolution's floor is now the WORKER lane + pool spin-up, not the
writer.
- **Linux kernel 8c: resolution 423.4 → 275.9s** — deletes 50-81 → **3.2s**,
backpressure 16.8 → 7.4s (fewer index writes → less WAL → cheaper folds,
compounding), ref recreate 10.3s, edge recreate 26.8s. Synthesis 149.0s.
**Envelope ≈ 11.0min** (phase sum 659.7s) — from 14.8min best-ever before
this arc's rounds. **The <10min-on-8c target needs ~1 more minute**; the
remaining mass is parse-loop 190s (extraction-bound on linux — R7b's
wasm-deferred C tail + the store lane) and synthesis 149s.
- Gates: dubbo/gson dumps byte-identical, linux counts exact + dump sha
`6dd1185b…` reproduced, suite green ×2.
## 4. Per-language tracker
Tiers: **T1** = mostly `.scm` + mapping config. **T2** = needs bespoke pre/post passes kept