perf(resolution): batch-loop de-quadratic — keyset reads, changes-based guard, DB-scaled valve caps + resolve profiler (#1339)

The §7a.2 per-ref profile overturned the assumption the whole arc was
built on: resolveOne owns only ~93s of the kernel-scale ~433s batch loop.
Loop-stage attribution (CODEGRAPH_RESOLVE_PROFILE, shipped here) named the
rest: backpressure folds 111.2s, count guard 93.9s, batch reads 54.6s,
deletes/inserts/marks ~84s, settle 85.7s.

- Non-progress guard O(remaining)→O(1): the per-batch COUNT(*) walked every
  remaining pending row (O(N²/batch) per run, 93.9s). The cleanup queries
  now return summed SQLite , and zero-removals-from-claimed-work
  is the guard signal — the DIRECT evidence the count diff inferred (a
  mismatched-name resolver makes keyed cleanup no-op ⇒ changes=0). A real
  COUNT runs only on that suspicious path and arbitrates exactly as before.
- Batch reads OFFSET→keyset (54.6s→O(batch)): OFFSET re-walked the
  accumulated failed-row prefix every read; seeking past the last-seen
  rowid is prefix-independent and enumeration-order identical.
- WAL valve caps scale with DB size (env still wins): every fold re-writes
  hot pages (#1231 in bounded form — 111.2s at the flat 256MB cap);
  soft=clamp(dbSize/4, 256MB, 2GB) trades ~4× fewer folds for a transient
  WAL ≈ project size.
- CODEGRAPH_RESOLVE_PROFILE: per-outcome resolveOne histogram + loop-stage
  attribution, main + workers, off by default.

Gates: dubbo dump byte-identical; suite 2,491 passed / 4 skipped (kernel
required). Kernel-scale payoff run lands in the plan doc next.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-17 11:27:30 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 19cf1ec75b
commit 7cc23668b5
7 changed files with 194 additions and 33 deletions
+9 -1
View File
@@ -61,11 +61,19 @@ const CHECK_INTERVAL_MS = 2000;
* Resolve the valve's soft threshold from the `CODEGRAPH_WAL_VALVE_MB`
* override; non-numeric / non-positive values fall back to the default.
*/
export function resolveWalValveMb(envVal: string | undefined): number {
export function resolveWalValveMb(envVal: string | undefined, dbSizeBytes?: number): number {
if (envVal !== undefined && envVal !== '') {
const n = Number(envVal);
if (Number.isFinite(n) && n > 0) return Math.floor(n);
}
// Scale with the project when the caller knows the DB size: every fold
// re-writes hot B-tree pages into the main file (the #1231 pathology in
// bounded form — 111s of a kernel-scale batch loop at the flat 256MB cap,
// §7a.2), so a big project affords a proportionally bigger transient WAL
// (~dbSize/4 soft ⇒ file cap ≈ dbSize) in exchange for ~4× fewer folds.
if (dbSizeBytes !== undefined && dbSizeBytes > 0) {
return Math.min(2048, Math.max(DEFAULT_WAL_VALVE_MB, Math.floor(dbSizeBytes / 4 / (1024 * 1024))));
}
return DEFAULT_WAL_VALVE_MB;
}