From 5955d04c97de3311476b0ca9d8d84cfff4fcb0d3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Sat, 18 Jul 2026 02:31:47 -0500 Subject: [PATCH] =?UTF-8?q?docs(kernel):=20=C2=A77a.6=20per-ref=20measurem?= =?UTF-8?q?ent=20round=20=E2=80=94=20pool=20works,=20two=20cache=20theorie?= =?UTF-8?q?s=20killed,=20writes-under-readers=20named=20(#1354)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fresh CODEGRAPH_RESOLVE_PROFILE tables at 2c and 8c on the round-2 build: the 8c settle stage is 3.6s — the double-buffer absorbs the entire resolveOne population, superseding §7a.2's core-invariant framing. The 8c cost is writes-under-readers: main-thread deletes+insertEdges run ~3× slower (+102s) with 4 readonly workers attached — mechanism unproven, named as the next probe. Two same-day experiments killed by measurement and reverted: budget-scaled name caches (v1 LRU regressed via delete+set-per-get churn + GC; v2 mutation-free second-chance cache landed exactly on baseline — the 11µs exact-match is per-ref floor, not refetch overhead) and lazy candidates JSON (read stage flat). New 2c record 16.5min clean-host; 8c range 15.0–16.4min across n=2 — ranges, never single runs. Round-2 deferral cuts did move 8c parse (202.6→178.5s). Levers re-ranked: writes-under-readers probe > cFnPtr native site extraction (~230s of synthesis) > backpressure byte volume > recreate. Co-authored-by: Claude Fable 5 --- docs/design/rust-kernel-migration-plan.md | 80 +++++++++++++++++++++-- 1 file changed, 76 insertions(+), 4 deletions(-) diff --git a/docs/design/rust-kernel-migration-plan.md b/docs/design/rust-kernel-migration-plan.md index 0b65d40..367316e 100644 --- a/docs/design/rust-kernel-migration-plan.md +++ b/docs/design/rust-kernel-migration-plan.md @@ -85,10 +85,15 @@ them are the ORIGINAL plan and carry expectations that measurement later correct most SYMBOLS on deferred files; the real win is EDGES (+6,585), phantom cleanup, and native-path coverage. Remaining deferral is policy-skips (CONFIG interleaves, TP_PROTO DSL, module_init-no-semi) - + small buckets — this lever is largely SPENT. Queue now: **per-ref - resolution path** (the core-invariant superphase) > backpressure - ~120s (checkpoint I/O floor) > E-scan/settle/read-mapping (~70–90s - each, approaching honest work). + + small buckets — this lever is largely SPENT. Per-ref measurement + round DONE 2026-07-18 (§7a.6): fresh 2c/8c stage tables; the pool + double-buffer WORKS (8c settle 3.6s — "core-invariant" superseded); + 2c record now 16.5min, 8c range 15.0–16.4 (n=2). Two cache + experiments killed by measurement same-day (nameCache scaling, lazy + candidates — §7a.6 has the numbers; code reverted). Queue now: + **writes-under-readers probe** (8c deletes+inserts +102s under the + pool — the biggest attributed delta) > **cFnPtr native site + extraction** (synthesis ~230s) > backpressure byte volume > recreate. - [x] **R7a. C/C++ port** — DONE 2026-07-17, same-day walker+gates after the survey (#1344) and grammar vendoring (#1345). One dual-language walker (`codegraph-kernel/src/ccpp/`), preParse HOISTED to the route point @@ -778,6 +783,73 @@ core-invariant resolution superphase. Deferral cuts can't materially move the graph richness + the 2c/low-core envelope. The 8c target now lives or dies on the per-ref resolution path (§7a.2's lever (a)). +#### 7a.6 Per-ref path measurement round (2026-07-18) — fresh tables, two falsifications, two live levers + +Fresh `CODEGRAPH_RESOLVE_PROFILE` tables on the round-2 build (v7.2-rc2 tree, +cg1212), then two cache experiments run against them — both killed by +measurement, code reverted same-day; this section is what survives. + +| stage (batch loop) | 2c sequential (clean host) | 8c pool-4 | +|---|---|---| +| read | 37.0s | 33.9s | +| settle (resolveOne) | 79.7s | **3.6s** | +| backpressure | 138.3s | 121.9s | +| createEdges | 3.4s | 7.6s | +| insertEdges | 33.8s | **55.3s** | +| deletes | 37.7s | **118.8s** | +| marks | 5.3s | 6.5s | +| **loop total** | **339s** | **357s** | + +2c: superphase 645s (loop + synth 251.6s [cFnPtr ~230: E 95.0 + strip 78.5 +at n=283k, budget-declined again + C/D 88.8] + recreate 54.5); envelope +**16.5min — the new 2c record** (the 17.1 r2-gate figure carried host +contamination). Settle decomposition: exact-match 35.1s @ 11µs × 3.17M, +import 16.9s, fail:calls 9.2s × 1.63M. 8c: superphase 655.5s (+ synth 242.6 ++ recreate 56.1), envelope 14.95min — **n=2 range 15.0–16.4min with the +morning's run; report ranges, never single runs on this box.** 8c parse +178.5s: round 2's deferral cuts DID move the 8c parse wall (202.6 → 178.5) +— §7a.5's "writer-floor won't move" prediction was partly wrong. + +- **The pool double-buffer WORKS.** settle 3.6s at 8c — the workers absorb + the entire 3.17M exact-match population (12–17s per worker, parallel). + §7a.2's "resolution is core-invariant" framing is superseded: the 8c cost + was never resolveOne. +- **THE 8c anomaly — writes-under-readers:** deletes 37.7 → 118.8s (+81) + and insertEdges 33.8 → 55.3s (+22) with 4 readonly workers attached. + Main-thread B-tree writes run ~3× slower under the pool. Mechanism + UNPROVEN — candidates: page-cache competition (4 × 32MB worker caches + + reads), WAL read-through depth while readers hold positions, wal-index + lock contention. Next probe: instrument (per-op delete timing vs worker + activity windows), then either shorten reader hold-times (worker + connection recycling at the barrier — §7a.1's original fix direction, + never built) or cut delete volume. Potential ≈ −100s at 8c. +- **Killed by measurement #1 — nameCache scaling (the 5k-thrash theory).** + v1: budget-scaled classic LRU (~478k entries) → settle 102.7s, + exact-match 52.7s @ 17µs — WORSE; delete+set-per-get churn on a huge Map + plus resident-array GC ate more than the SQLite statements saved. v2: + mutation-free second-chance cache at 250k → exact-match 37.9s @ 12µs ≈ + the 35.1s baseline. Verdict: the 11µs is NOT refetch overhead — the 5k + cache already holds the true Zipf head, the tail doesn't repeat enough + to cache at any size, and the floor is the per-ref JS around one indexed + lookup. Both variants byte-correct (counts 2,049,153/6,413,518; git dumps + byte-identical) — correctness was never the issue. Code reverted; the + second-chance design lives in this entry if a big-RAM-validated attempt + ever wants it. +- **Killed by measurement #2 — lazy `candidates` JSON parse:** read stage + 37.0 → 38–40s across variants (flat). The eager parse was never the read + cost; row materialization + the statement walk is. Reverted. +- **Levers, re-ranked:** writes-under-readers probe (+102s at 8c — the + single biggest attributed delta) > cFnPtr NATIVE SITE EXTRACTION + (synthesis ~230s: emit fn-ptr assignment sites from the C walker at parse + time for the now-66% kernel-routed population — E-scan 95s + reads + much + of strip 78s die; needs bug-for-bug regex-semantics parity in Rust and + the raw-vs-preParsed scan-text question settled first) > backpressure + byte volume (~122–138s I/O floor; value-neutral schema interning is + migration-wide — parked) > recreate 54–70s. +- Box note: cg1212's 6–7GB deliberately degrades the cFnPtr strip cache + (~80s paid in-container that a 24GB target-class box gets back free) — + container numbers UNDERSTATE the true 8-core-class target. + ### 7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation) Priority order, each gated by the standard A/B + node-explosion probes: 1. **Test→subject edges** (first-class `tests` edges at index time; we compute covering