github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf(kernel): direct-to-store decode — buffers flow to the store worker, main thread never materializes nodes

Browse Source 
Kernel-routed files ship their flat tables from the parse worker to the
store worker as buffers (tryKernelExtractRaw → kernelBuffers on the
result → KernelStoreBundle); the store worker decodes and finalizes
(finalizeStoreBundle shared with the object path so filter semantics
can never drift). Files with applicable framework extract() hooks keep
the decoded path; the no-writer fallback materializes via
materializeKernelResult. Byte-identical dumps re-verified on dubbo,
excalidraw, express, gson; full suite green (2,467).

Measurement (plan §4d): dubbo's parse-loop wall is 94% store-writer
busy time — the many-core fresh-index wall is single-writer SQLite
ingest, not extraction or main-thread work. d2s improves the writer
lane ~11% (structured-clone deserialization avoided on the writer) and
frees the main thread; the remaining many-core gap is a
store-architecture arc (deferred index builds, multi-file
transactions, buffer→bind), out of the kernel project's scope.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-07-16 23:38:32 -05:00
co-authored by Claude Fable 5
parent 03d54e47a1
commit 28068fa0f1
7 changed files with 277 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/rust-kernel-migration-plan.md
+25
View File
@@ -264,6 +264,31 @@ Default routing: `DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}` in
Windows VM still deferred (same fallback rationale as §4b).
- Default routing now includes `java`.
### 4d. Direct-to-store decode (2026-07-16) — and where the wall ACTUALLY is
Kernel-routed files now ship their flat buffers from the parse worker all the way
to the STORE WORKER, which decodes + finalizes them there (`tryKernelExtractRaw` →
`ExtractionResult.kernelBuffers` → `KernelStoreBundle` → `decodeKernelBundle`;
filter semantics shared via `finalizeStoreBundle`). The main thread's per-file work
drops to O(1) + the content hash — it never materializes per-node objects, and both
postMessage hops move flat bytes instead of object graphs. Files whose applicable
frameworks carry an `extract()` hook keep the decoded path (hooks merge into decoded
results); non-writer paths (main-thread store, tests) materialize via
`materializeKernelResult`. Byte-identical dumps re-verified on dubbo, excalidraw,
express, gson.
**Measurement that closes the §4c question:** with the store worker instrumented,
dubbo's parse-loop wall is **94% store-writer busy time** (4,202ms of 4,493ms on the
kernel arm). The many-core fresh-index wall is the single-writer SQLite ingest —
not extraction, not main-thread work. d2s still improves the writer lane ~11%
(4,726→4,202ms: buffers skip structured-clone deserialization ON the writer) and
frees the main thread, but the remaining cbm gap on many-core medium repos is a
STORE-ARCHITECTURE question (their RAM-first design defers all durability). Next
levers there (a separate perf arc, not this project): deferred/bulk index builds
during the parse phase, multi-file write transactions, buffer→bind without object
materialization. Note the #1320-arc post-mortem already measured statement batching
and sorted inserts as ~zero on this path — B-tree maintenance is the floor.
## 4. Per-language tracker
Tiers: **T1** = mostly `.scm` + mapping config. **T2** = needs bespoke pre/post passes kept