R7b batch 4 #1 (docs/design/r-kernel-port-checklist.md is the authoritative
quirk list; survey + probe record therein). The lightest-shared-surface,
heaviest-hook port: languages/r.ts works entirely through the visitNode hook
(every type list empty except callTypes:['call']), so the walker is a file
node + a faithful hook transcription + the generic extractCall + pre-order
recursion — four shared machineries (value-refs, static-member reads, type
annotations, fn-ref capture) are dead by language gates and stay dead.
Grammar prep is the first true no-op of the arc: the crates.io tree-sitter-r
1.2.0 tarball ships parser.c AND scanner.c sha-identical to the r-lib v1.2.0
tag the vendored wasm was built from — crate pin only, no wasm change, no
bump gate; kernel-grammar-parity gains the r row (ABI 14, same-revision).
Preserved bug-for-bug (all probe-pinned): calls "return" on every return(x)
(named node in v1.2.0), the import quintet's silent dynamic-arg consumption
vs class/generic fall-through asymmetry, library(help = pkg) importing the
named arg, class-idiom variable suppression by callee name, chained/right-
assign/precedence-ghost gaps, env$fn body-leak-to-file, raw-text callees
verbatim (pkg::fn, obj$meth, "strfn" quotes kept, (handler) conversion),
duplicate same-(kind,name,line) ids, roxygen dropped entirely, UTF-16
columns/slices.
Gates: parity sweeps first-run 0-diff on AnomalyDetection/dplyr/ggplot2/
shiny (838 files; deferrals exactly 0/0/0/1 — the 1 is the moustache-
template pseudo-R file, both-arm) — kernel-parity.mjs gained lowercased-
extension matching so .R files sweep (matches detectLanguage routing);
full-init dumps byte-identical kernel-vs-wasm on dplyr/ggplot2/shiny;
kernel-r-parity suite (torture fixture + in-memory CRLF + BOM variants +
defer pin + kernel-arm quirk pins); full suite 2,638 green ×2 with
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += r (16 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sixth R7b port — the T1½ batch finale. Checklist-first recipe
(docs/design/kotlin-kernel-port-checklist.md, 1,121 lines, dist-extractor
ground truth); parity passed FIRST RUN on all three repos.
THE NOVEL MECHANISM — vendored-grammar-C (the §4 tracker's prescription,
first use): the crates.io tree-sitter-kotlin 0.3.8 pins `tree-sitter >= 0.21,
< 0.23` (the kernel links 0.25) and tree-sitter-kotlin-ng is a DIFFERENT
grammar (8 fields vs 0, renamed kinds — extractor-breaking), so no crate dep
is possible. The fwcd 0.3.8 tag's sha-matched parser.c + scanner.c are
vendored into codegraph-kernel/grammars/kotlin and compiled by build.rs (cc),
exposed via tree-sitter-language::LanguageFn. The wasm re-vendor is
behavior-NEUTRAL (0 CST/error disagreements across 1,984 gate-repo files;
old-vs-new full-init dumps byte-identical ×3) — a reproducibility re-vendor,
ABI stays 14.
Walker firsts: extension-function receivers (getReceiverType →
`WidgetK::extend` QN OVERRIDE with no package prefix, the qualified-receiver
`com::qext` first-segment bug, and the owner-contains fallback that excludes
`interface` kinds and is source-order dependent) and extractModifiers
(expect/actual platform modifiers → the node DECORATORS wire field on every
created node — the KMP synthesizer's feed, incl. `actual typealias`).
Preserved bug-for-bug: the FIELD_COUNT-0 dead cluster (no signatures, ZERO
type-annotation refs), hook-consumed property initializers emitting nothing
(incl. `by lazy {}`), the bodiless-vs-bodied class header asymmetry, enum-
entry bodies being invisible, KDoc never a docstring AND chain-breaking,
comment-gluing into import/package extents, `@Anno(args)` emitting nothing
while `@Marker` decorates, zero instantiates refs, the paren-then-lambda
`trailing()` garbage callee, text-includes visibility/suspend false
positives, and the packaged-file value-ref target drop. The fun-interface
misparse-recovery hook is DEFER-SHIELDED (every such file has_error) and
deliberately not ported. The swift-sweep lesson pre-applied: the shared
`assignment` shadow-prune case is implemented alongside the
property_declaration case.
Gates: sweeps 0-diff okio 299/322, okhttp 531/580, kotlinx.coroutines
1031/1082 (deferrals exactly the predicted 23/49/51 — both-arm grammar
reality incl. PHANTOM hasError files with complete CSTs; the kernel trusts
the flag); full-init dumps byte-identical ×3 (46.5k/108.9k/92.3k lines); KMP
expect/actual synthesis IDENTICAL across arms (412 edges on
kotlinx.coroutines — the tracker's KMP validation); kernel-kotlin-parity
suite (torture reflowed off the phantom shapes + .kts script + CRLF variants
+ fun-interface and phantom defer pins) + kotlin grammar-parity row (the
C-build ↔ wasm table identity proof); full suite 2,633 green ×2 under
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += kotlin (15 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fifth R7b batch-3 port, checklist-first recipe
(docs/design/swift-kernel-port-checklist.md, 1,056 lines — the largest of the
arc, with a built-extractor-validated emission pin and a childForFieldName
truth table).
Grammar bump first, validated standalone: tree-sitter-wasms ^0.4.0 (ABI 13) →
crate 0.7.3 — with a provenance twist: the wasm is built from the CRATE
TARBALL's src/ (alex-pinkus keeps generated files off main and the
0.7.3-with-generated-files tag ships an older ABI-14 generation that can never
sha-match; grammar.json rules are JSON-equal; the tarball is byte-for-byte
what the kernel's cargo build compiles — table identity by construction).
Older crates evaluated and rejected: clean-parse shapes are byte-identical on
0.7.3 (53-line CST battery diff, all inert), so an older pin buys nothing and
loses the macro-era wins. Delta = error-set membership (63 old-error files
parse clean: swift-testing #expect, #Preview/#GET macros, package access,
typed throws — vapor 23.1%→9.3%; 21 NEW-only regressions in 3 probed
construct classes) + two gate-found categories: docstring boundaries near #if
directives (7 clean files, docstring-field-only — verified mechanically) and
array-literal-callee call refs (2 refs, 1 file). Every hunk classified via
the error-union rule + parked-ref↔edge ripple pairing.
Walker (the arc's biggest) centers on the #1020 DEDICATED property branch:
computed properties → property nodes with the getter walked under the
property (SwiftUI body), static let/var → constant/variable, stored → field,
decorator/type-annotation/@Siblings-attr-arg refs all attached to the
ENCLOSING TYPE, stored initializer calls attributed to the class. Preserved
bug-for-bug: the never-resolving 'parameter' field (zero param type refs,
zero signatures), present-false isAsync, open→internal visibility,
everything-is-extends inheritance (first type_identifier per specifier), no
instantiates refs ever, subscript reads as `calls arr`, `defer` as `calls
defer`, multi-case enum entries minting only the first case, /** */ block
docs ignored AND chain-breaking, init/deinit/subscript minting no nodes with
visitNode-routed bodies (calls → class, static reads → nothing), multi-
segment extension resolveName, sugar extension names, the #selector shapes,
and the value_argument label-forward skip. ONE fix found by the sweep (then
pinned in the fixture + checklist): the shared `assignment` shadow-prune case
is swift-live — declared-then-assigned `let X: T` prunes X as a value-ref
target.
Gates: sweeps 0-diff Alamofire 89/98, vapor 224/247, swift-nio 407/554
(--max-deferral 0.3 — swift error incidence is 9–27% on BOTH arms,
structural; every deferral count matches the survey's table exactly);
full-init dumps byte-identical ×3 (31.9k/20.7k/126.3k lines); the Alamofire
census reproduces property=348 (the #1020 number) on the kernel arm;
kernel-swift-parity suite (206-line torture + CRLF + the #if-between-enum-
cases defer fixture) + swift grammar-parity row; full suite 2,626 green ×2
under CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += swift (14 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
First R7b language port. Grammar: tree-sitter-rust pinned =0.24.2 + wasm
vendored from tag 77a3747 (parser.c/scanner.c sha-matched against the
crates.io tarball), replacing the 2023 ABI-14 tree-sitter-wasms build —
the bump alone is precision-positive on the wasm path (receiver-qualified
instance-method resolutions replace ambiguous bare-name matches; node
sections byte-identical on ripgrep/tokio).
Walker mirrors the TS reference bug-for-bug per
docs/design/rust-lang-kernel-port-checklist.md (survey artifact): dead-code
isAsync, impl-pushes-no-scope, the impl-Trait-for-Generic<T> trait-receiver
quirk, phantom const identifiers, use-binding triple emission,
wildcard-use-emits-nothing, scoped-supertrait drop, chained-call re-encode
gated on scoped_identifier, Rocket route macros body-only.
Gates: parity sweeps 0 diffs — ripgrep 101/101, tokio 790/790,
rust-analyzer 1217/1488 (271 deferrals are token-macro-table sources that
error on BOTH arms — grammar-inherent); full-init dump-diffs byte-identical
on all three (3,857 / 13,440 / 39,030 nodes); kernel-rustlang-parity suite
(torture + CRLF + defer) in npm test; full suite green x2 with
CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Store-architecture arc round 2. The batched resolution loop reads
unresolved_refs ONLY through the status index + the PK keyset pager;
the other five ref indexes (from_node, name, file_path, from_name,
failed_tail) serve sync-time paths — yet every per-batch DELETE of
resolved refs maintained all of them, the biggest single main-thread
stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase)
and 50-81s at kernel scale.
beginBulkRefLoad/endBulkRefLoad on DatabaseConnection, threaded as
refIndexLoad hooks next to the existing bulkEdgeLoad pair with the
same minRefsForPool gate (small syncs never pay): drop the five for
the loop, rebuild each in one scan at the end — where the table holds
only the surviving FAILED refs (resolved rows are deleted by then),
so the recreate is near-free. Crash inside the window heals on the
next open (schema.sql re-applies CREATE INDEX IF NOT EXISTS).
Measured:
- dubbo: deletes 1.2 → 0.2s, marks 0.6 → 0.3s, recreate 219ms; wall
~8.5s flat — the freed main-lane time shifts into settle (the worker
lane now binds the double-buffer at medium scale).
- Linux kernel 8c: resolution 423.4 → 275.9s (deletes 50-81 → 3.2s,
backpressure 16.8 → 7.4s — fewer index writes mean less WAL and
cheaper folds), ref recreate 10.3s. Envelope ≈ 11.0min, from the
14.8min pre-arc best; <10min-on-8c now needs ~1 more minute.
Gates: dubbo/gson dumps byte-identical; linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Store-architecture arc round 1 (the cbm speed bar: dubbo warm wall
10.7-11.2s vs their ~7.5). §4d measured dubbo's parse-loop as 94%
store-writer busy with B-tree maintenance as the floor (statement
batching and sorted inserts already killed at ~zero). This applies the
resolution phase's proven edge-index window to the whole parse lane:
beginBulkParseLoad/endBulkParseLoad on DatabaseConnection — FRESH-INIT
ONLY (incremental runs delete per-file rows through the file_path
indexes) — drop all 15 nodes/unresolved_refs/files secondary indexes
plus the 4 non-unique edge indexes for the parse phase's mass insert
(the UNIQUE edge identity index stays: OR-IGNORE dedup conflicts on it,
and its source prefix keeps mid-window reads indexed), then rebuild
each in one table scan before resolution, with a yield between builds
(the endBulkEdgeLoad watchdog rationale). A crash inside the window
heals on the next open — schema.sql re-applies CREATE INDEX IF NOT
EXISTS.
Measured:
- dubbo (cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild
665ms, warm fresh-init wall 10.5-11.3 → 8.46-9.39s (−19%); the bar
gap vs cbm shrinks from ~3s to ~1.1s.
- Linux kernel 8c: envelope ≈ 14.2min, best ever (prior 14.8). Parse
itself flat (linux parse is extraction-bound, not writer-bound) and
the rebuild costs 21.6s — but every downstream phase dropped
(resolution 517-589 → 423.4s, edge-recreate 36.5s, synthesis 157.1s,
maintenance 16.3s): bulk-rebuilt B-trees are densely packed where
incrementally-grown ones are fragmented, so every index-mediated read
for the rest of the run pays fewer pages.
Gates: dubbo/gson/express/excalidraw full dumps byte-identical
(dubbo's canonical 441,270 lines reproduced); linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests). Incremental
sync paths untouched by construction (freshDb gate).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Two nudge shapes probed at 8c kernel scale (passive checkpoint at the pool
recycle boundary, fire-and-forget and awaited). The awaited shape reached
every §7a.7 shallow floor (read 16.6s, inserts 31.4s, deletes 50.0s) and
paid exactly what it saved (207.4s of attributed folds); fire-and-forget
regressed deletes 81→171s via fold/delete I/O contention while the
size-blind growth baseline kept the hard-cap parks firing on top (143
nudges, still 22 parks). All three arms within ~2.5% of each other — the
phase's fold I/O is a fixed budget the baseline already overlaps
off-thread, and the §7a.7 "~45s gap to the floor" is illusory. Code
reverted; #1362's valve + recycling remain the optimum of this family.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Task #5 step 2. The fuse-then-link refactor (#1364) left the extraction
sweep as a clean per-file boundary: raw text in → collected facts out.
This ports that sweep to the native kernel: `cfnptr_scan_files`
(codegraph-kernel/src/cfnptr.rs) strips and scans a batch of 16 files
per NAPI call, and the TS side only reads files, ships batches, interns
the returned facts, and resolves include paths. The JS sweep remains as
the fallback (no binary, feature detection against older binaries,
CODEGRAPH_KERNEL=0, or CODEGRAPH_KERNEL_CFNPTR=0).
Parity discipline: the JS regexes are the spec, so the scanners are
hand-rolled byte machines reproducing that engine — ASCII \w/\b next to
UNICODE \s (NBSP/U+2000-200A/FEFF decoded from UTF-8), alternation
order, lastIndex resume, and the observable backtracking dimensions
(INIT/ARRAY modifier and struct/star/bracket optionals, DISPATCH's
greedy segment loop); greedy shortcuts only where backtracking provably
can't rescue a match. The native stripper blanks per UTF-16 code unit,
so its output is string-identical to the TS stripper — pinned by a new
kernel arm on the strip differential oracle (fixtures + 500 seeded
random cases).
Gates, all green: new differential suite (adversarial fixture project —
CRLF, NBSP, continuations, decoy strings, unterminated comments,
backtracking shapes — indexed native-vs-JS: identical edge streams,
plus a record-level scanner check); repo differential on
git/redis/vim/SameBoy (identical, 705/852/433/180 edges); probe-hash on
the live linux kernel DB reproduced f6e1713d… (279,335 rows); linux
init counts exact 2,049,153/6,413,518; dump sha 6dd1185b… reproduced
(10,446,478 lines); full suite green ×2 (153 files / 2588 tests).
Measured (8c cg1212, quiet host): cFnPtr sub A=47.9s B=1.1 C=40.9
D=24.1 E=36.8 = 150.9s vs step 1's 179s and the pre-arc 230s (−34%
cumulative); the sweep itself halved (94.5→47.9s, JS strips
132.4k→68.9k). callback-synthesis phase 199.9→171.1s. E's attributed
wall grew from overlap shift under parallel synthesis; the phase total
is the honest number. Full record: plan §7a.10.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Task #5 step 1 (plan §7a.8/§7a.9). The C/C++ function-pointer dispatch
synthesizer swept every file's text four times (typedefs, registrations,
propagation, dispatch); at kernel scale the all-or-nothing source cache
declines, so that was 4.4 read+strips per file — 78s of the ~230s pass.
Now ONE extraction sweep reads+strips each file once and collects typedef
names, struct-node field declarations (structurally parsed, fn-pointer
classification deferred until the typedef sets are complete), resolved
includes, an alias-shaped-macro name set, and per-file survival filters
(init type tokens, array element types, inline-struct summaries,
field-assign pairs, dispatch fields/array names — interned, a few MB on
linux). The linking stages then replay the ORIGINAL pass bodies verbatim:
struct layouts register in kind-scan order (same-name precedence is
order-sensitive), and registration/propagation/dispatch run only for
files their filter proves can have side effects, lazily re-stripping just
those. Filters only over-approximate (full-file no-skip scans ⊇ the real
passes' jump-cursor scans), and a filtered-out file is one where every
match fails the pass's own gates before any side effect — parity by
construction. Macro tables stay lazy: a sizing probe found 6.1M #define
lines on linux (amdgpu register headers), ruling out retention.
Measured (8c cg1212, quiet host, fresh kernel init): cFnPtr pass 230s →
179s (−22%); strips 283.5k → 132.4k (4.44 → 2.08/file, 78 → 46.6s);
dispatch stage 95 → 18.5s; callback-synthesis phase 250 → 199.9s.
Standalone probe on the live DB: 139 → 122s.
Byte-parity gates, all green: probe-hash identical on the live kernel DB
(279,335 edge rows both builds); git/redis/vim/SameBoy full dumps
byte-identical old-vs-new (macro tables, commands.def, #ifdef
include-units, inline structs, bare arrays exercised); kernel-parity
0-diffs on git/redis/fmt/protobuf, deferral unchanged; linux counts
exact 2,049,153/6,413,518 and dump sha 6dd1185b… reproduced
(10,446,478 lines); full suite green ×2 (152 files / 2563 tests).
Step 2 (native per-file extractor) now has its boundary: the extraction
sweep, raw text in → records out.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three measurements before any port. The stripCStyle split('') rewrite
(byte-identical segment-builder) measured 1.0× on 15.1M chars of linux C
— V8's ~73MB/s scan rate IS the cost, and 78s ≈ 4 strips/file × that
rate: the lever is the redundancy, not the scanner. Rewrite reverted;
the differential oracle test ships so any future rewrite stays pinned
byte-identical. E's regexes alone run at ~46MB/s (~30s of its 95s; the
rest is per-match logic and slicing).
Re-ordered attack recorded in §7a.8: step 1 = TS fuse-then-link refactor
(strip once per file, collect raw matches + declared-type tables,
text-free global linking; ≈ −70-90s, parity via collector insertion
order + the §7a.4 probe-hash gate); step 2 = native per-file extractor
behind the same boundary (raw disk text — no preParse interaction).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The §7a.6 anomaly probed to its mechanism with five discriminating runs
(§7a.7 table): main-thread B-tree writes triple under attached readers
because READERS PIN WAL checkpoint progress — the deep WAL taxes every
writer page operation (deletes 42.6s pool-off vs 118.8s pool-4 on
identical hardware; an aggressive 64MB valve recovers the writes but
overpays +129s in full-park folds; the v2 cache resurrection was
falsified — long-tail name traffic is uncacheable at any capacity).
Fix: workers close and reopen their read-only connections every 8
batches at the double-buffer's worker-idle boundary
(ResolverPool.recycleWorkers + QueryBuilder.rebind + a cadence call).
Reopens are sub-millisecond, resolver caches survive (only prepared
statements re-prepare), and the existing checkpoints advance instead of
parking. Failed recycle downgrades to sequential, same as a failed
fan-out.
Measured (8c pool-4, linux v7.2-rc2, cadence 25 → 8 iterated):
resolution superphase 715.0 → 633.6s (−11.4%), envelope best 14.8min,
recreate 59.7 → 45.3s. Byte-neutral everywhere: git dumps byte-identical
old-vs-new, linux dump sha 6dd1185b reproduced (10,446,478 lines),
counts 2,049,153/6,413,518, suite 2517 green. 2c unchanged by
construction (no pool → no recycling).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fresh CODEGRAPH_RESOLVE_PROFILE tables at 2c and 8c on the round-2 build:
the 8c settle stage is 3.6s — the double-buffer absorbs the entire
resolveOne population, superseding §7a.2's core-invariant framing. The
8c cost is writes-under-readers: main-thread deletes+insertEdges run ~3×
slower (+102s) with 4 readonly workers attached — mechanism unproven,
named as the next probe. Two same-day experiments killed by measurement
and reverted: budget-scaled name caches (v1 LRU regressed via
delete+set-per-get churn + GC; v2 mutation-free second-chance cache
landed exactly on baseline — the 11µs exact-match is per-ref floor, not
refetch overhead) and lazy candidates JSON (read stage flat). New 2c
record 16.5min clean-host; 8c range 15.0–16.4min across n=2 — ranges,
never single runs. Round-2 deferral cuts did move 8c parse (202.6→178.5s).
Levers re-ranked: writes-under-readers probe > cFnPtr native site
extraction (~230s of synthesis) > backpressure byte volume > recreate.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Census-driven cut of the top-ranked post-R7a lever. All passes TS-side,
C-only (preParseCSource), shared by both arms:
- parameterized-annotation whole-blank (__free/__printf/__counted_by/
__bpf_md_ptr…; extends through a stranded field `;`)
- type-keyword-arg scanner (kzalloc_obj(struct T), list_entry, multi-line
continuations behind nested-paren args; bounded hand scanner, head
exclusions + call-vs-declaration guard; blanks trailing stars)
- static/extern CAPS-macro declaration lines at any scope; the initialized
form is REWRITTEN to its expansion (name/tail keep exact offsets)
- va_arg qualified-type blank; GNU named-variadic #define dots-only blank
(post-restore); sandwiched notrace-family; C23 auto; multi-line
iterator-macro spans (hlist_for_each_entry_rcu + lockdep arg)
- word list += cacheline family (2- and 4-underscore spellings) + 10 more
census-confirmed annotations
Gates: five-repo parity sweeps 0 diffs (git deferral 16.1→12.2%, redis
25.3→24.1%, fmt/protobuf unchanged); linux full-tree both arms
2,049,153 nodes / 6,413,518 edges (+858/+6,585 vs R7a) with byte-identical
dumps (10,446,478 lines, sha256 6dd1185b); kernel-arm parse-loop 356→306s
at 2c; suite 2517 green under CODEGRAPH_KERNEL_EXPECT=1. Honesty note
recorded in the docs: error recovery was already salvaging most SYMBOLS on
deferred files — the graph win is relationships + phantom cleanup, and the
unreleased CHANGELOG entry was rewritten off the sweep-subset framing.
Also records §7a.5: post-R7a 8-core cg1212 re-run 16.4min (was 18.3min);
8c parse sits on the single-writer floor, so the <10min-on-8c gap re-ranks
to the per-ref resolution path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps
byte-identical on all five + linux at kernel scale (10.4M dump lines,
same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs
wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new
blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x).
Deferral guard corrected by measurement (C/C++ error incidence 9-42%;
--max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse
cost deferred files paid.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
tree-sitter-wasms shipped 2023-era C/C++ grammars; the kernel walker must
compile the crates.io versions, so production wasm upgrades FIRST and in
isolation (the R2 pattern). Built with tree-sitter-cli 0.25.10 from each
tag's CHECKED-IN parser.c: tree-sitter-c v0.24.2 (b780e47, parser.c
f2883ff9), tree-sitter-cpp v0.23.4 (f41e1a0, parser.c 2a35a43b, scanner.c
cf60387d) — sha-matched against the crates.io tarballs. .metal/.cu map to
language 'cpp' so the dialects ride the same coherent grammar.
Full suite green with the upgraded grammars (2,490; one unrelated daemon
idle-timeout flake passed solo 9/9) — incl. the UE-macro, misparse-guard,
Metal, and CUDA coverage. Checklist updated with the vendored revs and the
metal/cuda routing correction.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Step 1 of the §0a recipe for the C/C++ port: every tree-sitter.ts branch,
extractor-config field, name-salvage helper, and gate the walker must
mirror, with file:line anchors. Locks the architecture: preParse (all seven
blanking passes) hoists to the kernel route point so none of it ports to
Rust; Metal/CUDA stay on wasm this round; one dual-language walker module.
Key asymmetries surveyed: value-refs are C-only (cpp absent from
VALUE_REF_LANGS), static-member refs are cpp-only, fn-ref spec is
cFamilySpec with addressOfOnly for cpp.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cFnPtrEdges was 86% of kernel-scale synthesis (306s, §7a.3). Standalone
probe iterations against the live kernel DB attributed and fixed:
- sliceLines split the ENTIRE file per node (~1.6M full-file splits across
the B/D/E sweeps) — split once per file, slice from the line array
(D 46.2->20.5s, E 94.5->69.1s).
- The 128-entry strip cache re-stripped every file 4.4x across the four
file sweeps (71.8s). Sizing is now memory-budget-aware and ALL-OR-NOTHING
with 5% slack over the file count: a partial LRU on cyclic sweeps thrashes
to ~0% hit (measured twice: cap ~61k and cap == files both thrashed
against 63.8k + includes). Full-cache strips: exactly 1.0/file, 15.5s.
- Pass D pre-gates field names on fieldToStructs before the two regex type
resolutions — a->f = b->g matches every struct-field assignment in the
tree, ~99% of them data fields.
- recvTypeIn/varTypeIn compiled a new RegExp per call - cached per name.
- Line numbers counted incrementally per match (ascending indexes) instead
of splitting the body prefix per match; cursor rewinds between the two
dispatch scans.
- CODEGRAPH_SYNTH_TIMINGS now prints per-sweep walls + read/strip/nodes
accounting for this pass.
278.8s -> 134.8s standalone at the 2c envelope. Output identity proven at
full scale: probe edge set (merge-dedup applied, canonical sort) SHA256
21c2a971... == the original code's 274,762 edges extracted from the live
kernel DB. Pass suite 15/15.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The §7a.2 per-ref profile overturned the assumption the whole arc was
built on: resolveOne owns only ~93s of the kernel-scale ~433s batch loop.
Loop-stage attribution (CODEGRAPH_RESOLVE_PROFILE, shipped here) named the
rest: backpressure folds 111.2s, count guard 93.9s, batch reads 54.6s,
deletes/inserts/marks ~84s, settle 85.7s.
- Non-progress guard O(remaining)→O(1): the per-batch COUNT(*) walked every
remaining pending row (O(N²/batch) per run, 93.9s). The cleanup queries
now return summed SQLite , and zero-removals-from-claimed-work
is the guard signal — the DIRECT evidence the count diff inferred (a
mismatched-name resolver makes keyed cleanup no-op ⇒ changes=0). A real
COUNT runs only on that suspicious path and arbitrates exactly as before.
- Batch reads OFFSET→keyset (54.6s→O(batch)): OFFSET re-walked the
accumulated failed-row prefix every read; seeking past the last-seen
rowid is prefix-independent and enumeration-order identical.
- WAL valve caps scale with DB size (env still wins): every fold re-writes
hot pages (#1231 in bounded form — 111.2s at the flat 256MB cap);
soft=clamp(dbSize/4, 256MB, 2GB) trades ~4× fewer folds for a transient
WAL ≈ project size.
- CODEGRAPH_RESOLVE_PROFILE: per-outcome resolveOne histogram + loop-stage
attribution, main + workers, off by default.
Gates: dubbo dump byte-identical; suite 2,491 passed / 4 skipped (kernel
required). Kernel-scale payoff run lands in the plan doc next.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A truncate checkpoint started against an ACTIVE writer wins the lock and
then blocks that writer for its entire backfill; after the edge-index
recreate's multi-GB single-transaction burst that exceeds the writer's 5s
busy_timeout and fails the index with 'database is locked' (§7a.2 record
run, EXIT=1 at kernel scale — the small mid-resolution truncates folded in
ms and masked the hazard). Barrier truncates (backpressure/foldNow) are
collision-free by construction: the writer is awaiting the valve.
Dubbo gate: exit 0, peak 81MB (barrier folds carry containment), dump
byte-identical. Valve + sizing suites 27/27.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four §7a.1 instrumented-run findings, each measured:
1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still
grows the FILE without bound — the writer only restarts at frame 0 when a
commit finds zero reader marks, which the instrumented run showed never
happens (file marched 361→721MB through two COMPLETE backfills; 22GB by
phase end). backpressure() now also trips at 4× the soft cap on raw file
size and TRUNCATEs at the parked barrier; the timer path truncates
opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at
the same 16MB valve; dumps byte-identical under aggressive folding.
2. cgroup memory credit: memory.current counts reclaimable page cache — a
post-parse container read 57MB of headroom on a 6GB box and silently
disabled the pool. inactive_file is credited back (the docker-stats
working-set convention); the same run now reads a sane 4.4GB budget.
3. Pool at 2 cores reversed: sequential resolution measured FASTER than
pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is
Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing.
cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path.
4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower
(493s vs 369s) — main + store-worker don't fill the second core. Floor
restores the baseline (373.5s measured).
Plus the observability §7a.1 burned three 25-minute cycles for: valve
armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture,
pool sizing decisions (incl. the disabled path), backpressure-hook
presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG.
Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record
runs with this build follow in the migration plan §7a.1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three §7a.1 run-1 lessons (kernel-scale 2c/6GB: EXIT=137, WAL 22.2GB with
the backpressure hook DEPLOYED):
1. TRUNCATE at parked barriers: a completed passive backfill bounds the
un-checkpointed backlog but the FILE only stops growing when a commit
finds zero readers holding WAL marks — rare while pool workers cycle
(dubbo debug baseline: file climbed monotonically through six completed
pass-1 backfills). At a parked barrier the no-reader window is
guaranteed, so chop the file there with wal_checkpoint(TRUNCATE)
(off-thread, 2s busy_timeout — a racing reader degrades it to a no-op).
2. Futility latch: when backfill gives up (pinned reader), parking again at
every over-cap boundary burns a 20-pass checkpoint attempt — each a
worker thread + fresh connection against a multi-GB DB — per batch. Two
consecutive give-ups now disable parking for 60s; a pinned phase degrades
to pre-valve behavior instead of OOM-amplifying.
3. CODEGRAPH_WAL_VALVE_DEBUG=1 surfaces valve decisions without the
caller's verbose plumbing, and give-up lines print under
CODEGRAPH_SYNTH_TIMINGS — run 1 failed silently because give-ups were
verbose-gated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Pool sizing used os.cpus().length, which enumerates the HOST's CPUs: inside
a 2-CPU cpuset it sized 6 resolver workers (the §7a.1 false-'sequential'
premise) and 8 parse workers, and at true 8-core concurrency six ~1GB
workers OOM-killed a 7GB container (oom_kill=5) mid-synthesis — sizing had
no memory term and no override knob.
resolvePoolSize (pure, matrix-tested): explicit CODEGRAPH_RESOLVE_WORKERS
override (0 disables, cap 16); CPU term max(2, min(availableParallelism-1,
6)) — cpuset-honest, floored at 2 so true 2-core boxes keep pooled
synthesis's ~2×; memory term floor(budget*0.7 / clamp(0.2*dbSize, 256MB,
1.5GB)) with budget = min(freemem, cgroup v2/v1 headroom). Parse pool's
core input switches to availableParallelism. Dev machines are unchanged
(still 6 workers); the 8c/7GB kernel-scale container now sizes 4.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
At kernel scale the pooled resolution/synthesis superphase grew a 22GB WAL
on a 4.6GB DB (cg1212, §7a.1): autocheckpointing is deferred for the run,
and the valve's timer-driven passive checkpoints stay perpetually partial
against the pool's continuous reads — no mechanism ever completed a
backfill, so the WAL accreted the whole phase's write volume, blowing disk
and feeding page-cache pressure into the 8-core/7GB container OOM.
The valve's writer-side backpressure() hard-cap backstop existed but was
wired only into the PARSE orchestrator. Thread it into the resolution batch
loop at the double-buffer's one pool-idle boundary (batch settled, next not
yet fanned out), after the edge-index recreate, and through the synthesis
insert loops. Parked there, the backfill completes; readers re-enter at
SQLite's backfilled mark and the next persist commit wraps the WAL.
Dubbo validation, same build: valve@16MB peak WAL 251MB (floor = the
single-transaction edge-index recreate) vs defaults 914MB; dumps
byte-identical (441,270 rows); wall unchanged (11s). Suite 2,479 green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
On CRLF checkouts (every Windows autocrlf clone) the JS reference's
block-continuation strip /^\s*\*\s?/gm finds a line start after the \r and
its greedy \s* consumes the \n, leaving a bare \r in the docstring; the
kernel's (?m)^ pass matched after \n only and kept \r\n. Caught by the O2
Windows VM leg (6 kernel-tsjs-parity failures), reproduced on macOS by
CRLF-converting the fixtures.
js_multiline_strip now replicates the JS anchor set (\n, \r, U+2028, U+2029)
for all five line-marker passes; CRLF variants of every torture fixture are
pinned in kernel-tsjs-parity, derived in-memory so nothing can normalize
them away.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R1-R6 are done; §0 is now the open-work list in recommended order
(merge → Windows VM leg → P1 resolution → C/C++ port → long tail), and
§0a carries everything a fresh session needs: where the work lives, the
build/gate commands, the proven add-a-language recipe, and the paid-for
traps (encoding-dependent error recovery → defer policy, node-ID-string
dedupe, UTF-16 positions/slices, the exact-seam contract, crate+wasm
grammar lockstep). §1/§6 keep the original expectations with SUPERSEDED
annotations pointing at the measurements that corrected them; §7a now
carries the R6 numbers that make it the top open perf item.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cg1212 (Linux kernel, 2 CPU/6GB): completes in 26.4min vs the ~27min
baseline with all new machinery active — no regression; graph scale
identical (2,048,664 nodes / 6,405,964 edges). The §6 parse expectation
(6m → ~2m) was mis-premised: the tree is ~99% C, an unported T2
language, so it transfers to the C/C++ port (R7). Resolution remains
the kernel-scale wall (19.2m, 73% — P1). The tree's own Python tooling
files: 99/99 byte-parity. kernel-parity.mjs now skips dangling
symlinks (Linux dtc fixtures).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Python (codegraph-kernel/src/python.rs) and Go (src/go.rs) join the
native kernel, mirroring the wasm extractors bug-for-bug. Python:
decorated_definition docstrings/decorators (decorates refs only for
bare-identifier decorators — the call-kind quirk), function-in-class →
method, module-level assignments always extract as variable, from-import
per-name binding refs, self.x fn-ref candidates as bare names. Go:
receiver methods with Recv::name qualified names + contains edges to the
first earlier struct of that name, type_spec struct/interface
classification with embedding→extends and interface method nodes,
composite-literal instantiates keeping the package qualifier, top-level
var/const initializer walks attributed to the declared symbol (#693),
2-hop field chains (#1276), New().Method() re-encode (#645/#608), and
the GO_SPEC fn-ref layers.
Grammars: tree-sitter-python 0.23.6 + tree-sitter-go 0.23.4 crates, with
wasm vendored from the same tags (parser.c sha-matched) — both were
2023-era in tree-sitter-wasms.
Gates: extraction sweeps 100% (flask 83/83, django 3,035/3,038 +3
error-file deferrals, gin 99/99, prometheus 978/979 +1); full-init
dump-diffs byte-identical on flask (10,833 rows), gin (17,540), django
(360,794), and prometheus (213,758); torture fixtures enforced in npm
test. DEFAULT_ROUTED now covers typescript/tsx/javascript/jsx/java/
python/go. Full suite: 2,471 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kernel-routed files ship their flat tables from the parse worker to the
store worker as buffers (tryKernelExtractRaw → kernelBuffers on the
result → KernelStoreBundle); the store worker decodes and finalizes
(finalizeStoreBundle shared with the object path so filter semantics
can never drift). Files with applicable framework extract() hooks keep
the decoded path; the no-writer fallback materializes via
materializeKernelResult. Byte-identical dumps re-verified on dubbo,
excalidraw, express, gson; full suite green (2,467).
Measurement (plan §4d): dubbo's parse-loop wall is 94% store-writer
busy time — the many-core fresh-index wall is single-writer SQLite
ingest, not extraction or main-thread work. d2s improves the writer
lane ~11% (structured-clone deserialization avoided on the writer) and
frees the main thread; the remaining many-core gap is a
store-architecture arc (deferred index builds, multi-file
transactions, buffer→bind), out of the kernel project's scope.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring
the wasm extractor's Java paths bug-for-bug: package namespaces,
imports, javadoc, annotations→decorates, type_list inheritance,
static-final constants, enum constants, anonymous classes (including
the TS side's 0-based-line quirk on the extends ref), method_invocation
calls with the this.field unwrap and the Foo.getInstance().bar() chain
encoding (#645/#608), static-member value reads, method-reference
fn-refs (#756), value-reference edges, and the full Lombok member
synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/
EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared
docstring/textutil modules moved to crate level. Grammar:
tree-sitter-java 0.23.5, with the wasm grammar vendored from the same
tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build.
Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit
341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test;
full-init dump-diffs byte-identical on gson (49,766 rows), retrofit
(62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux
container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1.
The gate caught a real cross-language bug: fn-ref dedupe and value-ref
self-target checks must compare node ID STRINGS, not node-table rows —
ids collide for same-(kind, name, line) nodes, which minified one-line
bundles hit routinely (retrofit's website JS exposed it; latent in the
TS/JS walker since R2, never released). Fixed in both walkers.
Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat
(parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that
wall is main-thread-bound (reads + store), not worker-CPU-bound — the
§6 expectation assumed otherwise. Where worker CPU binds the kernel
delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s
(~1.25×). The identified lever for the many-core headline is decoding
kernel buffers directly into store rows (skipping per-node JS object
materialization); the buffer contract already carries everything.
DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate evidence (docs/design/rust-kernel-migration-plan.md §4b):
- Graph parity, byte-identical (stronger than the §5 ≤0.5% bar): full
codegraph-init dump-diffs kernel-vs-wasm on express (13,712 rows),
excalidraw (89,898), and vscode (2,378,238 rows) — identical bytes.
Python control repo (flask) identical + timing unchanged. The parity
harness is now ORDER-sensitive (emission order drives rowids, which
drive resolution order) and dumps come from the new
scripts/dump-graph.mjs (natural keys, no rowids/timestamps).
- The one real find, caught by the vscode tier: tree-sitter error
RECOVERY is encoding-dependent — byte-identical grammar sources and
the same core (0.25.10) recover erroring files differently under
UTF-8 (native) vs UTF-16 (web-tree-sitter) parsing; proven by
reproducing the wasm tree with a native UTF-16 parse. Policy: the
kernel defers any file whose tree has_error() to the wasm extractor
(silent 'defer:' signal, per file) — parity by construction on
erroring files (incidence 0-0.42% across the gate repos), and the
harness fails if deferrals exceed 10% so a broken kernel can't hide
behind the fallback.
- Retrieval invariants: canonical excalidraw flow (mutateElement →
renderStaticScene) connects end-to-end on the kernel-indexed graph;
synthesized-edge families present. Agent A/B is vacuous under
byte-identical DBs (same justification as #1320-#1322).
- Perf: vscode init 105.4s → 82.1s (1.28×) on an 11-core Mac;
excalidraw on a 2-CPU/6GB Linux container (the CI-runner envelope)
6.2-7.1s → 4.3-4.8s (~1.5×). Linux arm64 in-container build: all 22
kernel tests green under CODEGRAPH_KERNEL_EXPECT=1. Windows VM leg
deferred (VM stopped; prlctl start needs Parallels Pro) — benign: a
missing .node falls back to wasm, and the release matrix builds and
gates the win32 prebuilds.
- Full suite: 2,465 tests pass WITH default-on routing, so the entire
extraction corpus now exercises the kernel for TS/JS wherever a
.node is staged.
DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}. Override:
CODEGRAPH_KERNEL_LANGS (replaces the set) / CODEGRAPH_KERNEL=0 (kill).
Changelog entry added under [Unreleased].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the R1 seed .scm emitter with a bespoke Rust walker
(codegraph-kernel/src/tsjs/) that mirrors TreeSitterExtractor's TS/JS
paths function-for-function: declarations (incl. #808 field/property
classification), qualified names, docstrings (#780 wrapper climbs),
signatures, imports/re-exports + per-binding refs, calls with
receiver-qualified callees (#1230 literal-receiver skip), instantiations,
decorators, inheritance, type annotations (#381), type-alias members +
tuple contracts (#359/#634), React component recognition (#841
forwardRef/memo/styled), object-of-functions / zustand-through-middleware
/ RTK Query endpoints + generated hooks / vuex + pinia store shapes,
function-as-value capture with the flush gate (#756), and value-reference
edges with the shadow prune (#895/#897). The generic query emitter is
deleted — extraction parity needs logic .scm can't express; future
languages get walkers too (migration plan §4a).
Positions and JS string-slice semantics are emitted in UTF-16 code units
natively, so kernel output is byte-identical to web-tree-sitter's — no
column diff class exists.
Parity evidence (macOS): scripts/kernel-parity.mjs (full-object multiset
diff per file) — this repo 353/353 files, excalidraw 643/643 (10,650
nodes / 10,726 edges / 68,307 refs), plus torture fixtures checked into
__tests__/fixtures/kernel-parity/ and enforced in npm test by
kernel-tsjs-parity.test.ts. The strict compare caught one real decoder
bug the loose harness missed: refs must NOT carry denormalized
filePath/language at the extractFromSource seam (the store fills them).
Perf: extraction 2.6× single-thread on excalidraw (487ms vs 1,255ms,
identical outputs). Routing stays opt-in (CODEGRAPH_KERNEL_LANGS) until
the R3 equivalence gate (large repo, DB dump-diff, retrieval invariants,
agent A/B, Linux/Windows) passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 0 of the Rust extraction-kernel migration (docs/design/
rust-kernel-migration-plan.md, now checked in with §3a recording the
shipped state):
- codegraph-kernel/ napi-rs crate: extractFile(path, content, language)
→ five flat buffers (meta/nodes/edges/refs/arena), one JS boundary
crossing per file. Node ids computed Rust-side, byte-identical to
generateNodeId (pinned by test vector). Reserved per-node metrics slot
for the Arc 3.2 code-metrics work.
- Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures,
byte-range scope stack → ::-joined qualified names, contains edges,
refs attributed to the innermost enclosing symbol). Seed TS/JS queries
are smoke-level; R2 replaces them with the full port.
- Routing seam in extractFromSource with per-file wasm fallback.
DEFAULT_ROUTED is empty — no behavior change until a language passes
its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill
switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before
routing; EDGE_KINDS became a runtime array because kind order is now
wire contract.
- Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the
exact crate revisions (tree-sitter-typescript v0.23.2,
tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) —
the tree-sitter-wasms builds were 2023-era, which the new
kernel-grammar-parity test caught on day one. Production TS/JS parsing
gets 2.5 years of grammar fixes; full suite green (2456 tests).
- Build/release wiring: scripts/build-kernel.sh + npm run build:kernel;
release.yml kernel prebuild matrix (continue-on-error — the kernel is
optional everywhere, bundles fall back to the wasm path); bundles stage
lib/kernel/codegraph-kernel.node; release job runs the kernel suites
with CODEGRAPH_KERNEL_EXPECT=1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two hazards found by running today's full stack against the Linux kernel
(70,129 files) in the cg1212 repro container:
1. The parallel-synthesis fallback retried a worker-failed pass on the MAIN
thread. At multi-million-node scale a worker failure is usually a memory
ceiling, so the retry would OOM the process and take the whole index with
it. Above 1.5M nodes a failed pass is now skipped with a clear stderr
message (its synthesized edges are absent; the index completes). Below
that, the main-thread retry stays — small-scale worker crashes are
transient and the retry keeps coverage.
2. endBulkEdgeLoad rebuilt all four edge indexes in one synchronous span —
measured 79s at kernel scale, past the #850 liveness watchdog's 60s
stall window. A daemon-triggered re-index would have been SIGKILLed right
after doing the work. Now async with an event-loop yield between builds,
keeping each stall to a single index (~20s at kernel scale).
Validation: full Linux kernel index to completion in the repro container —
2,048,674 nodes / 6,405,964 edges, EXIT 0, zero passes skipped, on a 2-CPU
VM (worst case: pool disabled, sequential resolution + synthesis) in ~27min.
Phase walls: parse 6.0m, resolution 19.5m (incl. synthesis 6.3m, recreate
79s), maintenance 74s. Suite green (2444).
Also adds docs/design/native-extraction-kernel.md — the spike-validated
design for the native extraction kernel (Rust parse+walk over dubbo's Java:
202ms rayon / 1.07s single-thread vs 4.7s for the current wasm pipeline).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The resolution persist's measured cost is B-tree maintenance on the edges
table's five indexes (offline replay of a 224k-edge resolution set: 2.8s with
all indexes, 1.1s with only the unique identity index, +0.3s to recreate the
rest). On big runs (same >=150k-ref gate as the resolver pool) the four
non-unique edge indexes are now dropped for the batch loop and recreated in
one pass each before synthesis.
Why this is safe:
- idx_edges_identity stays: INSERT OR IGNORE's dedup conflicts on it (#1034),
and its leftmost column is `source`, so the only mid-window edge reads —
resolution's supertype walks (implements/extends by source) — keep an index
via its prefix (verified with EXPLAIN QUERY PLAN).
- The window closes BEFORE synthesis, whose passes read kind-keyed, and on
every error path (finally).
- A crash inside the window heals on the next DatabaseConnection open —
schema.sql re-applies CREATE INDEX IF NOT EXISTS, same recovery as the FTS
bulk-load pattern this mirrors.
- Concurrent readers (a daemon serving the project mid-index) stay correct;
target/kind-keyed reads degrade to scans only for the window's duration.
dubbo (4,402 files): persists 4.0s -> 3.0s, fresh init 11.9s -> ~11.1s,
graph byte-identical. excalidraw (below the gate): untouched, byte-identical.
Recreation cost ~250ms, logged under CODEGRAPH_SYNTH_TIMINGS as
edge-index-recreate. Suite green (2444).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The ~36 independent synthesis passes (callback/event/framework wiring) ran
sequentially on the indexer's main thread — 2.0s of a 4,402-file Java repo's
index, and the stage where kernel-class repos die (#1212). They now live in
an explicit registry (SYNTH_PASSES) and, when the resolver pool is alive
(>=150k-ref repos), fan out across its read-only workers: dubbo synthesis
2,024ms -> ~900ms (-55%), total fresh init 13.5s -> 11.9s. Graphs verified
byte-for-byte identical on both the pool path (dubbo) and the sequential
path (excalidraw).
Why this is safe: no pass's edges persist until the ordered merge, so every
pass sees the same committed post-resolution DB state in either mode, and
results merge in registry order regardless of completion order — the
first-seen dedup is unchanged. The pool now survives through synthesis
(destroy moved after it) instead of being torn down moments before the one
stage that could reuse it.
Robustness: a pass that fails on a worker (crash, OOM) is retried on the
main thread — a synthesizer blow-up now costs one worker instead of the
whole index, which is half the #1212 story on very large repos.
Also: ref-row cleanup deletes now run as one transaction with a cached
statement instead of one implicit commit per 500-row chunk (mechanically
fewer WAL commits; matters most on HDD-class storage). A set-based rewrite
of failed-ref parking was tried, measured ~zero on NVMe, and dropped — the
remaining persist cost is edge-index B-tree maintenance, not statement
dispatch.
SYNTH_PROGRESS_STEPS now derives from the registry (passes + fixed marks);
the pin test counts registry entries plus literal __mark sites.
Suite green (2444). Sequential-path timing unchanged on excalidraw.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fresh init on a 4,402-file Java repo (dubbo): 18.2s -> 13.5s (-26%), with
the resolution phase going 12.6s -> 7.9s (-38%). Graphs verified
byte-identical on both the pool path (dubbo) and the sequential path
(excalidraw).
Two changes:
1. The fastInit+pool path restored WAL for the resolver workers but left
wal_autocheckpoint at its default, so the persist loop inline-checkpointed
hot pages all phase long (#1231's pathology inside resolution — measured
at 58% of resolution wall). Checkpointing is now deferred behind the
bounded valve and folded once at maintenance, mirroring the deferWal path.
2. The resolution loop is double-buffered: batch k+1 is prefetched (OFFSET
past batch k's still-pending rows, under an explicit ORDER BY rowid) and
fanned out across the pool while batch k's ref cleanup runs on the main
thread. Batch settle-waits dropped 2572ms -> 117ms.
Correctness invariant found by the byte-identical gate and now documented in
the loop: batch k+1's resolution READS batch k's edges (resolveMethodOnType
walks supertype chains over extends/implements edges that resolution itself
inserts), so edges must persist BEFORE the next batch fans out; only the ref
cleanup overlaps.
Also extends the CODEGRAPH_SYNTH_TIMINGS instrumentation with phase labels
(grammar-init, parse-loop, fts-rebuild, resolver-reinit, resolution,
callback-synthesis) and pool timings (worker open/resolve, per-batch mode,
persist), so the next profile is one env var away.
Suite green (2444). Sync path timings unchanged. Pool floor re-validated:
forced-on at 40k refs is still net-slower, so the 150k threshold stands.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three compounding defects (#1196) made a query bag of object-literal
keys (`profileInfo isTrialEligible quotaInfo billingMethod`) return
unrelated results while the defining files never surfaced:
1. Step 5b title-cased interior humps (profileInfo -> Profileinfo) and
then compared case-SENSITIVELY, dropping every row SQLite's
case-insensitive LIKE had just recovered. The hump lookup is now
case-insensitive with an explicit uppercase-at-match requirement.
2. Step 5b/5c's kind whitelist held only type-like kinds — dead code on
method-centric codebases. Callable kinds (function/method/component)
are fetched as a SEPARATE LIKE batch so hot single-word terms can't
crowd classes out of the length-ordered 200-row batch.
3. explore's named-symbol seeding was exact-name only; a field token
seeded nothing. A camelCase token with ZERO exact defs now seeds its
camel-infix definers (callables, hump-boundary or prefix, shortest
first, capped at 3) — bare lowercase words keep the #1252 stopword
guard untouched.
The reporter's acceptance query is a pinned e2e test (definer files
present, exact-name seeding unaffected). excalidraw probe: the
canonical flow query (mutateElement renderStaticScene) is byte-
identical; NL queries shift toward more-central callables
(useUIAppState/getDefaultAppState over observer periphery).
Fixes#1196
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`.gitignore: /repos/` lists `repos/` as ONE ignored entry, while the
CLI hint (#1156) suggests `includeIgnored: ["repos/a/", "repos/b/"]` —
the child spelling. findIgnoredEmbeddedRepos tested the opt-in matcher
against the PARENT path only, which a child pattern never matches, so
the documented opt-in silently indexed nothing and init looped the
byte-identical suggestion back at the user (#1295).
Ignored dirs that don't match as a whole are now descended (the walk
was already bounded: depth 4 / 2000 entries, and only runs when
includeIgnored is configured) and each nested repo root is matched
individually — parent spelling opts in everything under the dir, child
spelling exactly the named repos. findUnindexedIgnoredRepos gets the
same per-repo check so the hint stops nagging about repos that are
already configured while still naming unopted siblings.
Fixes#1295
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
", ".join(sorted(x)) resolved by bare name to a project function named
join — one nested inside a DIFFERENT function, so scope alone rules the
edge out. Both defects from #1230, fixed independently:
1. Extraction: a member call on a LITERAL receiver (string, number,
collection, regex — across grammars) emits no call ref at all. A
literal's methods are the language's builtins, never project
symbols; the bare-name fallback let them exact-match any same-named
project function. Silent miss, never a wrong edge.
2. Resolution: matchByExactName filters out candidates nested inside a
same-file FUNCTION container unless the ref originates within that
container's line range. Class members (parent is a class-like node),
top-level symbols, and C++ namespace prefixes (no parent node) are
untouched.
requests re-index: byte-identical (813 calls edges). excalidraw: -27
edges, all literal-receiver refs by construction. The issue's repro is
pinned: join has exactly one caller (format_fields), report_missing
has zero project callees.
Fixes#1230
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
