Fresh CODEGRAPH_RESOLVE_PROFILE tables at 2c and 8c on the round-2 build:
the 8c settle stage is 3.6s — the double-buffer absorbs the entire
resolveOne population, superseding §7a.2's core-invariant framing. The
8c cost is writes-under-readers: main-thread deletes+insertEdges run ~3×
slower (+102s) with 4 readonly workers attached — mechanism unproven,
named as the next probe. Two same-day experiments killed by measurement
and reverted: budget-scaled name caches (v1 LRU regressed via
delete+set-per-get churn + GC; v2 mutation-free second-chance cache
landed exactly on baseline — the 11µs exact-match is per-ref floor, not
refetch overhead) and lazy candidates JSON (read stage flat). New 2c
record 16.5min clean-host; 8c range 15.0–16.4min across n=2 — ranges,
never single runs. Round-2 deferral cuts did move 8c parse (202.6→178.5s).
Levers re-ranked: writes-under-readers probe > cFnPtr native site
extraction (~230s of synthesis) > backpressure byte volume > recreate.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Census-driven cut of the top-ranked post-R7a lever. All passes TS-side,
C-only (preParseCSource), shared by both arms:
- parameterized-annotation whole-blank (__free/__printf/__counted_by/
__bpf_md_ptr…; extends through a stranded field `;`)
- type-keyword-arg scanner (kzalloc_obj(struct T), list_entry, multi-line
continuations behind nested-paren args; bounded hand scanner, head
exclusions + call-vs-declaration guard; blanks trailing stars)
- static/extern CAPS-macro declaration lines at any scope; the initialized
form is REWRITTEN to its expansion (name/tail keep exact offsets)
- va_arg qualified-type blank; GNU named-variadic #define dots-only blank
(post-restore); sandwiched notrace-family; C23 auto; multi-line
iterator-macro spans (hlist_for_each_entry_rcu + lockdep arg)
- word list += cacheline family (2- and 4-underscore spellings) + 10 more
census-confirmed annotations
Gates: five-repo parity sweeps 0 diffs (git deferral 16.1→12.2%, redis
25.3→24.1%, fmt/protobuf unchanged); linux full-tree both arms
2,049,153 nodes / 6,413,518 edges (+858/+6,585 vs R7a) with byte-identical
dumps (10,446,478 lines, sha256 6dd1185b); kernel-arm parse-loop 356→306s
at 2c; suite 2517 green under CODEGRAPH_KERNEL_EXPECT=1. Honesty note
recorded in the docs: error recovery was already salvaging most SYMBOLS on
deferred files — the graph win is relationships + phantom cleanup, and the
unreleased CHANGELOG entry was rewritten off the sweep-subset framing.
Also records §7a.5: post-R7a 8-core cg1212 re-run 16.4min (was 18.3min);
8c parse sits on the single-writer floor, so the <10min-on-8c gap re-ranks
to the per-ref resolution path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps
byte-identical on all five + linux at kernel scale (10.4M dump lines,
same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs
wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new
blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x).
Deferral guard corrected by measurement (C/C++ error incidence 9-42%;
--max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse
cost deferred files paid.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
tree-sitter-wasms shipped 2023-era C/C++ grammars; the kernel walker must
compile the crates.io versions, so production wasm upgrades FIRST and in
isolation (the R2 pattern). Built with tree-sitter-cli 0.25.10 from each
tag's CHECKED-IN parser.c: tree-sitter-c v0.24.2 (b780e47, parser.c
f2883ff9), tree-sitter-cpp v0.23.4 (f41e1a0, parser.c 2a35a43b, scanner.c
cf60387d) — sha-matched against the crates.io tarballs. .metal/.cu map to
language 'cpp' so the dialects ride the same coherent grammar.
Full suite green with the upgraded grammars (2,490; one unrelated daemon
idle-timeout flake passed solo 9/9) — incl. the UE-macro, misparse-guard,
Metal, and CUDA coverage. Checklist updated with the vendored revs and the
metal/cuda routing correction.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Step 1 of the §0a recipe for the C/C++ port: every tree-sitter.ts branch,
extractor-config field, name-salvage helper, and gate the walker must
mirror, with file:line anchors. Locks the architecture: preParse (all seven
blanking passes) hoists to the kernel route point so none of it ports to
Rust; Metal/CUDA stay on wasm this round; one dual-language walker module.
Key asymmetries surveyed: value-refs are C-only (cpp absent from
VALUE_REF_LANGS), static-member refs are cpp-only, fn-ref spec is
cFamilySpec with addressOfOnly for cpp.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cFnPtrEdges was 86% of kernel-scale synthesis (306s, §7a.3). Standalone
probe iterations against the live kernel DB attributed and fixed:
- sliceLines split the ENTIRE file per node (~1.6M full-file splits across
the B/D/E sweeps) — split once per file, slice from the line array
(D 46.2->20.5s, E 94.5->69.1s).
- The 128-entry strip cache re-stripped every file 4.4x across the four
file sweeps (71.8s). Sizing is now memory-budget-aware and ALL-OR-NOTHING
with 5% slack over the file count: a partial LRU on cyclic sweeps thrashes
to ~0% hit (measured twice: cap ~61k and cap == files both thrashed
against 63.8k + includes). Full-cache strips: exactly 1.0/file, 15.5s.
- Pass D pre-gates field names on fieldToStructs before the two regex type
resolutions — a->f = b->g matches every struct-field assignment in the
tree, ~99% of them data fields.
- recvTypeIn/varTypeIn compiled a new RegExp per call - cached per name.
- Line numbers counted incrementally per match (ascending indexes) instead
of splitting the body prefix per match; cursor rewinds between the two
dispatch scans.
- CODEGRAPH_SYNTH_TIMINGS now prints per-sweep walls + read/strip/nodes
accounting for this pass.
278.8s -> 134.8s standalone at the 2c envelope. Output identity proven at
full scale: probe edge set (merge-dedup applied, canonical sort) SHA256
21c2a971... == the original code's 274,762 edges extracted from the live
kernel DB. Pass suite 15/15.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The §7a.2 per-ref profile overturned the assumption the whole arc was
built on: resolveOne owns only ~93s of the kernel-scale ~433s batch loop.
Loop-stage attribution (CODEGRAPH_RESOLVE_PROFILE, shipped here) named the
rest: backpressure folds 111.2s, count guard 93.9s, batch reads 54.6s,
deletes/inserts/marks ~84s, settle 85.7s.
- Non-progress guard O(remaining)→O(1): the per-batch COUNT(*) walked every
remaining pending row (O(N²/batch) per run, 93.9s). The cleanup queries
now return summed SQLite , and zero-removals-from-claimed-work
is the guard signal — the DIRECT evidence the count diff inferred (a
mismatched-name resolver makes keyed cleanup no-op ⇒ changes=0). A real
COUNT runs only on that suspicious path and arbitrates exactly as before.
- Batch reads OFFSET→keyset (54.6s→O(batch)): OFFSET re-walked the
accumulated failed-row prefix every read; seeking past the last-seen
rowid is prefix-independent and enumeration-order identical.
- WAL valve caps scale with DB size (env still wins): every fold re-writes
hot pages (#1231 in bounded form — 111.2s at the flat 256MB cap);
soft=clamp(dbSize/4, 256MB, 2GB) trades ~4× fewer folds for a transient
WAL ≈ project size.
- CODEGRAPH_RESOLVE_PROFILE: per-outcome resolveOne histogram + loop-stage
attribution, main + workers, off by default.
Gates: dubbo dump byte-identical; suite 2,491 passed / 4 skipped (kernel
required). Kernel-scale payoff run lands in the plan doc next.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A truncate checkpoint started against an ACTIVE writer wins the lock and
then blocks that writer for its entire backfill; after the edge-index
recreate's multi-GB single-transaction burst that exceeds the writer's 5s
busy_timeout and fails the index with 'database is locked' (§7a.2 record
run, EXIT=1 at kernel scale — the small mid-resolution truncates folded in
ms and masked the hazard). Barrier truncates (backpressure/foldNow) are
collision-free by construction: the writer is awaiting the valve.
Dubbo gate: exit 0, peak 81MB (barrier folds carry containment), dump
byte-identical. Valve + sizing suites 27/27.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four §7a.1 instrumented-run findings, each measured:
1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still
grows the FILE without bound — the writer only restarts at frame 0 when a
commit finds zero reader marks, which the instrumented run showed never
happens (file marched 361→721MB through two COMPLETE backfills; 22GB by
phase end). backpressure() now also trips at 4× the soft cap on raw file
size and TRUNCATEs at the parked barrier; the timer path truncates
opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at
the same 16MB valve; dumps byte-identical under aggressive folding.
2. cgroup memory credit: memory.current counts reclaimable page cache — a
post-parse container read 57MB of headroom on a 6GB box and silently
disabled the pool. inactive_file is credited back (the docker-stats
working-set convention); the same run now reads a sane 4.4GB budget.
3. Pool at 2 cores reversed: sequential resolution measured FASTER than
pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is
Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing.
cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path.
4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower
(493s vs 369s) — main + store-worker don't fill the second core. Floor
restores the baseline (373.5s measured).
Plus the observability §7a.1 burned three 25-minute cycles for: valve
armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture,
pool sizing decisions (incl. the disabled path), backpressure-hook
presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG.
Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record
runs with this build follow in the migration plan §7a.1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three §7a.1 run-1 lessons (kernel-scale 2c/6GB: EXIT=137, WAL 22.2GB with
the backpressure hook DEPLOYED):
1. TRUNCATE at parked barriers: a completed passive backfill bounds the
un-checkpointed backlog but the FILE only stops growing when a commit
finds zero readers holding WAL marks — rare while pool workers cycle
(dubbo debug baseline: file climbed monotonically through six completed
pass-1 backfills). At a parked barrier the no-reader window is
guaranteed, so chop the file there with wal_checkpoint(TRUNCATE)
(off-thread, 2s busy_timeout — a racing reader degrades it to a no-op).
2. Futility latch: when backfill gives up (pinned reader), parking again at
every over-cap boundary burns a 20-pass checkpoint attempt — each a
worker thread + fresh connection against a multi-GB DB — per batch. Two
consecutive give-ups now disable parking for 60s; a pinned phase degrades
to pre-valve behavior instead of OOM-amplifying.
3. CODEGRAPH_WAL_VALVE_DEBUG=1 surfaces valve decisions without the
caller's verbose plumbing, and give-up lines print under
CODEGRAPH_SYNTH_TIMINGS — run 1 failed silently because give-ups were
verbose-gated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Pool sizing used os.cpus().length, which enumerates the HOST's CPUs: inside
a 2-CPU cpuset it sized 6 resolver workers (the §7a.1 false-'sequential'
premise) and 8 parse workers, and at true 8-core concurrency six ~1GB
workers OOM-killed a 7GB container (oom_kill=5) mid-synthesis — sizing had
no memory term and no override knob.
resolvePoolSize (pure, matrix-tested): explicit CODEGRAPH_RESOLVE_WORKERS
override (0 disables, cap 16); CPU term max(2, min(availableParallelism-1,
6)) — cpuset-honest, floored at 2 so true 2-core boxes keep pooled
synthesis's ~2×; memory term floor(budget*0.7 / clamp(0.2*dbSize, 256MB,
1.5GB)) with budget = min(freemem, cgroup v2/v1 headroom). Parse pool's
core input switches to availableParallelism. Dev machines are unchanged
(still 6 workers); the 8c/7GB kernel-scale container now sizes 4.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
At kernel scale the pooled resolution/synthesis superphase grew a 22GB WAL
on a 4.6GB DB (cg1212, §7a.1): autocheckpointing is deferred for the run,
and the valve's timer-driven passive checkpoints stay perpetually partial
against the pool's continuous reads — no mechanism ever completed a
backfill, so the WAL accreted the whole phase's write volume, blowing disk
and feeding page-cache pressure into the 8-core/7GB container OOM.
The valve's writer-side backpressure() hard-cap backstop existed but was
wired only into the PARSE orchestrator. Thread it into the resolution batch
loop at the double-buffer's one pool-idle boundary (batch settled, next not
yet fanned out), after the edge-index recreate, and through the synthesis
insert loops. Parked there, the backfill completes; readers re-enter at
SQLite's backfilled mark and the next persist commit wraps the WAL.
Dubbo validation, same build: valve@16MB peak WAL 251MB (floor = the
single-transaction edge-index recreate) vs defaults 914MB; dumps
byte-identical (441,270 rows); wall unchanged (11s). Suite 2,479 green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
On CRLF checkouts (every Windows autocrlf clone) the JS reference's
block-continuation strip /^\s*\*\s?/gm finds a line start after the \r and
its greedy \s* consumes the \n, leaving a bare \r in the docstring; the
kernel's (?m)^ pass matched after \n only and kept \r\n. Caught by the O2
Windows VM leg (6 kernel-tsjs-parity failures), reproduced on macOS by
CRLF-converting the fixtures.
js_multiline_strip now replicates the JS anchor set (\n, \r, U+2028, U+2029)
for all five line-marker passes; CRLF variants of every torture fixture are
pinned in kernel-tsjs-parity, derived in-memory so nothing can normalize
them away.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R1-R6 are done; §0 is now the open-work list in recommended order
(merge → Windows VM leg → P1 resolution → C/C++ port → long tail), and
§0a carries everything a fresh session needs: where the work lives, the
build/gate commands, the proven add-a-language recipe, and the paid-for
traps (encoding-dependent error recovery → defer policy, node-ID-string
dedupe, UTF-16 positions/slices, the exact-seam contract, crate+wasm
grammar lockstep). §1/§6 keep the original expectations with SUPERSEDED
annotations pointing at the measurements that corrected them; §7a now
carries the R6 numbers that make it the top open perf item.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cg1212 (Linux kernel, 2 CPU/6GB): completes in 26.4min vs the ~27min
baseline with all new machinery active — no regression; graph scale
identical (2,048,664 nodes / 6,405,964 edges). The §6 parse expectation
(6m → ~2m) was mis-premised: the tree is ~99% C, an unported T2
language, so it transfers to the C/C++ port (R7). Resolution remains
the kernel-scale wall (19.2m, 73% — P1). The tree's own Python tooling
files: 99/99 byte-parity. kernel-parity.mjs now skips dangling
symlinks (Linux dtc fixtures).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Python (codegraph-kernel/src/python.rs) and Go (src/go.rs) join the
native kernel, mirroring the wasm extractors bug-for-bug. Python:
decorated_definition docstrings/decorators (decorates refs only for
bare-identifier decorators — the call-kind quirk), function-in-class →
method, module-level assignments always extract as variable, from-import
per-name binding refs, self.x fn-ref candidates as bare names. Go:
receiver methods with Recv::name qualified names + contains edges to the
first earlier struct of that name, type_spec struct/interface
classification with embedding→extends and interface method nodes,
composite-literal instantiates keeping the package qualifier, top-level
var/const initializer walks attributed to the declared symbol (#693),
2-hop field chains (#1276), New().Method() re-encode (#645/#608), and
the GO_SPEC fn-ref layers.
Grammars: tree-sitter-python 0.23.6 + tree-sitter-go 0.23.4 crates, with
wasm vendored from the same tags (parser.c sha-matched) — both were
2023-era in tree-sitter-wasms.
Gates: extraction sweeps 100% (flask 83/83, django 3,035/3,038 +3
error-file deferrals, gin 99/99, prometheus 978/979 +1); full-init
dump-diffs byte-identical on flask (10,833 rows), gin (17,540), django
(360,794), and prometheus (213,758); torture fixtures enforced in npm
test. DEFAULT_ROUTED now covers typescript/tsx/javascript/jsx/java/
python/go. Full suite: 2,471 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kernel-routed files ship their flat tables from the parse worker to the
store worker as buffers (tryKernelExtractRaw → kernelBuffers on the
result → KernelStoreBundle); the store worker decodes and finalizes
(finalizeStoreBundle shared with the object path so filter semantics
can never drift). Files with applicable framework extract() hooks keep
the decoded path; the no-writer fallback materializes via
materializeKernelResult. Byte-identical dumps re-verified on dubbo,
excalidraw, express, gson; full suite green (2,467).
Measurement (plan §4d): dubbo's parse-loop wall is 94% store-writer
busy time — the many-core fresh-index wall is single-writer SQLite
ingest, not extraction or main-thread work. d2s improves the writer
lane ~11% (structured-clone deserialization avoided on the writer) and
frees the main thread; the remaining many-core gap is a
store-architecture arc (deferred index builds, multi-file
transactions, buffer→bind), out of the kernel project's scope.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring
the wasm extractor's Java paths bug-for-bug: package namespaces,
imports, javadoc, annotations→decorates, type_list inheritance,
static-final constants, enum constants, anonymous classes (including
the TS side's 0-based-line quirk on the extends ref), method_invocation
calls with the this.field unwrap and the Foo.getInstance().bar() chain
encoding (#645/#608), static-member value reads, method-reference
fn-refs (#756), value-reference edges, and the full Lombok member
synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/
EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared
docstring/textutil modules moved to crate level. Grammar:
tree-sitter-java 0.23.5, with the wasm grammar vendored from the same
tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build.
Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit
341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test;
full-init dump-diffs byte-identical on gson (49,766 rows), retrofit
(62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux
container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1.
The gate caught a real cross-language bug: fn-ref dedupe and value-ref
self-target checks must compare node ID STRINGS, not node-table rows —
ids collide for same-(kind, name, line) nodes, which minified one-line
bundles hit routinely (retrofit's website JS exposed it; latent in the
TS/JS walker since R2, never released). Fixed in both walkers.
Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat
(parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that
wall is main-thread-bound (reads + store), not worker-CPU-bound — the
§6 expectation assumed otherwise. Where worker CPU binds the kernel
delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s
(~1.25×). The identified lever for the many-core headline is decoding
kernel buffers directly into store rows (skipping per-node JS object
materialization); the buffer contract already carries everything.
DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate evidence (docs/design/rust-kernel-migration-plan.md §4b):
- Graph parity, byte-identical (stronger than the §5 ≤0.5% bar): full
codegraph-init dump-diffs kernel-vs-wasm on express (13,712 rows),
excalidraw (89,898), and vscode (2,378,238 rows) — identical bytes.
Python control repo (flask) identical + timing unchanged. The parity
harness is now ORDER-sensitive (emission order drives rowids, which
drive resolution order) and dumps come from the new
scripts/dump-graph.mjs (natural keys, no rowids/timestamps).
- The one real find, caught by the vscode tier: tree-sitter error
RECOVERY is encoding-dependent — byte-identical grammar sources and
the same core (0.25.10) recover erroring files differently under
UTF-8 (native) vs UTF-16 (web-tree-sitter) parsing; proven by
reproducing the wasm tree with a native UTF-16 parse. Policy: the
kernel defers any file whose tree has_error() to the wasm extractor
(silent 'defer:' signal, per file) — parity by construction on
erroring files (incidence 0-0.42% across the gate repos), and the
harness fails if deferrals exceed 10% so a broken kernel can't hide
behind the fallback.
- Retrieval invariants: canonical excalidraw flow (mutateElement →
renderStaticScene) connects end-to-end on the kernel-indexed graph;
synthesized-edge families present. Agent A/B is vacuous under
byte-identical DBs (same justification as #1320-#1322).
- Perf: vscode init 105.4s → 82.1s (1.28×) on an 11-core Mac;
excalidraw on a 2-CPU/6GB Linux container (the CI-runner envelope)
6.2-7.1s → 4.3-4.8s (~1.5×). Linux arm64 in-container build: all 22
kernel tests green under CODEGRAPH_KERNEL_EXPECT=1. Windows VM leg
deferred (VM stopped; prlctl start needs Parallels Pro) — benign: a
missing .node falls back to wasm, and the release matrix builds and
gates the win32 prebuilds.
- Full suite: 2,465 tests pass WITH default-on routing, so the entire
extraction corpus now exercises the kernel for TS/JS wherever a
.node is staged.
DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}. Override:
CODEGRAPH_KERNEL_LANGS (replaces the set) / CODEGRAPH_KERNEL=0 (kill).
Changelog entry added under [Unreleased].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the R1 seed .scm emitter with a bespoke Rust walker
(codegraph-kernel/src/tsjs/) that mirrors TreeSitterExtractor's TS/JS
paths function-for-function: declarations (incl. #808 field/property
classification), qualified names, docstrings (#780 wrapper climbs),
signatures, imports/re-exports + per-binding refs, calls with
receiver-qualified callees (#1230 literal-receiver skip), instantiations,
decorators, inheritance, type annotations (#381), type-alias members +
tuple contracts (#359/#634), React component recognition (#841
forwardRef/memo/styled), object-of-functions / zustand-through-middleware
/ RTK Query endpoints + generated hooks / vuex + pinia store shapes,
function-as-value capture with the flush gate (#756), and value-reference
edges with the shadow prune (#895/#897). The generic query emitter is
deleted — extraction parity needs logic .scm can't express; future
languages get walkers too (migration plan §4a).
Positions and JS string-slice semantics are emitted in UTF-16 code units
natively, so kernel output is byte-identical to web-tree-sitter's — no
column diff class exists.
Parity evidence (macOS): scripts/kernel-parity.mjs (full-object multiset
diff per file) — this repo 353/353 files, excalidraw 643/643 (10,650
nodes / 10,726 edges / 68,307 refs), plus torture fixtures checked into
__tests__/fixtures/kernel-parity/ and enforced in npm test by
kernel-tsjs-parity.test.ts. The strict compare caught one real decoder
bug the loose harness missed: refs must NOT carry denormalized
filePath/language at the extractFromSource seam (the store fills them).
Perf: extraction 2.6× single-thread on excalidraw (487ms vs 1,255ms,
identical outputs). Routing stays opt-in (CODEGRAPH_KERNEL_LANGS) until
the R3 equivalence gate (large repo, DB dump-diff, retrieval invariants,
agent A/B, Linux/Windows) passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 0 of the Rust extraction-kernel migration (docs/design/
rust-kernel-migration-plan.md, now checked in with §3a recording the
shipped state):
- codegraph-kernel/ napi-rs crate: extractFile(path, content, language)
→ five flat buffers (meta/nodes/edges/refs/arena), one JS boundary
crossing per file. Node ids computed Rust-side, byte-identical to
generateNodeId (pinned by test vector). Reserved per-node metrics slot
for the Arc 3.2 code-metrics work.
- Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures,
byte-range scope stack → ::-joined qualified names, contains edges,
refs attributed to the innermost enclosing symbol). Seed TS/JS queries
are smoke-level; R2 replaces them with the full port.
- Routing seam in extractFromSource with per-file wasm fallback.
DEFAULT_ROUTED is empty — no behavior change until a language passes
its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill
switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before
routing; EDGE_KINDS became a runtime array because kind order is now
wire contract.
- Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the
exact crate revisions (tree-sitter-typescript v0.23.2,
tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) —
the tree-sitter-wasms builds were 2023-era, which the new
kernel-grammar-parity test caught on day one. Production TS/JS parsing
gets 2.5 years of grammar fixes; full suite green (2456 tests).
- Build/release wiring: scripts/build-kernel.sh + npm run build:kernel;
release.yml kernel prebuild matrix (continue-on-error — the kernel is
optional everywhere, bundles fall back to the wasm path); bundles stage
lib/kernel/codegraph-kernel.node; release job runs the kernel suites
with CODEGRAPH_KERNEL_EXPECT=1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two hazards found by running today's full stack against the Linux kernel
(70,129 files) in the cg1212 repro container:
1. The parallel-synthesis fallback retried a worker-failed pass on the MAIN
thread. At multi-million-node scale a worker failure is usually a memory
ceiling, so the retry would OOM the process and take the whole index with
it. Above 1.5M nodes a failed pass is now skipped with a clear stderr
message (its synthesized edges are absent; the index completes). Below
that, the main-thread retry stays — small-scale worker crashes are
transient and the retry keeps coverage.
2. endBulkEdgeLoad rebuilt all four edge indexes in one synchronous span —
measured 79s at kernel scale, past the #850 liveness watchdog's 60s
stall window. A daemon-triggered re-index would have been SIGKILLed right
after doing the work. Now async with an event-loop yield between builds,
keeping each stall to a single index (~20s at kernel scale).
Validation: full Linux kernel index to completion in the repro container —
2,048,674 nodes / 6,405,964 edges, EXIT 0, zero passes skipped, on a 2-CPU
VM (worst case: pool disabled, sequential resolution + synthesis) in ~27min.
Phase walls: parse 6.0m, resolution 19.5m (incl. synthesis 6.3m, recreate
79s), maintenance 74s. Suite green (2444).
Also adds docs/design/native-extraction-kernel.md — the spike-validated
design for the native extraction kernel (Rust parse+walk over dubbo's Java:
202ms rayon / 1.07s single-thread vs 4.7s for the current wasm pipeline).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The resolution persist's measured cost is B-tree maintenance on the edges
table's five indexes (offline replay of a 224k-edge resolution set: 2.8s with
all indexes, 1.1s with only the unique identity index, +0.3s to recreate the
rest). On big runs (same >=150k-ref gate as the resolver pool) the four
non-unique edge indexes are now dropped for the batch loop and recreated in
one pass each before synthesis.
Why this is safe:
- idx_edges_identity stays: INSERT OR IGNORE's dedup conflicts on it (#1034),
and its leftmost column is `source`, so the only mid-window edge reads —
resolution's supertype walks (implements/extends by source) — keep an index
via its prefix (verified with EXPLAIN QUERY PLAN).
- The window closes BEFORE synthesis, whose passes read kind-keyed, and on
every error path (finally).
- A crash inside the window heals on the next DatabaseConnection open —
schema.sql re-applies CREATE INDEX IF NOT EXISTS, same recovery as the FTS
bulk-load pattern this mirrors.
- Concurrent readers (a daemon serving the project mid-index) stay correct;
target/kind-keyed reads degrade to scans only for the window's duration.
dubbo (4,402 files): persists 4.0s -> 3.0s, fresh init 11.9s -> ~11.1s,
graph byte-identical. excalidraw (below the gate): untouched, byte-identical.
Recreation cost ~250ms, logged under CODEGRAPH_SYNTH_TIMINGS as
edge-index-recreate. Suite green (2444).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The ~36 independent synthesis passes (callback/event/framework wiring) ran
sequentially on the indexer's main thread — 2.0s of a 4,402-file Java repo's
index, and the stage where kernel-class repos die (#1212). They now live in
an explicit registry (SYNTH_PASSES) and, when the resolver pool is alive
(>=150k-ref repos), fan out across its read-only workers: dubbo synthesis
2,024ms -> ~900ms (-55%), total fresh init 13.5s -> 11.9s. Graphs verified
byte-for-byte identical on both the pool path (dubbo) and the sequential
path (excalidraw).
Why this is safe: no pass's edges persist until the ordered merge, so every
pass sees the same committed post-resolution DB state in either mode, and
results merge in registry order regardless of completion order — the
first-seen dedup is unchanged. The pool now survives through synthesis
(destroy moved after it) instead of being torn down moments before the one
stage that could reuse it.
Robustness: a pass that fails on a worker (crash, OOM) is retried on the
main thread — a synthesizer blow-up now costs one worker instead of the
whole index, which is half the #1212 story on very large repos.
Also: ref-row cleanup deletes now run as one transaction with a cached
statement instead of one implicit commit per 500-row chunk (mechanically
fewer WAL commits; matters most on HDD-class storage). A set-based rewrite
of failed-ref parking was tried, measured ~zero on NVMe, and dropped — the
remaining persist cost is edge-index B-tree maintenance, not statement
dispatch.
SYNTH_PROGRESS_STEPS now derives from the registry (passes + fixed marks);
the pin test counts registry entries plus literal __mark sites.
Suite green (2444). Sequential-path timing unchanged on excalidraw.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fresh init on a 4,402-file Java repo (dubbo): 18.2s -> 13.5s (-26%), with
the resolution phase going 12.6s -> 7.9s (-38%). Graphs verified
byte-identical on both the pool path (dubbo) and the sequential path
(excalidraw).
Two changes:
1. The fastInit+pool path restored WAL for the resolver workers but left
wal_autocheckpoint at its default, so the persist loop inline-checkpointed
hot pages all phase long (#1231's pathology inside resolution — measured
at 58% of resolution wall). Checkpointing is now deferred behind the
bounded valve and folded once at maintenance, mirroring the deferWal path.
2. The resolution loop is double-buffered: batch k+1 is prefetched (OFFSET
past batch k's still-pending rows, under an explicit ORDER BY rowid) and
fanned out across the pool while batch k's ref cleanup runs on the main
thread. Batch settle-waits dropped 2572ms -> 117ms.
Correctness invariant found by the byte-identical gate and now documented in
the loop: batch k+1's resolution READS batch k's edges (resolveMethodOnType
walks supertype chains over extends/implements edges that resolution itself
inserts), so edges must persist BEFORE the next batch fans out; only the ref
cleanup overlaps.
Also extends the CODEGRAPH_SYNTH_TIMINGS instrumentation with phase labels
(grammar-init, parse-loop, fts-rebuild, resolver-reinit, resolution,
callback-synthesis) and pool timings (worker open/resolve, per-batch mode,
persist), so the next profile is one env var away.
Suite green (2444). Sync path timings unchanged. Pool floor re-validated:
forced-on at 40k refs is still net-slower, so the 150k threshold stands.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three compounding defects (#1196) made a query bag of object-literal
keys (`profileInfo isTrialEligible quotaInfo billingMethod`) return
unrelated results while the defining files never surfaced:
1. Step 5b title-cased interior humps (profileInfo -> Profileinfo) and
then compared case-SENSITIVELY, dropping every row SQLite's
case-insensitive LIKE had just recovered. The hump lookup is now
case-insensitive with an explicit uppercase-at-match requirement.
2. Step 5b/5c's kind whitelist held only type-like kinds — dead code on
method-centric codebases. Callable kinds (function/method/component)
are fetched as a SEPARATE LIKE batch so hot single-word terms can't
crowd classes out of the length-ordered 200-row batch.
3. explore's named-symbol seeding was exact-name only; a field token
seeded nothing. A camelCase token with ZERO exact defs now seeds its
camel-infix definers (callables, hump-boundary or prefix, shortest
first, capped at 3) — bare lowercase words keep the #1252 stopword
guard untouched.
The reporter's acceptance query is a pinned e2e test (definer files
present, exact-name seeding unaffected). excalidraw probe: the
canonical flow query (mutateElement renderStaticScene) is byte-
identical; NL queries shift toward more-central callables
(useUIAppState/getDefaultAppState over observer periphery).
Fixes#1196
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`.gitignore: /repos/` lists `repos/` as ONE ignored entry, while the
CLI hint (#1156) suggests `includeIgnored: ["repos/a/", "repos/b/"]` —
the child spelling. findIgnoredEmbeddedRepos tested the opt-in matcher
against the PARENT path only, which a child pattern never matches, so
the documented opt-in silently indexed nothing and init looped the
byte-identical suggestion back at the user (#1295).
Ignored dirs that don't match as a whole are now descended (the walk
was already bounded: depth 4 / 2000 entries, and only runs when
includeIgnored is configured) and each nested repo root is matched
individually — parent spelling opts in everything under the dir, child
spelling exactly the named repos. findUnindexedIgnoredRepos gets the
same per-repo check so the hint stops nagging about repos that are
already configured while still naming unopted siblings.
Fixes#1295
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
", ".join(sorted(x)) resolved by bare name to a project function named
join — one nested inside a DIFFERENT function, so scope alone rules the
edge out. Both defects from #1230, fixed independently:
1. Extraction: a member call on a LITERAL receiver (string, number,
collection, regex — across grammars) emits no call ref at all. A
literal's methods are the language's builtins, never project
symbols; the bare-name fallback let them exact-match any same-named
project function. Silent miss, never a wrong edge.
2. Resolution: matchByExactName filters out candidates nested inside a
same-file FUNCTION container unless the ref originates within that
container's line range. Class members (parent is a class-like node),
top-level symbols, and C++ namespace prefixes (no parent node) are
untouched.
requests re-index: byte-identical (813 calls edges). excalidraw: -27
edges, all literal-receiver refs by construction. The issue's repro is
pinned: join has exactly one caller (format_fields), report_missing
has zero project callees.
Fixes#1230
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
target.conn.Exec("insert") with `conn *sql.DB` emitted a BARE `Exec`
ref (the receiver chain was dropped for non-identifier receivers), and
exact-match then bound it to the only local `Exec` — an unrelated
interface's method — fabricating an internal dependency (#1276).
Extraction now keeps Go 2-hop selector chains (`base.field.Method`),
and a dedicated matcher resolves them EXCLUSIVELY via two inference
hops: base's type from the enclosing scope (#1108 machinery), field's
declared type from the struct's own declaration lines (comment-
stripped, per-line — chi's "the tree router" doc comment otherwise
donates a phantom type). resolveMethodOnType validates the target.
Package-qualified field types are followed only when the package is
in-module — `handler http.Handler` must not bind a same-named local
decoy. Failure at any hop leaves the ref unresolved: chained Go
receivers never fall through to the bare-name strategies (they were
never emitted before, so no prior recall depends on that path).
chi before/after: node count stable (1,181); 8 correct field-chain
edges gained (mx.tree.FindRoute/InsertRoute/routes, validated,
including the unexported `node` type); the removed edges are the
prior bare-name guesses on external receivers.
Fixes#1276
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
reproStore.notifyJoinGuildStatus() after `import { reproStore }`
resolved its calls edge to the exported CONSTANT (resolvedBy:'import'),
while the identical same-file call resolved to the method via
local-variable receiver inference (#1108) — so `callers <method>`
missed every cross-file use and a widely-used method could look
unused (#1292).
resolveViaImport's member-descend now handles imported VALUES alongside
the #825 static-member case: when the base resolves to a
constant/variable, the value's type is inferred from ITS OWN
declaration lines in the exporting file (the shared #1108 pattern
table: `= new T(...)` initializers and type annotations) and the member
is resolved AND VALIDATED on that type via resolveMethodOnType. A
failed inference or validation keeps the existing constant edge —
never a fabricated one. Calls only; plain member reads still reference
the value.
excalidraw control: byte-identical graph (10,653 nodes / 19,483 calls
edges before and after).
Fixes#1292
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The CLI's bare-symbol branch passes includeCode=true to the
codegraph_node handler, but the symbol-pinned-to-file branch didn't —
so exactly when a user disambiguated an overloaded name to one file
(the point of -f), they got Location + trail with no code (#1284).
Fixes#1284
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A directory deletion arrives as ONE event on the directory's own path.
That path has no source extension, so handleChange dropped it at the
isSourceFile gate before ever scheduling a sync — and the files inside
may never get events of their own (Windows's recursive watcher reports
only the top-most removed entry; FSEvents can coalesce a tree deletion
the same way). Every child record then sat stale in the index until an
unrelated edit happened to trigger a sync (#1285).
A non-source path that no longer EXISTS on disk now schedules the
debounced sync; the sync's scan-diff removes whatever vanished (already
correct — verified: manual `codegraph sync` cascades fine). Events for
live non-source files stay fully ignored, so build churn schedules
nothing.
Fixes#1285
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The #1242 fix (WAL deferral + checkpoint valve, the 26x win on
HDD-class storage) was wired only into indexAll. CodeGraph.sync never
touched wal_autocheckpoint, so every incremental run kept the default
1000-page cadence and re-triggered the #1231 per-page checkpoint
thrash — a 7-file sync took 2m 2s at 0-2% CPU on the reporter's
hardware, because the cost scales with the EXISTING database's hot
pages, not the change size.
sync now mirrors indexAll exactly: defer autocheckpoint + start the
valve for the run, fold the store phase's WAL before the post-store
reads, restore the interval in the finally. Same kill switch
(CODEGRAPH_NO_WAL_DEFER=1). Idle valve cost is one timer, so
watcher-frequency syncs stay cheap.
Fixes#1248
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
SEC_ATTR UINT32 LostName(VOID) — an unknown attribute macro before a
typedef'd return type — misparses in tree-sitter's C grammar: the macro
becomes the type, the return type the declarator, and the PARAMETER
LIST is stored as the function name ("(VOID)"). The C++ grammar
recovers this shape via recoverMangledCppName, but in C the real name
never reaches the mangled string, so only a pre-parse blank can help.
Attribute macros are project-specific, so the blank keys on structure:
line-leading ALL-CAPS token followed by TWO identifiers then `(` — the
`MACRO Ret name(` definition shape. Plain typedef'd returns, ALL-CAPS
calls, #define lines, multi-word builtin returns, and mid-line uses are
all rejected by construction. Offset-preserving like the C++ blanks.
curl re-index: 5,531 C functions before and after, zero name changes;
7 nodes in memdebug.c improve start-line accuracy by 1 (the macro line
no longer counts as part of the definition).
Fixes#1211
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An out-of-line member definition inside a namespace block takes its
qualifiedName from the declarator's receiver, which is spelled RELATIVE
to the enclosing namespace — so `namespace simulator {
ManifestStartup::Output ManifestStartup::Apply(...) {} }` indexed as
ManifestStartup::Apply while the class node carried
simulator::ManifestStartup. Fully-qualified call sites
(simulator::ManifestStartup::Apply(...)) never resolved; callers and
file impact came up empty (#1291).
The receiver-based qualifiedName now composes the active namespace
prefix, anchored at the first prefix segment the receiver re-spells
(so `namespace sim { void sim::M::f() {} }` doesn't double-prefix).
namespacePrefix is only ever non-empty for C++ — Go/Rust/Kotlin/Lua
receivers pass through unchanged.
leveldb re-index: node count byte-stable (3,044), calls edges +6,
namespace-qualified method names 947 -> 1,252.
Fixes#1291
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
template<typename T> T Box<T>::get() stored qualified_name Box<T>::get —
the <T> qualifier never matched the class node indexed as Box, so the
method didn't link to its class, while the inline form of the same
method produced Box::get. ICU-shaped multi-line template parameter
lists leaked whole <…> blocks (newlines included) into qualified_name,
exceeding NAME_MAX for downstream consumers.
extractCppReceiverType now applies stripCppTemplateArgs (the #1043
normalization for base-class refs) to the receiver qualifier.
fmt re-index: template-arg-in-qualifier names 25 -> 4 (remaining are a
FMT_BEGIN_EXPORT misparse artifact and gmock conversion-operator names,
both distinct pre-existing shapes), node count byte-stable at 7,536.
Fixes#1286
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cache.Put("a", 1), store.Get("config", out), bus.Handle("user.created",
h) — any verb-named method with a string first arg — were indexed as
HTTP routes (38 of 82 route nodes were false positives on the
reporter's 200 KLOC Go codebase). A registration's first argument must
now start with "/" (every router style), or be a Go 1.22
"METHOD /path" mux pattern on Handle/HandleFunc — which now also
extracts the real method instead of ANY.
Validated on go-chi/chi (212 real routes retained, all path-shaped)
and golang/groupcache (0 route nodes).
Fixes#1259
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
codegraph's glyphs were ASCII on every Windows console while
@clack/prompts drew its Unicode frame around them, so one index block
mixed `|` and `│` rails (#398). supportsUnicode() now mirrors the
is-unicode-supported detection clack bundles (Windows Terminal, VS
Code, ConEmu/Cmder, Alacritty, xterm-256color, JetBrains, CI), so both
systems always pick the same glyph family.
The shimmer worker's raw fs.writeSync(1) bytes still decode through the
console codepage (OEM codepages mojibake UTF-8 even under Windows
Terminal — the #168 regression to avoid), so:
- the raw path gets its own supportsUnicodeRawWrites() that stays ASCII
on win32 unless CODEGRAPH_UNICODE=1, and
- the persistent "phase done" lines move from the worker to the parent,
written via process.stdout (wide-char console API, codepage-immune) at
phase transitions — the main thread is alive there, it's delivering
the progress callback. Only transient, self-erasing animation frames
remain on the raw path, so ASCII never lands in scrollback.
Fixes#398
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
List commands (status, query, callers, callees, impact, files) embedded
ANSI color codes even when stdout was a pipe, and NO_COLOR had no effect.
One switch now decides color for all codegraph-authored output:
--no-color > --color > NO_COLOR > FORCE_COLOR > stdout TTY > CI.
Piped init/index/sync also stop emitting shimmer animation frames
(\r + erase-line rewrites) and print one plain line per phase instead;
a TTY with NO_COLOR keeps the animation but drops the color codes.
The detection mirrors picocolors' so @clack frames and our own lines
agree within a run.
Fixes#1281
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* perf(index): ~34% faster fresh indexing, byte-identical graphs
Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed
the main thread as the critical path: per-row SQLite statement calls,
repeated import-resolution walks, and per-row FTS trigger firings, with
the parse workers ~75% idle behind it. This lands the semantics-preserving
tranche of fixes:
- Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments)
behind cached per-batch-size prepared statements; row order preserved,
so rowid-based resolution determinism (#1015) is unchanged.
- storeFileBundle: one transaction per file instead of four; nested
transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no
caller depended on nested rollback).
- Dedicated store-writer thread for the fresh-DB bulk path (bundles
applied in file order on a single writer connection; main thread does
no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1.
- Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load,
rebuild once at the end; crash inside the window self-heals on the
next open.
- Per-context memos for resolveImportPath/findExportedSymbol + a per-file
exported-symbol index, invalidated exactly where clearCaches() already
resets the resolver's own caches.
- Fast-init on completely fresh DBs (journal in memory, no fsync until
the index completes; interrupted init re-runs from scratch). Kill
switch: CODEGRAPH_NO_FAST_INIT=1.
- MaybeYield returns undefined on the not-due path so per-ref yield
checks stop paying a promise + microtask hop each.
- Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and
worker entry points.
Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series).
Graph dumps byte-identical across init, re-index, and sync paths; full
suite green (2403 passed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perf(resolution): parallel reference resolution with canonical admission
Fan resolution batches across a pool of read-only worker threads, each
hosting a full ReferenceResolver over its own SQLite connection; results
are admitted on the main thread in chunk order, so edge insertion order,
row cleanup, failure parking, and deferred post-pass queues are exactly
the sequence the single-threaded loop produces. Per-ref inputs match the
baseline because the sequential path already resolves each batch against
the state committed BEFORE that batch.
Validated byte-identical on excalidraw (pool forced on) and apache/dubbo
(4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical
graph dumps (91,495 nodes / 223,953 edges).
The pool only engages when total pending refs clear a threshold (default
150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1
to disable): measured on a ~58k-ref repo the workers' boot CPU contends
with resolution on the same cores and makes indexing slower, so small
repos keep the sequential path. When fast-init left the DB in
memory-journal mode, WAL is restored before resolution only when the pool
will run (readers + rollback-journal writers don't mix).
Also: sqlite adapter readOnly open support.
TreeCursor spine rewrite of the body walker was built, measured neutral
on real repos and equal in a 20k-child microbench (web-tree-sitter's
namedChild(i) is not quadratic in this binding), and rejected — per-node
JS<->WASM marshaling is the floor, which a traversal swap cannot remove.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Follow-up to #1299: the per-pass bar still parked on one number while a
single long pass ran — on C-heavy repos that's the fn-pointer dispatch
pass, which sweeps every C/C++ file four times (typedefs, registrations,
field propagation, dispatch sites) and dominates the linking phase.
The pass now reports a real fraction of its dominant work
(scannedFiles / files×4, at the same per-16-files cadence as its
cooperative yield), and the orchestrator surfaces instrumented passes'
fractions as fractional steps, throttled to whole-percent movement so
the UI message volume stays bounded. The mechanism is opt-in per pass —
any synthesizer that a real repo shows parking the bar can adopt the
same callback.
Verified on the 1,342-file C repo from the report: the linking bar now
moves through 88→89→90 where it previously sat at 88 for the whole
pass; graph byte-identical (50,520 nodes / 148,232 edges).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Two first-run UX bugs surfaced by indexing a real 1,342-file C repo:
1. After 'Resolving refs' hit 100%, the ~40 dynamic-dispatch synthesis
passes ran with no progress surface, so the bar sat frozen at 100%
long enough to read as a hang (the C fn-pointer pass alone can hold
for a while on C-heavy repos). Synthesis now reports per-pass
progress through a new 'linking' IndexProgress phase, rendered as
'Linking dynamic dispatch'. The step total is pinned by a test to
the synthesizer's actual __mark() count so adding a pass without
bumping it fails loudly.
2. node:sqlite's ExperimentalWarning is emitted once per THREAD, so the
main process plus every parse worker printed it mid-index,
interleaved with the progress UI. All launch paths now pass
--disable-warning=ExperimentalWarning: both bundle launchers, the
Windows npm-shim invocation, and the CLI self-relaunch
(NODE_RUNTIME_FLAGS, deliberately excluded from the re-exec gate so
an older installed launcher never triggers a pointless re-exec, and
version-gated off nodes older than the flag).
Verified end-to-end on the same repo: zero warnings, live linking bar,
byte-identical graph (50,520 nodes / 148,232 edges). Full suite green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
