b9ca4b7981116909900368cc1686a1074cd4d4c1
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Colby MchenryandGitHub
|
838006c947 |
fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581) (#1600)
Fixes #1581. ## What was wrong `codegraph init` / `codegraph index` died with `Segmentation fault` — the whole CLI process, not a parse worker — on a C/C++ file with very deep brace nesting (llvm's `clang/test/Parser/parser_overflow.c`, 16,384 nested `{`). The reporter's diagnosis is exactly right: tree-sitter's parser is iterative, so the file parses fine, and then the native kernel's **recursive walker** (`visit_node` → `visit_for_calls_and_structure` → …, one frame per AST level) overflowed the thread's stack. A native overflow can't be caught the way a wasm abort can, and a parse worker is a thread of the `codegraph` process, so the SIGSEGV took the entire indexer down — no message, no per-file fallback, no partial index. Two things made "just give the worker a bigger stack" the wrong fix: - it only moves the cliff — reproduced here: the reporter's 16,384-deep file kills a default 4 MiB worker (rc=132 on macOS / 139 on Linux), and a 100k-deep file kills the 8 MiB **main** thread too; - the walkers are shared by every kernel-routed language (20 of them), and each has several recursion points with different frame sizes, so no single stack size is a provable bound. Meanwhile the wasm path already handles this shape gracefully: its JS walker catches its own `RangeError` per file and stores a partial result with a `parse_error`. The kernel just needed a way to get there instead of dying. ## What this does **The kernel guards its own recursion against the calling thread's real stack bounds and defers a too-deep file to wasm** — the same `defer:` routing signal it already uses for files with parse errors, which `src/extraction/kernel/index.ts` treats as "take the wasm path for this file", silently. - `codegraph-kernel/src/stack.rs`: per-thread stack bounds from the OS, computed once per thread and cached — glibc/musl `pthread_getattr_np` + `pthread_attr_getstack`, macOS `pthread_get_stackaddr_np` + `pthread_get_stacksize_np`, Win32 `GetCurrentThreadStackLimits` (a hand-declared `kernel32` extern; no `windows-sys`). `exhausted()` is one thread-local load and one compare: true once the stack pointer is within a 256 KiB red zone of the limit, and it latches a flag. Where the OS can't report bounds it falls back to a fixed 1 MiB descent budget measured from the entry stack pointer — safe on anything from Node's 4 MiB worker default up. So the guard is exact on the 4 MiB worker, the 8 MiB main thread, and any `resourceLimits.stackSizeMb` alike. - `stack_guard!()` (defined in `lib.rs`) is the first statement of every recursive walker function — all **150** self-recursive or on-cycle functions across the 15 walker modules, found by script (every cycle in the call graph, not just direct self-calls). It returns `Default::default()` (`()`, `false`, `None`, `""`) so an exhausted walk simply stops descending; a hook returning `false` sends its caller down the generic child walk, whose own guard returns at once. - `extract_file` runs the whole walk under `stack::run_guarded`: if the flag is set afterwards the (truncated) result is discarded and replaced by `defer: nesting too deep for the native walker — wasm recovery handles it`. - `parse-pool.ts`: a comment at `new Worker(scriptPath)` records why there is deliberately no `resourceLimits.stackSizeMb` bump. - No new crates beyond `libc` as a direct unix dependency (already in `Cargo.lock` transitively). No wire/ABI change. Net effect for the reporter's repo: `deep.c` goes to the wasm path, lands as `function foo` plus a recorded parse warning, and the other 31,607 files index normally. `CODEGRAPH_KERNEL=0` and the `exclude` workaround are no longer needed. ## Tests **Rust unit tests** (`cargo test`, 21 passed — 7 new in `stack.rs`): the walkers for C, C++, Rust, TypeScript and Python are driven on a **1 MiB** thread (a quarter of Node's worker default) with 30k-deep nesting and must return `defer:` instead of crashing; shallow files are untouched; the latch resets between runs; the OS bounds are sane on the main thread and describe a small thread's own stack. **`__tests__/kernel-deep-nesting.test.ts`** (new, 8 tests — skips without a staged `.node`, fails under `CODEGRAPH_KERNEL_EXPECT=1` if the kernel is missing, like the other kernel suites): - every default-routed language (all 20) survives a 60k-deep expression on the main thread — clean result or the wasm fallback's partial result, never a crash; - the reporter's exact 16,384-brace C file is indexed (partial) on the main thread; - 200-deep expressions in every language still take the kernel path clean (the guard never trips on normal code); - inside a **default-sized 4 MiB `worker_threads` Worker** through `dist/`: the reporter's `deep.c` and a 60k-deep expression in every language come back `deferred` with exit 0, and a normal file still extracts natively; - end-to-end through the built CLI: `codegraph init` on a repo holding `deep.c` + `ok.c` exits 0 and records both files, with both functions. **Existing kernel suites**: all 15 (`kernel-*-parity`, `kernel-scaffold`, `kernel-retry-materialize`, `kernel-grammar-parity`) pass unchanged, 147 tests — the guard never fires on the parity fixtures. **Reporter's probes** (`one.js` from the issue, default 4 MiB worker, this build): `deep.c` → `deferred`, exitCode=0 (was rc=132/139); `deep100k.c` → `deferred`, exitCode=0. Main thread: `deep.c` / `deep100k.c` → wasm partial with `Parse error: Maximum call stack size exceeded`; a 6,000-term binary expression and a 3,000-branch `else if` chain stay on the kernel path with clean results. **Perf** (same `dist/`, only the `.node` swapped via `CODEGRAPH_KERNEL_PATH`; interleaved main/new ×3, `codegraph init`, macOS arm64): | repo | main (median) | guarded (median) | nodes / edges | |---|---|---|---| | express (141 files) | 0.60 s (0.58–0.65) | 0.61 s (0.58–0.61) | 1,084 / identical | | redis (786 C/H files) | 4.44 s (4.39–4.66) | 4.49 s (4.41–4.70) | 19,942 / 76,446 identical | Within run-to-run noise, as expected for one TLS load + compare per recursion entry. **Linux (Docker, `node:22-bookworm`, kernel built in-container, `docker run --rm --init`)** — the reporter's platform and the glibc `pthread_getattr_np` bounds path: ``` === platform === Linux efe3cc86947b 6.12.54-linuxkit #1 SMP Tue Nov 4 21:21:47 UTC 2025 aarch64 GNU/Linux v22.22.3 -rwxr-xr-x 1 root root 35332288 Aug 22 18:02 codegraph-kernel/prebuilds/linux-arm64/codegraph-kernel.node === reporter repro (issue #1581): 16,384-brace deep.c, codegraph init === │ └ Done init exit code: 0 file: deep.c file: deep100k.c file: ok.c function: add function: bar function: foo === worker probe: kernel raw extract in a default 4 MiB worker === deep.c: deferred deep.c: worker exitCode=0 deep100k.c: deferred deep100k.c: worker exitCode=0 ok.c: kernel nodes=2 ok.c: worker exitCode=0 === cargo test stack:: (glibc pthread_getattr_np bounds path) === test stack::tests::os_bounds_are_sane_on_this_platform ... ok test stack::tests::small_stack_reports_its_own_bounds ... ok test stack::tests::normal_files_are_untouched_by_the_guard ... ok test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok test stack::tests::latch_resets_between_runs ... ok test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.23s === vitest: kernel-deep-nesting + kernel-scaffold === ✓ __tests__/kernel-scaffold.test.ts (10 tests) 30ms ✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 36989ms Test Files 2 passed (2) Tests 18 passed (18) ``` (The pre-fix crash was reproduced on macOS — rc=132 in a default worker, rc=139 on the main thread at 100k depth — not re-run inside this container; the reporter's Linux x86_64 trace is the SIGSEGV form of the same overflow.) **Windows (Parallels ARM64 VM, MSVC 14.44, `cargo 1.97`, kernel built on the VM, `GetCurrentThreadStackLimits` path)**: ``` head: cbf8485 fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581) === cargo build --release (win32-arm64) === Finished `release` profile [optimized] target(s) in 2m 04s staged: 35086848 bytes === cargo test (stack guard unit tests) === test stack::tests::normal_files_are_untouched_by_the_guard ... ok test stack::tests::os_bounds_are_sane_on_this_platform ... ok test stack::tests::small_stack_reports_its_own_bounds ... ok test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok test stack::tests::latch_resets_between_runs ... ok test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.49s === reporter repro: codegraph init on a 16,384-brace deep.c === └ Done init exit code: 0 === vitest: deep-nesting + scaffold (CODEGRAPH_KERNEL_EXPECT=1) === ✓ __tests__/kernel-scaffold.test.ts (10 tests) 55ms ✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 67239ms ✓ every default-routed language survives a 60k-deep expression on the main thread 52801ms ✓ inside a default-sized (4 MiB) parse worker, through dist/ > defers a 60k-deep expression in every default-routed language 13050ms ✓ end-to-end: codegraph init on a repo holding the deep file > exits 0 and records deep.c alongside the normal files 936ms Test Files 2 passed (2) Tests 18 passed (18) ``` (The end-to-end test is what reads the Windows index back through `node:sqlite` — `files` = `deep.c`, `ok.c`; functions `add`, `foo`.) Full `npm test` on this branch (macOS arm64, kernel staged): **190 files passed, 3,185 tests passed, 10 skipped, 0 failed.** Clippy note: `cargo clippy` on the current toolchain (1.92) reports 18 pre-existing lints (`manual_contains`, `unnecessary_to_owned`, …) in walker code this PR only touched by inserting guard lines; none are in `stack.rs`/`lib.rs`. Left alone to keep the diff reviewable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK |
||
|
Colby MchenryandGitHub
|
7963672689 |
fix(rust): resolve self.field.method() on the field's declared type instead of a same-named method (#1585) (#1599)
Fixes #1585. **Stacked on #1596** (the base branch is `fix/1588-rust-impl-type-qualification`; this PR's own diff is the second commit). Merge #1596 first, then retarget/merge this one. ## What was wrong ```rust impl Outer { pub fn run(&mut self) { self.inner.run(); // inner: Inner } } ``` produced `Outer::run -> Outer::run` — recursion the source doesn't contain. The extractor collapsed every `self.<field>.<method>()` receiver to the bare method name (`run`), so the resolver only ever saw `run` and exact-matched the nearest same-named method — the calling method itself, or a method of an unrelated type. Nothing marked the edge as a guess, and no row stayed in `unresolved_refs`, so a consumer had no way to tell. The same happened when the field's type isn't a project type at all (`its: std::vec::IntoIter<_>` → `self.its.next()`, `matcher: Regex` → `self.matcher.is_match()`): the bare `next` / `is_match` attached to whatever local method shared the name. ripgrep had 279 self-edges on `main`; the issue lists three sites, all of this shape. (The issue's C++ control — "`Outer::run -> Inner::run` resolves correctly" — doesn't actually hold on `main`: `inner.h` is classified as C by the `.h` heuristic, so `Inner::run` never exists and the C++ repro self-edges too. That's #1592, fixed separately.) ## What this does Rust struct fields are not graph nodes, so the field's type can only come from the struct's declaration text. This follows the Go 2-hop precedent exactly (`matchGoFieldChainCall`, #1276), including its exclusivity rule: 1. **Extraction (TS walker + native kernel, identical, parity-tested):** a call whose receiver is `self.<field>` keeps the owner-field shape — `self.inner.run()` is emitted as `self.inner.run`. Deeper chains (`self.a.b.m()`), call receivers (`self.f().m()`), parenthesized receivers and bare `self` keep the bare name, exactly as before. 2. **Resolution (`matchRustSelfFieldCall`):** owner type = the calling method's qualified-name prefix (`Outer::run` → `Outer`); the field's declared type is read from the owner struct's **own declaration lines** (comment-stripped, line by line — same discipline as the Go helper); the method is resolved **and validated** on that type by `resolveMethodOnType` (confidence 0.85, `instance-method`). 3. **Exclusive:** when the field is declared with an external type, a generic parameter (`T`), a container that doesn't auto-deref (`Option`/`Vec`/`Mutex`/…), or can't be found, the ref **stays unresolved** — it never falls through to the bare-name strategies. That is the safe behaviour the issue asks for, and it is what #1276 already chose for Go. `rustFieldTypeName` looks through exactly the layers Rust's method-call auto-deref looks through: references (`&`, `&'a mut`) and the owning smart pointers `Box`/`Rc`/`Arc`. `Box<dyn Source>` yields the trait, whose method node the interface-impl synthesizer then fans out to every implementation. `Option<Inner>` is left alone — `self.inner.take()` is Option's method and must not become `Inner::take`. Why it stacks on #1596: the owner is taken from the method's qualified name, which for a generic/lifetime impl was the trait's name before that fix. ## Measured on ripgrep (110 `.rs` files, #1596 build vs this branch) | | #1596 | this PR | |---|---|---| | nodes | 4029 | 4029 | | `calls` self-edges | 279 | **146** (none of the `self.<field>` shape remain — 116 bare-receiver, 30 other dotted) | | `self.<field>.m()` calls resolved through a validated field type | — | **292** (`DecompressionMatcher::command -> GlobSet::matches`, `Parser::find_long -> FlagMap::find`, `Haystack::path -> DirEntry::path`, …) | | `self.<field>.m()` calls left unresolved | — | **417** — every sampled one is a std/container method: `self.commands.push`, `self.child.wait`, `self.pre.is_some`, `self.colors.clone`, `self.path_terminator.unwrap_or` | | `calls` edges total | 9150 | 8878 (the 272 removed are the former bare-name guesses for those 417) | The issue's three sites: `walk.rs:824` now resolves to `IgnoreBuilder::add_custom_ignore_filename` (was a self-edge); `walk.rs:1195` (`self.its.next`, `IntoIter`) and `globset/lib.rs:983` (`self.matcher.is_match`, `Regex`) are parked as unresolved instead of guessed. The issue's repro gives `Outer::run -> Inner::run` (`instance-method`, confidence 0.85) on both the kernel path and `CODEGRAPH_KERNEL=0`. ## Tests - `__tests__/extraction.test.ts`: only the single-hop `self.<field>.<method>()` call keeps the prefix; deeper / call / parenthesized / bare-`self` receivers and a local receiver are unchanged. - `__tests__/resolution.test.ts` (end-to-end, Cargo layout): the issue's repro → `Outer::run -> Inner::run`, no self-edge; an external field type (`std::vec::IntoIter`) with a local `next` decoy → no edge at all; `Box<Inner>` and `&'a mut Inner` resolve, `Option<Inner>` does not (even though `Inner` declares the method); a generic `T` field → no edge; genuine `self.run()` recursion keeps its self-edge; the #1588 repro's `UsesFile::go` / `UsesBuf::go` resolve to `FileSource::read` / `BufSource::read`, and a `Box<dyn Source>` field lands on `Source::read` with the synthesizer fanning out to both impls. - `__tests__/fixtures/kernel-parity/torture.rs` grows the receiver shapes; all 15 kernel parity suites pass against the rebuilt kernel (147 tests). - Full `npm test` on this branch: 189 files, 3187 passed, 9 skipped, 0 failed. Re-index after upgrading. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK |
||
|
Colby MchenryandGitHub
|
12f7a59f26 |
fix(rust): qualify generic/lifetime impl methods by the implementing type, not the trait (#1588) (#1596)
Fixes #1588. ## What was wrong The receiver of an `impl` block — the name that qualifies its methods, owns the `contains` edge, and sources the `implements` edge — was found positionally: the **last bare `type_identifier` child** of the `impl_item`. That works for `impl Source for FileSource`. But once the implementing type carries parameters it parses as a `generic_type`, and the only bare identifier left is the **trait's**: ```rust impl Source for FileSource → FileSource::read ✓ impl<T> Source for BufSource<T> → Source::read ✗ (should be BufSource::read) impl<'a> Iterator for Parents<'a> → Iterator::next ✗ impl Trait for &Foo → Trait::method ✗ ``` Two consequences, both reproduced on `main`: - `BufSource::read` did not exist in the graph, so `resolveMethodOnType("BufSource", "read")` and "who calls `BufSource::read`" had no answer, and every generic implementation of a trait collapsed onto the same trait-qualified name. - Because the impl's method carried the trait's qualified name, the interface-impl synthesizer treated the impl **body** as a second trait declaration and emitted a dispatch edge from it (`Source::read -> FileSource::read`, registered at the generic impl's line — a body of `{ 0 }` containing no call at all). The native kernel (`rustlang.rs`) mirrored the positional rule deliberately, bug-for-bug, to hold byte-parity with the TS walker — its header said "preserve, never fix via the grammar's trait:/type: fields". So the fix has to land on both sides at once. ## What this does Both extractors now read the grammar's **named fields** instead of scanning children. One shared rule (`rustImplTypeName` in `languages/rust.ts`, `impl_type_name` in the kernel), applied to `impl_item.type`: | implementing type | node | receiver | |---|---|---| | `Foo` | `type_identifier` | `Foo` | | `Foo<T>` / `Foo<'a>` | `generic_type` → its `type` field | `Foo` | | `m::Foo` | `scoped_type_identifier` → its `name` field | `Foo` (was: no receiver) | | `&Foo` / `&'a mut Foo` | `reference_type` → its `type` field | `Foo` | | `(A, B)`, `dyn Tr`, `*const T`, `u32`, fn types | anything else | none — extracted as plain functions, exactly as before | The `implements` back-reference reads `impl_item.trait` (full text, so `fmt::Display` and `From<u32>` keep their spelling) and bails when the field is absent (inherent impl). Everything else — the no-scope impl quirk, the source-order `contains` owner scan, method extraction — is untouched; the `contains` edge simply lands on the implementing type now instead of the trait. The kernel header comment, the parity test's description, and the two design docs that documented the quirk as "preserve" are updated to say what changed. ## Measured on ripgrep (110 `.rs` files, `main` build vs this branch) | | main | this PR | |---|---|---| | nodes / methods | 4029 / 2202 | 4029 / 2202 | | impl methods qualified by a **trait** name (node outside that trait's extent) | 61 | **0** | | `Iterator::*` methods | 2 | 0 | | duplicate method qualified names | 77 | 42 | | synthesized `interface-impl` edges originating **outside** any trait declaration (the phantom fan-outs) | 38 | **0** | | synthesized `interface-impl` edges originating at a real trait declaration | 33 | **52** | | plain (non-heuristic) `calls` edges | 9098 | 9098 | So the synthesizer lost every phantom edge and *gained* 19 legitimate fan-outs to implementations it could not previously see as implementations. `contains` edges went 5237 → 5224: the 13 removed were trait→impl-method edges produced by the mis-qualification. The issue's repro now gives `BufSource::read` at line 12, `BufSource -> Source`, and both synthesized edges registered at the declaration (line 2) — identical on the kernel path and with `CODEGRAPH_KERNEL=0`. (The remaining `UsesFile::go -> BufSource::read` exact-match guess there is the separate `self.field.method()` receiver problem, #1585, which stacks on this.) ## Tests - `__tests__/extraction.test.ts` (Rust Extraction): method qualified names for generic / lifetime / reference / scoped / generic-trait impls; the trait's qualified name names exactly one node; `implements` refs come from the implementing type for every shape; the `contains` edge lands on the type; tuple / `dyn` impls keep producing plain functions with no `implements` ref. - `__tests__/resolution.test.ts` (end-to-end): `Source::read` names only the declaration; dispatch fans out to **both** `FileSource::read` and `BufSource::read`, every synthesized edge registered at line 2; neither impl body sprouts a synthesized call. - `__tests__/fixtures/kernel-parity/torture.rs` grows all the new impl shapes; `kernel-rustlang-parity` (LF + CRLF) passes against the rebuilt kernel. - `CODEGRAPH_KERNEL_EXPECT=1 npx vitest run __tests__/kernel-*.test.ts` — all 15 suites, 147 tests pass. - Full `npm test`: 3180 passed, 9 skipped, 1 failed — `mcp-daemon.test.ts > daemon idle-times-out after the last client disconnects`, a 30 s timing test that passed on re-run in isolation (the machine was running four parallel suites and kernel builds at the time); unrelated to extraction. Re-index after upgrading to pick up the corrected names. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK |
||
 ctype_lab
|
8e3cde6994 | feat(kernel): preserve union nodes across extraction | ||
 ctype_labandClaude Opus 5
|
85e9ac6762 |
fix(c,cpp,objc,rust): index union declarations
A `union` declaration produced no symbol at all in any of the four
languages that have one. The type never entered the graph, and neither
did anything attached to it — in Rust every `impl Trait for MyUnion`
lost its edge, and the impl's methods were left with a qualifiedName
pointing at a type the graph did not contain.
`union_specifier` / `union_item` were absent from the extraction layer
entirely: no `<x>Types` list on the TS side, no dispatch branch in
either kernel walker.
They join `structTypes` (kind `struct` — NodeKind has no `union`), which
is the extension point the table-driven extractors already provide. The
body guard in extractStruct is untouched, so a bodiless `union U;` stays
a forward declaration and is still skipped, exactly like `struct U;`.
`resolveTypeAliasKind` accepts `union_specifier` too, so
`typedef union { … } N;` takes the typedef's name the way
`typedef struct { … } N;` already did. Without it the anonymous union
body would mint a second `<anonymous>` node beside the alias.
Both walkers change together so kernel<->wasm parity holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
f1ca991943 |
feat(kernel): R7b Rust walker — rustlang module, tree-sitter-rust 0.24.2 bump, rust default-routed (#1371)
First R7b language port. Grammar: tree-sitter-rust pinned =0.24.2 + wasm vendored from tag 77a3747 (parser.c/scanner.c sha-matched against the crates.io tarball), replacing the 2023 ABI-14 tree-sitter-wasms build — the bump alone is precision-positive on the wasm path (receiver-qualified instance-method resolutions replace ambiguous bare-name matches; node sections byte-identical on ripgrep/tokio). Walker mirrors the TS reference bug-for-bug per docs/design/rust-lang-kernel-port-checklist.md (survey artifact): dead-code isAsync, impl-pushes-no-scope, the impl-Trait-for-Generic<T> trait-receiver quirk, phantom const identifiers, use-binding triple emission, wildcard-use-emits-nothing, scoped-supertrait drop, chained-call re-encode gated on scoped_identifier, Rocket route macros body-only. Gates: parity sweeps 0 diffs — ripgrep 101/101, tokio 790/790, rust-analyzer 1217/1488 (271 deferrals are token-macro-table sources that error on BOTH arms — grammar-inherent); full-init dump-diffs byte-identical on all three (3,857 / 13,440 / 39,030 nodes); kernel-rustlang-parity suite (torture + CRLF + defer) in npm test; full suite green x2 with CODEGRAPH_KERNEL_EXPECT=1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |