codegraph

github/codegraph

Fork 0

Commit Graph

Author	SHA1	Message	Date
Colby MchenryandGitHub	838006c947	fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581 ) (#1600 ) Fixes #1581. ## What was wrong `codegraph init` / `codegraph index` died with `Segmentation fault` — the whole CLI process, not a parse worker — on a C/C++ file with very deep brace nesting (llvm's `clang/test/Parser/parser_overflow.c`, 16,384 nested `{`). The reporter's diagnosis is exactly right: tree-sitter's parser is iterative, so the file parses fine, and then the native kernel's recursive walker (`visit_node` → `visit_for_calls_and_structure` → …, one frame per AST level) overflowed the thread's stack. A native overflow can't be caught the way a wasm abort can, and a parse worker is a thread of the `codegraph` process, so the SIGSEGV took the entire indexer down — no message, no per-file fallback, no partial index. Two things made "just give the worker a bigger stack" the wrong fix: - it only moves the cliff — reproduced here: the reporter's 16,384-deep file kills a default 4 MiB worker (rc=132 on macOS / 139 on Linux), and a 100k-deep file kills the 8 MiB main thread too; - the walkers are shared by every kernel-routed language (20 of them), and each has several recursion points with different frame sizes, so no single stack size is a provable bound. Meanwhile the wasm path already handles this shape gracefully: its JS walker catches its own `RangeError` per file and stores a partial result with a `parse_error`. The kernel just needed a way to get there instead of dying. ## What this does The kernel guards its own recursion against the calling thread's real stack bounds and defers a too-deep file to wasm — the same `defer:` routing signal it already uses for files with parse errors, which `src/extraction/kernel/index.ts` treats as "take the wasm path for this file", silently. - `codegraph-kernel/src/stack.rs`: per-thread stack bounds from the OS, computed once per thread and cached — glibc/musl `pthread_getattr_np` + `pthread_attr_getstack`, macOS `pthread_get_stackaddr_np` + `pthread_get_stacksize_np`, Win32 `GetCurrentThreadStackLimits` (a hand-declared `kernel32` extern; no `windows-sys`). `exhausted()` is one thread-local load and one compare: true once the stack pointer is within a 256 KiB red zone of the limit, and it latches a flag. Where the OS can't report bounds it falls back to a fixed 1 MiB descent budget measured from the entry stack pointer — safe on anything from Node's 4 MiB worker default up. So the guard is exact on the 4 MiB worker, the 8 MiB main thread, and any `resourceLimits.stackSizeMb` alike. - `stack_guard!()` (defined in `lib.rs`) is the first statement of every recursive walker function — all 150 self-recursive or on-cycle functions across the 15 walker modules, found by script (every cycle in the call graph, not just direct self-calls). It returns `Default::default()` (`()`, `false`, `None`, `""`) so an exhausted walk simply stops descending; a hook returning `false` sends its caller down the generic child walk, whose own guard returns at once. - `extract_file` runs the whole walk under `stack::run_guarded`: if the flag is set afterwards the (truncated) result is discarded and replaced by `defer: nesting too deep for the native walker — wasm recovery handles it`. - `parse-pool.ts`: a comment at `new Worker(scriptPath)` records why there is deliberately no `resourceLimits.stackSizeMb` bump. - No new crates beyond `libc` as a direct unix dependency (already in `Cargo.lock` transitively). No wire/ABI change. Net effect for the reporter's repo: `deep.c` goes to the wasm path, lands as `function foo` plus a recorded parse warning, and the other 31,607 files index normally. `CODEGRAPH_KERNEL=0` and the `exclude` workaround are no longer needed. ## Tests Rust unit tests (`cargo test`, 21 passed — 7 new in `stack.rs`): the walkers for C, C++, Rust, TypeScript and Python are driven on a 1 MiB thread (a quarter of Node's worker default) with 30k-deep nesting and must return `defer:` instead of crashing; shallow files are untouched; the latch resets between runs; the OS bounds are sane on the main thread and describe a small thread's own stack. `__tests__/kernel-deep-nesting.test.ts` (new, 8 tests — skips without a staged `.node`, fails under `CODEGRAPH_KERNEL_EXPECT=1` if the kernel is missing, like the other kernel suites): - every default-routed language (all 20) survives a 60k-deep expression on the main thread — clean result or the wasm fallback's partial result, never a crash; - the reporter's exact 16,384-brace C file is indexed (partial) on the main thread; - 200-deep expressions in every language still take the kernel path clean (the guard never trips on normal code); - inside a default-sized 4 MiB `worker_threads` Worker through `dist/`: the reporter's `deep.c` and a 60k-deep expression in every language come back `deferred` with exit 0, and a normal file still extracts natively; - end-to-end through the built CLI: `codegraph init` on a repo holding `deep.c` + `ok.c` exits 0 and records both files, with both functions. Existing kernel suites: all 15 (`kernel--parity`, `kernel-scaffold`, `kernel-retry-materialize`, `kernel-grammar-parity`) pass unchanged, 147 tests — the guard never fires on the parity fixtures. Reporter's probes* (`one.js` from the issue, default 4 MiB worker, this build): `deep.c` → `deferred`, exitCode=0 (was rc=132/139); `deep100k.c` → `deferred`, exitCode=0. Main thread: `deep.c` / `deep100k.c` → wasm partial with `Parse error: Maximum call stack size exceeded`; a 6,000-term binary expression and a 3,000-branch `else if` chain stay on the kernel path with clean results. Perf (same `dist/`, only the `.node` swapped via `CODEGRAPH_KERNEL_PATH`; interleaved main/new ×3, `codegraph init`, macOS arm64): \| repo \| main (median) \| guarded (median) \| nodes / edges \| \|---\|---\|---\|---\| \| express (141 files) \| 0.60 s (0.58–0.65) \| 0.61 s (0.58–0.61) \| 1,084 / identical \| \| redis (786 C/H files) \| 4.44 s (4.39–4.66) \| 4.49 s (4.41–4.70) \| 19,942 / 76,446 identical \| Within run-to-run noise, as expected for one TLS load + compare per recursion entry. Linux (Docker, `node:22-bookworm`, kernel built in-container, `docker run --rm --init`) — the reporter's platform and the glibc `pthread_getattr_np` bounds path: ``` === platform === Linux efe3cc86947b 6.12.54-linuxkit #1 SMP Tue Nov 4 21:21:47 UTC 2025 aarch64 GNU/Linux v22.22.3 -rwxr-xr-x 1 root root 35332288 Aug 22 18:02 codegraph-kernel/prebuilds/linux-arm64/codegraph-kernel.node === reporter repro (issue #1581): 16,384-brace deep.c, codegraph init === │ └ Done init exit code: 0 file: deep.c file: deep100k.c file: ok.c function: add function: bar function: foo === worker probe: kernel raw extract in a default 4 MiB worker === deep.c: deferred deep.c: worker exitCode=0 deep100k.c: deferred deep100k.c: worker exitCode=0 ok.c: kernel nodes=2 ok.c: worker exitCode=0 === cargo test stack:: (glibc pthread_getattr_np bounds path) === test stack::tests::os_bounds_are_sane_on_this_platform ... ok test stack::tests::small_stack_reports_its_own_bounds ... ok test stack::tests::normal_files_are_untouched_by_the_guard ... ok test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok test stack::tests::latch_resets_between_runs ... ok test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.23s === vitest: kernel-deep-nesting + kernel-scaffold === ✓ __tests__/kernel-scaffold.test.ts (10 tests) 30ms ✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 36989ms Test Files 2 passed (2) Tests 18 passed (18) ``` (The pre-fix crash was reproduced on macOS — rc=132 in a default worker, rc=139 on the main thread at 100k depth — not re-run inside this container; the reporter's Linux x86_64 trace is the SIGSEGV form of the same overflow.) Windows (Parallels ARM64 VM, MSVC 14.44, `cargo 1.97`, kernel built on the VM, `GetCurrentThreadStackLimits` path): ``` head: cbf8485 fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581) === cargo build --release (win32-arm64) === Finished `release` profile [optimized] target(s) in 2m 04s staged: 35086848 bytes === cargo test (stack guard unit tests) === test stack::tests::normal_files_are_untouched_by_the_guard ... ok test stack::tests::os_bounds_are_sane_on_this_platform ... ok test stack::tests::small_stack_reports_its_own_bounds ... ok test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok test stack::tests::latch_resets_between_runs ... ok test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.49s === reporter repro: codegraph init on a 16,384-brace deep.c === └ Done init exit code: 0 === vitest: deep-nesting + scaffold (CODEGRAPH_KERNEL_EXPECT=1) === ✓ __tests__/kernel-scaffold.test.ts (10 tests) 55ms ✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 67239ms ✓ every default-routed language survives a 60k-deep expression on the main thread 52801ms ✓ inside a default-sized (4 MiB) parse worker, through dist/ > defers a 60k-deep expression in every default-routed language 13050ms ✓ end-to-end: codegraph init on a repo holding the deep file > exits 0 and records deep.c alongside the normal files 936ms Test Files 2 passed (2) Tests 18 passed (18) ``` (The end-to-end test is what reads the Windows index back through `node:sqlite` — `files` = `deep.c`, `ok.c`; functions `add`, `foo`.) Full `npm test` on this branch (macOS arm64, kernel staged): 190 files passed, 3,185 tests passed, 10 skipped, 0 failed. Clippy note: `cargo clippy` on the current toolchain (1.92) reports 18 pre-existing lints (`manual_contains`, `unnecessary_to_owned`, …) in walker code this PR only touched by inserting guard lines; none are in `stack.rs`/`lib.rs`. Left alone to keep the diff reviewable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK	2026-08-26 10:38:29 -05:00

Author

SHA1

Message

Date

Colby MchenryandGitHub

838006c947

fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581 ) (#1600 )

Fixes #1581.

## What was wrong

`codegraph init` / `codegraph index` died with `Segmentation fault` — the whole CLI
process, not a parse worker — on a C/C++ file with very deep brace nesting (llvm's
`clang/test/Parser/parser_overflow.c`, 16,384 nested `{`). The reporter's diagnosis is
exactly right: tree-sitter's parser is iterative, so the file parses fine, and then the
native kernel's **recursive walker** (`visit_node` → `visit_for_calls_and_structure` → …,
one frame per AST level) overflowed the thread's stack. A native overflow can't be caught
the way a wasm abort can, and a parse worker is a thread of the `codegraph` process, so
the SIGSEGV took the entire indexer down — no message, no per-file fallback, no partial
index.

Two things made "just give the worker a bigger stack" the wrong fix:

- it only moves the cliff — reproduced here: the reporter's 16,384-deep file kills a
  default 4 MiB worker (rc=132 on macOS / 139 on Linux), and a 100k-deep file kills the
  8 MiB **main** thread too;
- the walkers are shared by every kernel-routed language (20 of them), and each has
  several recursion points with different frame sizes, so no single stack size is a
  provable bound.

Meanwhile the wasm path already handles this shape gracefully: its JS walker catches its
own `RangeError` per file and stores a partial result with a `parse_error`. The kernel
just needed a way to get there instead of dying.

## What this does

**The kernel guards its own recursion against the calling thread's real stack bounds and
defers a too-deep file to wasm** — the same `defer:` routing signal it already uses for
files with parse errors, which `src/extraction/kernel/index.ts` treats as "take the wasm
path for this file", silently.

- `codegraph-kernel/src/stack.rs`: per-thread stack bounds from the OS, computed once per
  thread and cached — glibc/musl `pthread_getattr_np` + `pthread_attr_getstack`, macOS
  `pthread_get_stackaddr_np` + `pthread_get_stacksize_np`, Win32
  `GetCurrentThreadStackLimits` (a hand-declared `kernel32` extern; no `windows-sys`).
  `exhausted()` is one thread-local load and one compare: true once the stack pointer is
  within a 256 KiB red zone of the limit, and it latches a flag. Where the OS can't report
  bounds it falls back to a fixed 1 MiB descent budget measured from the entry stack
  pointer — safe on anything from Node's 4 MiB worker default up. So the guard is exact on
  the 4 MiB worker, the 8 MiB main thread, and any `resourceLimits.stackSizeMb` alike.
- `stack_guard!()` (defined in `lib.rs`) is the first statement of every recursive walker
  function — all **150** self-recursive or on-cycle functions across the 15 walker modules,
  found by script (every cycle in the call graph, not just direct self-calls). It returns
  `Default::default()` (`()`, `false`, `None`, `""`) so an exhausted walk simply stops
  descending; a hook returning `false` sends its caller down the generic child walk, whose
  own guard returns at once.
- `extract_file` runs the whole walk under `stack::run_guarded`: if the flag is set
  afterwards the (truncated) result is discarded and replaced by
  `defer: nesting too deep for the native walker — wasm recovery handles it`.
- `parse-pool.ts`: a comment at `new Worker(scriptPath)` records why there is deliberately
  no `resourceLimits.stackSizeMb` bump.
- No new crates beyond `libc` as a direct unix dependency (already in `Cargo.lock`
  transitively). No wire/ABI change.

Net effect for the reporter's repo: `deep.c` goes to the wasm path, lands as
`function foo` plus a recorded parse warning, and the other 31,607 files index normally.
`CODEGRAPH_KERNEL=0` and the `exclude` workaround are no longer needed.

## Tests

**Rust unit tests** (`cargo test`, 21 passed — 7 new in `stack.rs`): the walkers for
C, C++, Rust, TypeScript and Python are driven on a **1 MiB** thread (a quarter of Node's
worker default) with 30k-deep nesting and must return `defer:` instead of crashing;
shallow files are untouched; the latch resets between runs; the OS bounds are sane on the
main thread and describe a small thread's own stack.

**`__tests__/kernel-deep-nesting.test.ts`** (new, 8 tests — skips without a staged `.node`,
fails under `CODEGRAPH_KERNEL_EXPECT=1` if the kernel is missing, like the other kernel
suites):
- every default-routed language (all 20) survives a 60k-deep expression on the main thread
  — clean result or the wasm fallback's partial result, never a crash;
- the reporter's exact 16,384-brace C file is indexed (partial) on the main thread;
- 200-deep expressions in every language still take the kernel path clean (the guard never
  trips on normal code);
- inside a **default-sized 4 MiB `worker_threads` Worker** through `dist/`: the reporter's
  `deep.c` and a 60k-deep expression in every language come back `deferred` with exit 0,
  and a normal file still extracts natively;
- end-to-end through the built CLI: `codegraph init` on a repo holding `deep.c` + `ok.c`
  exits 0 and records both files, with both functions.

**Existing kernel suites**: all 15 (`kernel-*-parity`, `kernel-scaffold`,
`kernel-retry-materialize`, `kernel-grammar-parity`) pass unchanged, 147 tests — the guard
never fires on the parity fixtures.

**Reporter's probes** (`one.js` from the issue, default 4 MiB worker, this build):
`deep.c` → `deferred`, exitCode=0 (was rc=132/139); `deep100k.c` → `deferred`, exitCode=0.
Main thread: `deep.c` / `deep100k.c` → wasm partial with
`Parse error: Maximum call stack size exceeded`; a 6,000-term binary expression and a
3,000-branch `else if` chain stay on the kernel path with clean results.

**Perf** (same `dist/`, only the `.node` swapped via `CODEGRAPH_KERNEL_PATH`; interleaved
main/new ×3, `codegraph init`, macOS arm64):

| repo | main (median) | guarded (median) | nodes / edges |
|---|---|---|---|
| express (141 files) | 0.60 s (0.58–0.65) | 0.61 s (0.58–0.61) | 1,084 / identical |
| redis (786 C/H files) | 4.44 s (4.39–4.66) | 4.49 s (4.41–4.70) | 19,942 / 76,446 identical |

Within run-to-run noise, as expected for one TLS load + compare per recursion entry.

**Linux (Docker, `node:22-bookworm`, kernel built in-container, `docker run --rm --init`)** —
the reporter's platform and the glibc `pthread_getattr_np` bounds path:

```
=== platform ===
Linux efe3cc86947b 6.12.54-linuxkit #1 SMP Tue Nov  4 21:21:47 UTC 2025 aarch64 GNU/Linux
v22.22.3
-rwxr-xr-x 1 root root 35332288 Aug 22 18:02 codegraph-kernel/prebuilds/linux-arm64/codegraph-kernel.node

=== reporter repro (issue #1581): 16,384-brace deep.c, codegraph init ===
│
└  Done

init exit code: 0
  file: deep.c
  file: deep100k.c
  file: ok.c
  function: add
  function: bar
  function: foo

=== worker probe: kernel raw extract in a default 4 MiB worker ===
deep.c: deferred
deep.c: worker exitCode=0
deep100k.c: deferred
deep100k.c: worker exitCode=0
ok.c: kernel nodes=2
ok.c: worker exitCode=0

=== cargo test stack:: (glibc pthread_getattr_np bounds path) ===
test stack::tests::os_bounds_are_sane_on_this_platform ... ok
test stack::tests::small_stack_reports_its_own_bounds ... ok
test stack::tests::normal_files_are_untouched_by_the_guard ... ok
test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok
test stack::tests::latch_resets_between_runs ... ok
test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.23s

=== vitest: kernel-deep-nesting + kernel-scaffold ===
✓ __tests__/kernel-scaffold.test.ts (10 tests) 30ms
✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 36989ms
Test Files  2 passed (2)
Tests  18 passed (18)
```

(The pre-fix crash was reproduced on macOS — rc=132 in a default worker, rc=139 on the main thread at 100k depth — not re-run inside this container; the reporter's Linux x86_64 trace is the SIGSEGV form of the same overflow.)

**Windows (Parallels ARM64 VM, MSVC 14.44, `cargo 1.97`, kernel built on the VM,
`GetCurrentThreadStackLimits` path)**:

```
head: cbf8485 fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581)
=== cargo build --release (win32-arm64) ===
    Finished `release` profile [optimized] target(s) in 2m 04s
staged: 35086848 bytes
=== cargo test (stack guard unit tests) ===
test stack::tests::normal_files_are_untouched_by_the_guard ... ok
test stack::tests::os_bounds_are_sane_on_this_platform ... ok
test stack::tests::small_stack_reports_its_own_bounds ... ok
test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok
test stack::tests::latch_resets_between_runs ... ok
test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.49s
=== reporter repro: codegraph init on a 16,384-brace deep.c ===
└  Done
init exit code: 0
=== vitest: deep-nesting + scaffold (CODEGRAPH_KERNEL_EXPECT=1) ===
✓ __tests__/kernel-scaffold.test.ts (10 tests) 55ms
✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 67239ms
   ✓ every default-routed language survives a 60k-deep expression on the main thread 52801ms
   ✓ inside a default-sized (4 MiB) parse worker, through dist/ > defers a 60k-deep expression in every default-routed language 13050ms
   ✓ end-to-end: codegraph init on a repo holding the deep file > exits 0 and records deep.c alongside the normal files 936ms
Test Files  2 passed (2)
Tests  18 passed (18)
```

(The end-to-end test is what reads the Windows index back through `node:sqlite` — `files` = `deep.c`, `ok.c`; functions `add`, `foo`.)

Full `npm test` on this branch (macOS arm64, kernel staged): **190 files passed, 3,185 tests passed, 10 skipped, 0 failed.**

Clippy note: `cargo clippy` on the current toolchain (1.92) reports 18 pre-existing lints
(`manual_contains`, `unnecessary_to_owned`, …) in walker code this PR only touched by
inserting guard lines; none are in `stack.rs`/`lib.rs`. Left alone to keep the diff
reviewable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK

2026-08-26 10:38:29 -05:00