feat(extraction): C deferral round 2 — 8 new preParse passes, linux kernel/+mm/ deferral 58.6%→33.9% (#1353)

Census-driven cut of the top-ranked post-R7a lever. All passes TS-side,
C-only (preParseCSource), shared by both arms:

- parameterized-annotation whole-blank (__free/__printf/__counted_by/
  __bpf_md_ptr…; extends through a stranded field `;`)
- type-keyword-arg scanner (kzalloc_obj(struct T), list_entry, multi-line
  continuations behind nested-paren args; bounded hand scanner, head
  exclusions + call-vs-declaration guard; blanks trailing stars)
- static/extern CAPS-macro declaration lines at any scope; the initialized
  form is REWRITTEN to its expansion (name/tail keep exact offsets)
- va_arg qualified-type blank; GNU named-variadic #define dots-only blank
  (post-restore); sandwiched notrace-family; C23 auto; multi-line
  iterator-macro spans (hlist_for_each_entry_rcu + lockdep arg)
- word list += cacheline family (2- and 4-underscore spellings) + 10 more
  census-confirmed annotations

Gates: five-repo parity sweeps 0 diffs (git deferral 16.1→12.2%, redis
25.3→24.1%, fmt/protobuf unchanged); linux full-tree both arms
2,049,153 nodes / 6,413,518 edges (+858/+6,585 vs R7a) with byte-identical
dumps (10,446,478 lines, sha256 6dd1185b); kernel-arm parse-loop 356→306s
at 2c; suite 2517 green under CODEGRAPH_KERNEL_EXPECT=1. Honesty note
recorded in the docs: error recovery was already salvaging most SYMBOLS on
deferred files — the graph win is relationships + phantom cleanup, and the
unreleased CHANGELOG entry was rewritten off the sweep-subset framing.

Also records §7a.5: post-R7a 8-core cg1212 re-run 16.4min (was 18.3min);
8c parse sits on the single-writer floor, so the <10min-on-8c gap re-ranks
to the per-ref resolution path.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-18 01:00:00 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 2d72891b59
commit b9d0f57a64
6 changed files with 776 additions and 18 deletions
+1 -1
View File
@@ -34,7 +34,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- Deleting a whole directory is now picked up by watch mode: the files inside it are removed from the index on the next auto-sync instead of lingering as stale records until an unrelated edit happened to trigger one. Operating systems often report a directory deletion as a single event on the directory itself (with no per-file events for its contents), which the watcher previously discarded. (#1285)
- `codegraph sync` now gets the same slow-disk fix that made full indexing fast in 1.4.0: database checkpointing is deferred for the whole incremental run instead of firing every few megabytes of writes. On mechanical drives and other high-latency storage, a small sync on a large index no longer stalls for minutes at near-zero CPU — the cost of a sync scales with what changed, not with the size of the existing index. The same `CODEGRAPH_NO_WAL_DEFER=1` switch turns it off. (#1248)
- C functions declared with a project-specific attribute macro in front of a typedef'd return type (`SEC_ATTR UINT32 MyFunc(VOID)` — common in embedded and kernel code) are now indexed under their real names. Previously the parser tripped over the unknown macro and stored the parameter list as the function name, leaving entries like `"(VOID)"` in the graph and making the real function unfindable. (#1211)
- Macro-heavy C and C++ code indexes much more completely. Six ubiquitous idioms that previously tripped the parser into error recovery — dropping or garbling the surrounding symbols — now parse cleanly: the `#ifdef __cplusplus` / `extern "C" {` compatibility guard in C headers, iterator macros in statement position (`list_for_each_entry(pos, head, member) { … }` and the whole Linux-kernel/git/jemalloc family, braced or single-statement), the Linux/sparse declaration annotations (`static int __init foo(void)`, `void __user *buf`, `container_of(p, struct T, m)`), trailing parameter annotations (`int argc UNUSED`, git's house style), namespace-management macros alone on a line (`FMT_BEGIN_NAMESPACE`, Qt's `Q_OBJECT`), and function attribute macros in front of C++ return types. On git's own repository this nearly doubles the number of indexed symbols, and on the Linux kernel's `kernel/` and `mm/` directories it triples them; blast radius and callers get correspondingly more complete. A related fix stops the existing macro handling from corrupting `#define` lines that mention the same macro names, which removed a class of phantom parse errors in fmt-style headers.
- Macro-heavy C and C++ code indexes much more completely. Fifteen ubiquitous idioms that previously tripped the parser into error recovery — dropping symbols, garbling names, minting phantom entries, or losing the relationships between real ones — now parse cleanly: the `#ifdef __cplusplus` / `extern "C" {` compatibility guard in C headers, iterator macros in statement position (`list_for_each_entry(pos, head, member) { … }` and the whole Linux-kernel/git/jemalloc family — including calls that wrap across lines, like `hlist_for_each_entry_rcu(…)` with a `lockdep_is_held` argument), the Linux/sparse declaration annotations (`static int __init foo(void)`, `void __user *buf`, `container_of(p, struct T, m)`), parameterized annotations (`__free(kfree)`, `__printf(4, 0)`, `__counted_by(count)`), per-CPU and work-queue declaration macros (`static DEFINE_PER_CPU(struct T, name);`, initialized forms included), bare type arguments to allocator and list macros (`kzalloc_obj(struct T)`, `list_entry(p, struct T, member)`), `va_arg` with a qualified or pointer type, GNU named-variadic macro definitions (`#define dbg(fmt, args...)`), `static notrace`-style compiler markers, C23 `auto` declarations, trailing parameter annotations (`int argc UNUSED`, git's house style), namespace-management macros alone on a line (`FMT_BEGIN_NAMESPACE`, Qt's `Q_OBJECT`), and function attribute macros in front of C++ return types. On the Linux kernel this cuts files lost to parse errors on the core directories by nearly half and recovers thousands of call and reference relationships that error recovery silently dropped, with git and redis seeing smaller cuts of the same kind; graphs stay byte-for-byte identical between the native and fallback engines. A related fix stops the macro handling itself from corrupting `#define` lines that mention the same macro names, which removed a class of phantom parse errors in fmt-style headers.
- C++ methods defined out-of-line inside a namespace (`namespace sim { Output MyClass::Apply(...) { ... } }`) now carry the namespace in their qualified name, matching their class. Fully-qualified call sites from other files (`sim::MyClass::Apply(...)`) resolve to the definition again, so `codegraph callers` and file impact no longer come up empty for this pattern. (#1291)
- C++ methods defined out-of-line on a template class (`template <typename T> T Box<T>::get() { ... }`) no longer keep the template parameter list in their qualified name. They now index as `Box::get` — identical to an inline definition of the same method — so they link to their class and resolve from call sites again, and pathological multi-line template parameter lists can no longer blow the qualified name past filesystem name limits. (#1286)
- Go route detection no longer misidentifies ordinary method calls that share HTTP verb names — `cache.Put("key", value)`, `store.Get("config", out)`, `bus.Handle("user.created", handler)` and the like were being indexed as HTTP routes, polluting route listings in cache-heavy codebases. A registration now has to look like one: its first argument must be a `/`-prefixed path (all routers) or a Go 1.22 `"METHOD /path"` pattern on `Handle`/`HandleFunc`, which now also extracts the method instead of listing the route as `ANY`. (#1259)