b2f9ab1800d15477ee08ddbd6a3d8a26caf34410
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
69ea438bac |
perf(kernel): cFnPtr native extraction sweep — step 2, pass 230→151s across the arc (§7a.10) (#1365)
Task #5 step 2. The fuse-then-link refactor (#1364) left the extraction sweep as a clean per-file boundary: raw text in → collected facts out. This ports that sweep to the native kernel: `cfnptr_scan_files` (codegraph-kernel/src/cfnptr.rs) strips and scans a batch of 16 files per NAPI call, and the TS side only reads files, ships batches, interns the returned facts, and resolves include paths. The JS sweep remains as the fallback (no binary, feature detection against older binaries, CODEGRAPH_KERNEL=0, or CODEGRAPH_KERNEL_CFNPTR=0). Parity discipline: the JS regexes are the spec, so the scanners are hand-rolled byte machines reproducing that engine — ASCII \w/\b next to UNICODE \s (NBSP/U+2000-200A/FEFF decoded from UTF-8), alternation order, lastIndex resume, and the observable backtracking dimensions (INIT/ARRAY modifier and struct/star/bracket optionals, DISPATCH's greedy segment loop); greedy shortcuts only where backtracking provably can't rescue a match. The native stripper blanks per UTF-16 code unit, so its output is string-identical to the TS stripper — pinned by a new kernel arm on the strip differential oracle (fixtures + 500 seeded random cases). Gates, all green: new differential suite (adversarial fixture project — CRLF, NBSP, continuations, decoy strings, unterminated comments, backtracking shapes — indexed native-vs-JS: identical edge streams, plus a record-level scanner check); repo differential on git/redis/vim/SameBoy (identical, 705/852/433/180 edges); probe-hash on the live linux kernel DB reproduced f6e1713d… (279,335 rows); linux init counts exact 2,049,153/6,413,518; dump sha 6dd1185b… reproduced (10,446,478 lines); full suite green ×2 (153 files / 2588 tests). Measured (8c cg1212, quiet host): cFnPtr sub A=47.9s B=1.1 C=40.9 D=24.1 E=36.8 = 150.9s vs step 1's 179s and the pre-arc 230s (−34% cumulative); the sweep itself halved (94.5→47.9s, JS strips 132.4k→68.9k). callback-synthesis phase 199.9→171.1s. E's attributed wall grew from overlap shift under parallel synthesis; the phase total is the honest number. Full record: plan §7a.10. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
c6850d737b |
perf(resolution): cFnPtr fuse-then-link — one extraction sweep + filtered verbatim linking, pass −22% at kernel scale (#1364)
Task #5 step 1 (plan §7a.8/§7a.9). The C/C++ function-pointer dispatch synthesizer swept every file's text four times (typedefs, registrations, propagation, dispatch); at kernel scale the all-or-nothing source cache declines, so that was 4.4 read+strips per file — 78s of the ~230s pass. Now ONE extraction sweep reads+strips each file once and collects typedef names, struct-node field declarations (structurally parsed, fn-pointer classification deferred until the typedef sets are complete), resolved includes, an alias-shaped-macro name set, and per-file survival filters (init type tokens, array element types, inline-struct summaries, field-assign pairs, dispatch fields/array names — interned, a few MB on linux). The linking stages then replay the ORIGINAL pass bodies verbatim: struct layouts register in kind-scan order (same-name precedence is order-sensitive), and registration/propagation/dispatch run only for files their filter proves can have side effects, lazily re-stripping just those. Filters only over-approximate (full-file no-skip scans ⊇ the real passes' jump-cursor scans), and a filtered-out file is one where every match fails the pass's own gates before any side effect — parity by construction. Macro tables stay lazy: a sizing probe found 6.1M #define lines on linux (amdgpu register headers), ruling out retention. Measured (8c cg1212, quiet host, fresh kernel init): cFnPtr pass 230s → 179s (−22%); strips 283.5k → 132.4k (4.44 → 2.08/file, 78 → 46.6s); dispatch stage 95 → 18.5s; callback-synthesis phase 250 → 199.9s. Standalone probe on the live DB: 139 → 122s. Byte-parity gates, all green: probe-hash identical on the live kernel DB (279,335 edge rows both builds); git/redis/vim/SameBoy full dumps byte-identical old-vs-new (macro tables, commands.def, #ifdef include-units, inline structs, bare arrays exercised); kernel-parity 0-diffs on git/redis/fmt/protobuf, deferral unchanged; linux counts exact 2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478 lines); full suite green ×2 (152 files / 2563 tests). Step 2 (native per-file extractor) now has its boundary: the extraction sweep, raw text in → records out. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d510de4766 |
perf(synthesis): cFnPtrEdges 2.07x at kernel scale — probe-profiled, edge set hash-identical (#1341)
cFnPtrEdges was 86% of kernel-scale synthesis (306s, §7a.3). Standalone probe iterations against the live kernel DB attributed and fixed: - sliceLines split the ENTIRE file per node (~1.6M full-file splits across the B/D/E sweeps) — split once per file, slice from the line array (D 46.2->20.5s, E 94.5->69.1s). - The 128-entry strip cache re-stripped every file 4.4x across the four file sweeps (71.8s). Sizing is now memory-budget-aware and ALL-OR-NOTHING with 5% slack over the file count: a partial LRU on cyclic sweeps thrashes to ~0% hit (measured twice: cap ~61k and cap == files both thrashed against 63.8k + includes). Full-cache strips: exactly 1.0/file, 15.5s. - Pass D pre-gates field names on fieldToStructs before the two regex type resolutions — a->f = b->g matches every struct-field assignment in the tree, ~99% of them data fields. - recvTypeIn/varTypeIn compiled a new RegExp per call - cached per name. - Line numbers counted incrementally per match (ascending indexes) instead of splitting the body prefix per match; cursor rewinds between the two dispatch scans. - CODEGRAPH_SYNTH_TIMINGS now prints per-sweep walls + read/strip/nodes accounting for this pass. 278.8s -> 134.8s standalone at the 2c envelope. Output identity proven at full scale: probe edge set (merge-dedup applied, canonical sort) SHA256 21c2a971... == the original code's 274,762 edges extracted from the live kernel DB. Pass suite 15/15. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
246aee8373 |
fix(ui): within-pass progress for the C fn-pointer linking pass (#1300)
Follow-up to #1299: the per-pass bar still parked on one number while a single long pass ran — on C-heavy repos that's the fn-pointer dispatch pass, which sweeps every C/C++ file four times (typedefs, registrations, field propagation, dispatch sites) and dominates the linking phase. The pass now reports a real fraction of its dominant work (scannedFiles / files×4, at the same per-16-files cadence as its cooperative yield), and the orchestrator surfaces instrumented passes' fractions as fractional steps, throttled to whole-percent movement so the UI message volume stays bounded. The mechanism is opt-in per pass — any synthesizer that a real repo shows parking the bar can adopt the same callback. Verified on the 1,342-file C repo from the report: the linking bar now moves through 88→89→90 where it previously sat at 88 for the whole pass; graph byte-identical (50,520 nodes / 148,232 edges). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
a3f90089e8 |
fix(indexing): bounded-memory yielding pipeline tail + daemon session fixes (#1212) (#1226)
Large-codebase indexing died at the end of "Resolving refs" two ways: watchdog kills of healthy work (24k-file Java on Windows, #1212 — third iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale, where v1.3.0 could not complete at any watchdog setting). Root causes: ~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no yield points, several materialized whole-graph snapshots (kotlin expect/actual opened with getAllNodes() — 2M nodes in one array; the C fn-pointer pass retained every C file's contents twice plus every function node), and the post-index WAL checkpoint ran minutes of synchronous IO on the main thread, killing even a successful index at the finish line. The pipeline tail now follows the same discipline as the rest: never hold O(graph) in the heap, yield everywhere. - All synthesis passes stream node-kind scans (cursors, not arrays) and yield on time-budgeted checkpoints; language gates skip passes whose filters a project's file languages provably can't satisfy. - kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded, units stream one file at a time, and the all-functions array + write-only id map are gone; spring reads each .java once, not twice. - runMaintenance moved to a worker thread (own SQLite connection); per-file store commits chunk with yields behind a serialized flush chain (preserving #1015 file-order determinism); resolver warm-up streams the DISTINCT name set; resolution batch-tail and merged-edge inserts run in bounded sub-transactions. - Daemon: fixed a socket-handoff race that could leave a fresh MCP session permanently silent (client-hello tail unshifted into a flowing stream with zero listeners — the long-standing #662 test flake was this real bug); first tool call no longer queues behind the query pool's cold start (pool.ready gate). Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully indexes in 27m8s on a 2-core/6GB container at default heap + default watchdog; llvm-project (180k files) completes under 1GB RSS including kill-and-sync recovery; synthesized-edge and full-graph parity are byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky daemon test passed 25/25 under load. Env-gated diagnostics kept: CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop tracing. Design record: docs/design/main-thread-stall-followup.md. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
703629edc3 |
feat(c/c++): resolve function-pointer command tables — macro-built, conditional-compilation & bare arrays (#991) (#1003)
* feat(c/c++): resolve macro-built function-pointer command tables (#991) C/C++ commands dispatched through macro-built function-pointer tables were dead-ends in the graph: redis' `call` never showed up as a caller of any command (`c->cmd->proc(c)`), because the table is generated into a #included `.def`, the handler is buried inside `MAKE_CMD(...)`, the struct type is itself a macro alias, the `proc` field uses a function-TYPE typedef, and the receiver is a chained field access. #954 deferred exactly this shape. Six composable additions to c-fnptr-synthesizer.ts close it: - function-type typedefs (`typedef RET T(...)` + `T *f`) flag the field as a function pointer; - multi-declarator fields (`struct redisCommand *cmd, *last`) each count as a slot/type (needed for positional alignment and the chain walk); - chained/array receivers (`c->cmd->proc`) resolve through field types across all same-named struct layouts (redis has two unrelated `client` structs); - `#include "x"` directives are followed (from raw source) so a non-indexed `.def` is read as a registration unit with the includer's effective macro env; - function-like + object-like macros are expanded (params->args, type aliases) before positional/designated registration; - a macro that expands to a brace-wrapped element (sqlite `FUNCTION(...)`) has one outer brace layer peeled. Validated on two independent macro-table lineages at 100% target precision: redis (209 commands via redisCommand.proc, `call`->every command) and sqlite (69 FuncDef.xSFunc targets). No regression on the controls: git (cmd_struct.fn, 138 builtins), curl (Curl_cftype.*), lua (0). 0 non-function targets across all five; +3 synthetic fixtures; full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(c/c++): resolve conditional-compilation command tables (vim) (#991) Vim's `:ex` and normal-mode command tables are the hardest fn-pointer-table shape: the struct is defined INLINE with the array, the whole thing is behind `#ifdef DO_DECLARE_EXCMD`/`DO_DECLARE_NVCMD` (switched on by the includer), built by a macro the file conditionally redefines (`EXCMD`/`NVCMD` = the table element under the switch, a bare enum id otherwise), and dispatched by a parenthesized array subscript through a file-scope table: `(cmdnames[i].cmd_func)(&ea)`. Four more composable additions on top of the macro-table work: - a focused `#ifdef`/`#ifndef`/`#if defined`/`#else`/`#elif`/`#endif` evaluator drops inactive arms (unevaluable `#if EXPR` keeps its body); an indexed header is re-scanned in an includer's context only when that includer #defines a switch the header guards, with the include's macros re-read from the resolved text (the plain last-wins parse picks the wrong, enum, arm); - inline `struct TAG {…} var[] = {…}` tables whose struct never became a node are parsed in place and registered; - array-subscript receivers (`tbl[i].f`) strip the subscript and resolve the base through a global-var → struct-type map; - an optional `)` before the call covers the parenthesized `(….f)(args)` form. Validated on vim: 273 `:ex` commands (`do_one_cmd`→every command) + 67 normal-mode commands, 0 non-function targets, 0 cross-table misroute (registering both tables is what stops `normal_cmd`'s `nv_cmds[i].cmd_func` from falling back to the `cmdname` owner of the shared field name). Controls unchanged at 0 non-function (redis/sqlite/git/curl gain coverage from array/global dispatch, lua still 0); +1 synthetic fixture; full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(c/c++): resolve bare arrays of function pointers (#991) The C/C++ fn-pointer synthesizer keyed everything on (struct type, fn-pointer field), so a dispatch through a bare array of function pointers — no struct, no field — was unbridged: an opcode/handler table like `static op_t *opcodes[256] = {nop,…}` invoked `opcodes[op](…)` left every handler with zero callers. Closes the last #991 deferred item. Keyed by the array VARIABLE name (a new `arrayReg`, parallel to the struct `reg`). Registration detects an array whose element type is a function typedef — a function-TYPE typedef element (`opcode_t *ops[]`, the `*` making it an array of pointers) or a function-pointer typedef element (`zend_rc_dtor_func_t t[]`) — and reads its literal entries, whether positional (`fn`/`&fn`), designated by index (`[IDX]=fn`), or cast-wrapped (`(cast)fn`). Dispatch is `tbl[i](…)` / `(*tbl[i])(…)`, gated on `tbl` being a known fn-pointer array (the precision anchor); the fan-out reaches the whole set (a runtime subscript hits any entry), like a command table. The same-file table wins on a name collision, so two file-local `static opcodes[256]` (SameBoy's CPU + disassembler) never cross. The fn-pointer typedef/field regexes now also tolerate a calling-convention macro before the `*` (`(ZEND_FASTCALL *name)`), which hardens the existing struct-field path too. Validated on two independent lineages: SameBoy (GB emulator) — 147 edges via `opcodes[]`, 0 cross-file leak; php-src (Zend) — 54 edges across 7 tables in the designated+cast+CC-typedef form. Control: lua 0 — its `lua_CFunction searchers[]` is pushed into the VM, never C-dispatched, so the call-gate fires nothing. No regression on the #991 corpus: redis (835) / sqlite (683) struct edges byte-identical, git +3 / curl +20 legitimate new bare-array edges, vim 433 with all guards holding; 0 non-function targets across all. + 4 fixtures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
ba209d9489 |
feat(c/c++): resolve function-pointer dispatch (#932) (#954)
C/C++ polymorphism is the function pointer: a struct fn-pointer field, concrete
functions registered into it through a table (`{"add", cmd_add}`), a designated
initializer (`.handler = on_open`), or an assignment, then dispatched indirectly
(`p->fn(argv)`). Static extraction captures neither the registration→field
binding nor the indirect call, so the dispatcher→handler edge was missing — git's
run_builtin looked like it called nothing, a vtable's implementations had no
callers, and the hook_demo.c in the issue was unreachable.
Add a resolution-layer synthesizer keyed by (struct type, fn-pointer field). It
reads source (the established Celery/Sidekiq/Spring pattern — C extraction has no
struct fields or indirect-call edges to build on) in passes: collect fn-pointer
typedefs, parse struct field layouts, collect registrations (positional matched
by field index, designated, and assignment), propagate field←field assignments
(so a generic hook slot reassigned from a registry — the hook_demo.c
`h->func = found->fn` shape — inherits the registry field's handlers), then link
each indirect dispatch site to the registered handlers. Receiver type resolves
from the enclosing function's params/locals, falling back to a field name unique
to one struct. Covers both the command-table idiom (git, redis) and the
ops-struct/vtable idiom (curl content-encoders, protocol handlers).
Pure edge synthesis (no node growth); high precision via the (struct, field) key.
Validated: git 502 edges (run_builtin→cmd_* plus git_hash_algo/archiver/reftable
vtables), redis 357 (dictType.hashFunction, connection + reply-object vtables),
curl 478 (Curl_cwtype.do_init → deflate/gzip/brotli/zstd); 0 non-function targets
on all three; node-stable; 0 on the lua control (its {name,fn} tables register
into the Lua VM, with no C indirect call to bridge). Full suite 1665 pass.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|