perf(kernel): cFnPtr native extraction sweep — step 2, pass 230→151s across the arc (§7a.10) (#1365)

Task #5 step 2. The fuse-then-link refactor (#1364) left the extraction
sweep as a clean per-file boundary: raw text in → collected facts out.
This ports that sweep to the native kernel: `cfnptr_scan_files`
(codegraph-kernel/src/cfnptr.rs) strips and scans a batch of 16 files
per NAPI call, and the TS side only reads files, ships batches, interns
the returned facts, and resolves include paths. The JS sweep remains as
the fallback (no binary, feature detection against older binaries,
CODEGRAPH_KERNEL=0, or CODEGRAPH_KERNEL_CFNPTR=0).

Parity discipline: the JS regexes are the spec, so the scanners are
hand-rolled byte machines reproducing that engine — ASCII \w/\b next to
UNICODE \s (NBSP/U+2000-200A/FEFF decoded from UTF-8), alternation
order, lastIndex resume, and the observable backtracking dimensions
(INIT/ARRAY modifier and struct/star/bracket optionals, DISPATCH's
greedy segment loop); greedy shortcuts only where backtracking provably
can't rescue a match. The native stripper blanks per UTF-16 code unit,
so its output is string-identical to the TS stripper — pinned by a new
kernel arm on the strip differential oracle (fixtures + 500 seeded
random cases).

Gates, all green: new differential suite (adversarial fixture project —
CRLF, NBSP, continuations, decoy strings, unterminated comments,
backtracking shapes — indexed native-vs-JS: identical edge streams,
plus a record-level scanner check); repo differential on
git/redis/vim/SameBoy (identical, 705/852/433/180 edges); probe-hash on
the live linux kernel DB reproduced f6e1713d… (279,335 rows); linux
init counts exact 2,049,153/6,413,518; dump sha 6dd1185b… reproduced
(10,446,478 lines); full suite green ×2 (153 files / 2588 tests).

Measured (8c cg1212, quiet host): cFnPtr sub A=47.9s B=1.1 C=40.9
D=24.1 E=36.8 = 150.9s vs step 1's 179s and the pre-arc 230s (−34%
cumulative); the sweep itself halved (94.5→47.9s, JS strips
132.4k→68.9k). callback-synthesis phase 199.9→171.1s. E's attributed
wall grew from overlap shift under parallel synthesis; the phase total
is the honest number. Full record: plan §7a.10.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-19 17:48:27 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent c6850d737b
commit 69ea438bac
8 changed files with 1830 additions and 2 deletions
+1 -1
View File
@@ -22,7 +22,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify <file> -R colbymchenry/codegraph`. - Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify <file> -R colbymchenry/codegraph`.
- Indexing inside CPU- or memory-limited containers (Docker, CI runners) now sizes its worker pools from the container's actual allowance instead of the host machine's, and giant codebases no longer balloon temporary database files during indexing (previously tens of GB of transient disk on Linux-kernel-scale projects). Together these prevent out-of-memory and out-of-disk failures on constrained machines; set `CODEGRAPH_RESOLVE_WORKERS` to override the resolution worker count explicitly. - Indexing inside CPU- or memory-limited containers (Docker, CI runners) now sizes its worker pools from the container's actual allowance instead of the host machine's, and giant codebases no longer balloon temporary database files during indexing (previously tens of GB of transient disk on Linux-kernel-scale projects). Together these prevent out-of-memory and out-of-disk failures on constrained machines; set `CODEGRAPH_RESOLVE_WORKERS` to override the resolution worker count explicitly.
- Indexing very large projects on multi-core machines got faster again: the parallel-resolution workers now periodically refresh their read-only database connections, which lets database housekeeping advance instead of silently building up a backlog behind long-lived readers — a backlog that was taxing the indexer's own writes. Graphs remain byte-for-byte identical; the win is largest at Linux-kernel scale on many-core machines. - Indexing very large projects on multi-core machines got faster again: the parallel-resolution workers now periodically refresh their read-only database connections, which lets database housekeeping advance instead of silently building up a backlog behind long-lived readers — a backlog that was taxing the indexer's own writes. Graphs remain byte-for-byte identical; the win is largest at Linux-kernel scale on many-core machines.
- Indexing large C and C++ codebases spends much less time in the function-pointer dispatch analysis (the pass that connects handler tables like a command table or an ops struct to their call sites): each source file is now read and prepared once instead of four times, and files that can't contribute any dispatch wiring are skipped outright in the later linking steps. On a Linux-kernel-scale tree the pass runs about a fifth faster and the end-of-indexing dispatch-linking stage drops accordingly, with graphs byte-for-byte identical. - Indexing large C and C++ codebases spends much less time in the function-pointer dispatch analysis (the pass that connects handler tables like a command table or an ops struct to their call sites): each source file is now read and prepared once instead of four times, files that can't contribute any dispatch wiring are skipped outright in the later linking steps, and on platforms with the native engine the per-file scanning itself now runs natively too. On a Linux-kernel-scale tree the pass runs about a third faster end-to-end, with graphs byte-for-byte identical; platforms without a native binary keep the same results on the previous path.
### Fixes ### Fixes
+214
View File
@@ -0,0 +1,214 @@
/**
* cFnPtr native extraction sweep differential gate (task #5 step 2).
*
* The synthesizer's extraction sweep has two implementations: the JS regex
* sweep and the kernel's `cfnptrScanFiles` (codegraph-kernel/src/cfnptr.rs).
* They must be record-identical, which this suite pins end-to-end: the same
* adversarial project is indexed twice CODEGRAPH_KERNEL_CFNPTR toggled
* and the synthesized fn-pointer-dispatch edges must match EXACTLY, including
* order (edge order is observable through FANOUT_CAP truncation).
*
* The fixture deliberately stacks the sweep's edge cases: macro-built tables
* behind a non-indexed include, `#ifdef`-guarded inline structs, object-macro
* type aliases, bare fn-pointer arrays with casts and designators, chained
* receivers, fieldfield propagation, CRLF line endings, NBSP whitespace,
* `\`-continuations, strings containing decoy syntax, an unterminated block
* comment, and modifier/type backtracking shapes (`static x = {…}`).
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
import { CodeGraph } from '../src';
import { getKernel } from '../src/extraction/kernel/loader';
const kernel = getKernel();
const nativeAvailable = !!kernel && typeof kernel.cfnptrScanFiles === 'function';
interface EdgeRow {
src: string;
tgt: string;
via: string;
line: number;
}
describe.runIf(nativeAvailable)('cFnPtr sweep: native vs JS differential', () => {
let dir: string;
beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cfp-k-')); });
afterEach(() => {
delete process.env.CODEGRAPH_KERNEL_CFNPTR;
fs.rmSync(dir, { recursive: true, force: true });
});
const write = (rel: string, body: string) => {
const p = path.join(dir, rel);
fs.mkdirSync(path.dirname(p), { recursive: true });
fs.writeFileSync(p, body);
};
const indexAndCollect = async (): Promise<{ edges: EdgeRow[]; nodes: number }> => {
fs.rmSync(path.join(dir, '.codegraph'), { recursive: true, force: true });
const cg = await CodeGraph.init(dir, { silent: true });
await cg.indexAll();
const db = (cg as any).db.db;
const edges: EdgeRow[] = db
.prepare(
`SELECT s.name src, t.name tgt, json_extract(e.metadata,'$.via') via, e.line line
FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target
WHERE json_extract(e.metadata,'$.synthesizedBy') = 'fn-pointer-dispatch'
ORDER BY e.id`
)
.all();
const nodes = db.prepare('SELECT count(*) c FROM nodes').get().c as number;
cg.close?.();
return { edges, nodes };
};
const writeFixture = () => {
// The git shape + designated init + assignment registration.
write('cmd.c', `
struct cmd { const char *name; int (*fn)(int argc); };
static int cmd_add(int argc) { return argc + 1; }
static int cmd_rm(int argc) { return argc - 1; }
static struct cmd commands[] = {
{ "add", cmd_add },
{ "rm", cmd_rm },
};
int run(int i, int argc) { return commands[i].fn(argc); }
`);
// Macro-built table with an object-macro struct alias and a non-indexed
// include, redis-style; plus a typedef'd fn-TYPE field.
write('table.c', `
#include "table.h"
#include "cmds.def"
int dispatch(struct client *c, int a) { return c->cur->proc(a); }
`);
write('table.h', `
typedef int cmdProc(int a);
#define TBL_STRUCT redisCmd
struct redisCmd { const char *name; cmdProc *proc; };
struct client { struct redisCmd *cur; };
#define MK(nm, fn) { nm, fn }
static int getCmd(int a);
static int setCmd(int a);
`);
write('cmds.def', `
struct TBL_STRUCT tbl[] = {
MK("get", getCmd),
MK("set", setCmd),
};
`);
write('impl.c', `
#include "table.h"
static int getCmd(int a) { return a; }
static int setCmd(int a) { return a + 1; }
`);
// #ifdef-guarded inline struct table + parenthesized subscript dispatch
// (the vim shape), switched on by the includer.
write('ex.c', `
#define WANT_TABLE
#include "ex_cmds.h"
int exec(int i, int a) { return (cmdtab[i].cmd_fn)(a); }
`);
write('ex_cmds.h', `
#ifdef WANT_TABLE
static int ex_quit(int a);
struct excmd { char *nm; int (*cmd_fn)(int); } cmdtab[] = { { "q", ex_quit } };
#endif
`);
write('ex_impl.c', `static int ex_quit(int a) { return -a; }\n`);
// Bare arrays: fn-TYPE typedef with star, casts, index designators, and a
// same-named file-local collision (the SameBoy/Zend shapes).
write('ops.c', `
typedef int op_t(int);
static int nop(int x) { return x; }
static int halt(int x) { return -x; }
static op_t *ops[4] = { nop, [2] = (op_t *)halt };
int step(int pc, int x) { return ops[pc](x); }
`);
write('ops2.c', `
typedef int op_t(int);
static int trace(int x) { return x * 2; }
static op_t *ops[4] = { trace };
int step2(int pc, int x) { return (*ops[pc])(x); }
`);
// Field←field propagation (the hook_demo shape) + chained receiver.
write('hook.c', `
typedef void hook_fn(int);
struct entry { const char *nm; hook_fn *fn; };
struct hook { hook_fn *func; };
static void on_commit(int v) { (void)v; }
static struct entry entries[] = { { "commit", on_commit } };
void wire(struct hook *h, struct entry *found) { h->func = found->fn; }
void fire(struct hook *h, int v) { h->func(v); }
`);
// Adversarial text: CRLF, NBSP after 'struct', continuation before a
// #define, decoy syntax inside strings, backtick, unterminated comment,
// and the `static x = {` backtracking shape.
write('nasty.c', [
'struct weird { int (*go)(int); };',
'static int impl_go(int a) { return a; }\r',
'static struct weird w = { impl_go };\r',
// NBSP (U+00A0) between `struct` and the tag: JS `\s` is the Unicode
// class, so the initializer scan crosses it — the native sweep must too.
'static struct\u00A0weird w2 = { impl_go };',
'int poke(struct weird *p, int a) { return p->go(a); }',
'static x = {1};',
'const char *s = "struct fake { int (*f)(int); } decoy[] = { impl_go };";',
'int bt = 0; /* unterminated ` tick',
].join('\n'));
};
it('indexes to identical fn-pointer-dispatch edges with the sweep native vs JS', async () => {
writeFixture();
process.env.CODEGRAPH_KERNEL_CFNPTR = '0';
const js = await indexAndCollect();
delete process.env.CODEGRAPH_KERNEL_CFNPTR;
const native = await indexAndCollect();
expect(native.nodes).toBe(js.nodes);
expect(native.edges).toEqual(js.edges);
// The fixture must actually exercise the synthesizer, not vacuously pass.
expect(js.edges.length).toBeGreaterThanOrEqual(8);
const vias = new Set(js.edges.map((e) => e.via));
expect([...vias].some((v) => v.endsWith('[]'))).toBe(true); // bare-array path
expect([...vias].some((v) => v.includes('.'))).toBe(true); // struct-field path
}, 120_000);
it('native facts match the JS sweep on the raw scanner surface', () => {
// Direct record-level check of one adversarial file (no indexing): the
// kernel's per-file facts vs what the JS sweep's scans produce. Guards
// the scanner surface even for shapes the edge-level fixture might not
// reach (alias names, d-pairs, include capture order).
const text = [
'#define ALIAS realStruct',
'#define NUM 0x10',
'#define FN(x) x',
'typedef void (*cb_t)(int);',
'typedef int fnt(int);',
'#include "a.def"',
'#include "b.h"',
'struct realStruct { cb_t cb; fnt *f; int n; };',
'void go(struct realStruct *r, struct realStruct *q) {',
' r->cb = q->cb;',
' r->cb(1);',
' tbl[NUM](2);',
'}',
'static struct ALIAS one = { 0 };',
'static x = {1};',
].join('\n');
const out = kernel!.cfnptrScanFiles!([{ text, structs: [] }])[0]!;
expect(out.fnPtrTypedefs).toEqual(['cb_t']);
expect(out.fnTypeTypedefs).toEqual(['fnt']);
expect(out.aliasNames).toEqual(['ALIAS']); // NUM numeric, FN function-like
expect(out.includes).toEqual(['a.def', 'b.h']);
expect(out.dPairs).toEqual(['cb\0cb']);
expect(out.dispatchFields).toContain('cb');
expect(out.arrayDispatchNames).toContain('tbl');
expect(out.initTokens).toContain('ALIAS');
expect(out.initTokens).toContain('static'); // the backtracking shape
// `struct realStruct { … };` is followed by `;`, so it fails the
// inline-TABLE var check (`^\s*(\w+)`) — no candidate, like the JS scan.
expect(out.inlineTags).toEqual([]);
});
});
@@ -1,5 +1,6 @@
import { describe, it, expect } from 'vitest'; import { describe, it, expect } from 'vitest';
import { stripCommentsForRegex } from '../src/resolution/strip-comments'; import { stripCommentsForRegex } from '../src/resolution/strip-comments';
import { getKernel } from '../src/extraction/kernel/loader';
/** /**
* The pre-optimization split('')-based stripCStyle, kept verbatim as the * The pre-optimization split('')-based stripCStyle, kept verbatim as the
@@ -113,3 +114,32 @@ describe('stripCStyle segment-builder vs split-based oracle', () => {
expect(stripCommentsForRegex(src, 'c')).toBe(src); expect(stripCommentsForRegex(src, 'c')).toBe(src);
}); });
}); });
// The native kernel's C stripper (codegraph-kernel/src/cfnptr.rs) blanks per
// UTF-16 code unit precisely so its output is string-identical to the TS
// stripper — the cFnPtr extraction sweep's scanners then run over the same
// character stream on both paths. Pinned here against the same fixtures and
// randomized corpus as the TS rewrite.
const kernelStrip = getKernel()?.cfnptrStripC;
describe.runIf(typeof kernelStrip === 'function')('native cfnptrStripC vs TS stripper (c mode)', () => {
for (const [name, src] of FIXTURES) {
it(`fixture: ${name}`, () => {
expect(kernelStrip!(src)).toBe(stripCommentsForRegex(src, 'c'));
});
}
it('randomized differential (seeded, 500 cases)', () => {
let seed = 0x2fn;
const rand = (max: number): number => {
seed = (seed * 6364136223846793005n + 1442695040888963407n) & 0xffffffffffffffffn;
return Number(seed % BigInt(max));
};
const ATOMS = ['/*', '*/', '//', '\n', '"', "'", '`', '\\', 'x', ' ', '/', '*', 'é', '🚀', '\r\n', 'int a;'];
for (let caseN = 0; caseN < 500; caseN++) {
let s = '';
const len = rand(40);
for (let k = 0; k < len; k++) s += ATOMS[rand(ATOMS.length)]!;
expect(kernelStrip!(s), `case ${caseN}: ${JSON.stringify(s)}`).toBe(stripCommentsForRegex(s, 'c'));
}
});
});
File diff suppressed because it is too large Load Diff
+106
View File
@@ -18,6 +18,7 @@
mod buffers; mod buffers;
mod ccpp; mod ccpp;
mod cfnptr;
mod docstring; mod docstring;
mod ids; mod ids;
mod go; mod go;
@@ -98,6 +99,111 @@ pub fn grammar_info(language: String) -> Option<GrammarInfo> {
}) })
} }
/// One struct node's extent for the cFnPtr sweep (mirror of the TS caller's
/// `{ id, startLine, endLine }`, with `endLine ?? startLine` applied TS-side).
#[napi(object)]
pub struct CfnptrStructIn {
pub id: String,
pub start_line: u32,
pub end_line: u32,
}
#[napi(object)]
pub struct CfnptrFileIn {
/// RAW file text, exactly as the resolver's readFile returned it.
pub text: String,
pub structs: Vec<CfnptrStructIn>,
}
#[napi(object)]
pub struct CfnptrField {
pub name: String,
pub index: u32,
pub ptr: bool,
#[napi(js_name = "type")]
pub ty: String,
}
#[napi(object)]
pub struct CfnptrStructOut {
pub id: String,
pub parsed: bool,
pub fields: Vec<CfnptrField>,
}
/// The cFnPtr extraction-sweep facts for one file — see cfnptr.rs (and the
/// TS synthesizer's `FileFacts`) for field semantics.
#[napi(object)]
pub struct CfnptrFacts {
pub fn_ptr_typedefs: Vec<String>,
pub fn_type_typedefs: Vec<String>,
pub structs: Vec<CfnptrStructOut>,
pub inline_ptr: bool,
pub inline_types: Vec<String>,
pub inline_tags: Vec<String>,
pub init_tokens: Vec<String>,
pub array_elems: Vec<String>,
pub alias_names: Vec<String>,
pub d_pairs: Vec<String>,
pub dispatch_fields: Vec<String>,
pub array_dispatch_names: Vec<String>,
pub includes: Vec<String>,
}
/// Batched cFnPtr extraction sweep (task #5 step 2): one call scans a batch
/// of files and returns their collected facts, amortizing the NAPI boundary.
/// Feature-detected by the TS loader — absent on older binaries, where the
/// synthesizer keeps its JS sweep.
#[napi]
pub fn cfnptr_scan_files(files: Vec<CfnptrFileIn>) -> Vec<CfnptrFacts> {
files
.into_iter()
.map(|f| {
let structs: Vec<cfnptr::StructExtent> = f
.structs
.into_iter()
.map(|s| cfnptr::StructExtent { id: s.id, start_line: s.start_line, end_line: s.end_line })
.collect();
let facts = cfnptr::scan_file(&f.text, &structs);
CfnptrFacts {
fn_ptr_typedefs: facts.fn_ptr_typedefs,
fn_type_typedefs: facts.fn_type_typedefs,
structs: facts
.structs
.into_iter()
.map(|s| CfnptrStructOut {
id: s.id,
parsed: s.parsed,
fields: s
.fields
.into_iter()
.map(|fl| CfnptrField { name: fl.name, index: fl.index, ptr: fl.ptr, ty: fl.ty })
.collect(),
})
.collect(),
inline_ptr: facts.inline_ptr,
inline_types: facts.inline_types,
inline_tags: facts.inline_tags,
init_tokens: facts.init_tokens,
array_elems: facts.array_elems,
alias_names: facts.alias_names,
d_pairs: facts.d_pairs,
dispatch_fields: facts.dispatch_fields,
array_dispatch_names: facts.array_dispatch_names,
includes: facts.includes,
}
})
.collect()
}
/// Debug/differential hook: the native `stripCommentsForRegex(text, 'c')`.
/// Exists so the strip differential oracle can pin the Rust stripper against
/// the TS reference directly.
#[napi]
pub fn cfnptr_strip_c(text: String) -> String {
String::from_utf8_lossy(&cfnptr::strip_c(text.as_bytes())).into_owned()
}
#[napi] #[napi]
pub fn extract_file(file_path: String, content: String, language: String) -> Result<ExtractBuffers> { pub fn extract_file(file_path: String, content: String, language: String) -> Result<ExtractBuffers> {
let out = match language.as_str() { let out = match language.as_str() {
+51
View File
@@ -998,6 +998,57 @@ graph access inside the sweep except `getNodesInFile` for struct extents. Its
94.5s (46.6s strip + scans) is the native-extractor prize; C's 39.7s 94.5s (46.6s strip + scans) is the native-extractor prize; C's 39.7s
(macro-env + include units) and D's 24.8s stay TS. (macro-env + include units) and D's 24.8s stay TS.
#### 7a.10 cFnPtr native sweep landed (2026-07-19) — step 2 done, pass 230→151s across the arc
Step 2 shipped: `cfnptr_scan_files` in the kernel (codegraph-kernel/src/
cfnptr.rs) runs the entire extraction sweep natively — strip + all ten
scanners — batched 16 files per NAPI call; the TS sweep remains as the
fallback (no binary, feature detection against older binaries,
`CODEGRAPH_KERNEL=0`, or the scanner's own `CODEGRAPH_KERNEL_CFNPTR=0`).
What made it land at byte-parity:
- **Hand-rolled byte machines, not the regex crate.** The JS engine's
semantics are the spec: `\w`/`\b` are ASCII while `\s` is the Unicode
class (NBSP/U+2000-200A/FEFF — decoded explicitly from UTF-8), alternation
order, `lastIndex` resume, and the observable backtracking dimensions
(INIT/ARRAY modifier-count and `struct`/star/bracket optionals, DISPATCH's
greedy segment loop) are reproduced structurally; greedy-only shortcuts are
taken solely where analysis shows backtracking can never rescue a match
(documented per scanner).
- **The native stripper blanks per UTF-16 code unit** (two spaces for an
astral char), so its output is string-identical to the TS stripper — the
scanners run over the very character stream the JS regexes see, and the
strip differential oracle gained a kernel arm pinning that equality on the
same fixtures + 500 seeded random cases.
- **Gates, all green:** record/edge differential suite (adversarial fixture
project indexed native-vs-JS: identical edge streams; CRLF, NBSP,
continuations, decoy strings, unterminated comments, backtracking shapes);
repo differential on git/redis/vim/SameBoy (edge streams IDENTICAL,
705/852/433/180); probe-hash on the live kernel DB reproduced
`f6e1713d…` (279,335 rows) exactly; linux init counts exact
2,049,153/6,413,518 and dump sha `6dd1185b…` reproduced; full suite green
×2 (153 files / 2588).
Measured (8c cg1212, quiet host — one contaminated run discarded: the Mac
slept mid-init on battery and froze the VM, inflating resolution 4×; pmset
log confirmed, re-run caffeinated): cFnPtr sub **A=47.9s B=1.1 C=40.9 D=24.1
E=36.8 = 150.9s** vs step 1's 179s (28s) and the pre-arc 230s (**79s
cumulative, 34%**). The sweep itself halved (94.5 → 47.9s; JS strips
132.4k → 68.9k — exactly the sweep's share moved native). E's attributed
wall grew (18.5 → 36.8s): parallel synthesis overlap shifted as A finishes
earlier — stage walls absorb concurrent passes' contention; the phase total
is the honest number and **callback-synthesis fell 199.9 → 171.1s**.
Remaining cFnPtr ledger: C 40.9s (macro envs + include units, TS),
E's replay + overlap, D 24.1s, A's remaining 47.9s (reads, batching,
interning, DB struct extents — diminishing). The pass is no longer the
dominant synthesis lever; next per §7a.7 ranking: continuous-shallow WAL,
backpressure bytes.
Deploy note: this change ships RUST code — dist-only deploys are no longer
sufficient for it; rebuild the `.node` per platform (cg1212: cargo build in
`rust:1-bookworm` with `CARGO_TARGET_DIR=target-linux`, stage the `.so` as
`prebuilds/linux-arm64/codegraph-kernel.node`).
### 7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation) ### 7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation)
Priority order, each gated by the standard A/B + node-explosion probes: Priority order, each gated by the standard A/B + node-explosion probes:
1. **Test→subject edges** (first-class `tests` edges at index time; we compute covering 1. **Test→subject edges** (first-class `tests` edges at index time; we compute covering
+31
View File
@@ -53,10 +53,41 @@ export interface KernelGrammarInfo {
fieldNames: string[]; fieldNames: string[];
} }
/** Input to the cFnPtr extraction sweep: one file's raw text + its struct
* node extents (`endLine ?? startLine` applied by the caller). */
export interface CfnptrFileIn {
text: string;
structs: { id: string; startLine: number; endLine: number }[];
}
/** Per-file facts from the native cFnPtr extraction sweep mirror of the
* Rust `CfnptrFacts` (see codegraph-kernel/src/cfnptr.rs); semantics match
* the JS sweep in src/resolution/c-fnptr-synthesizer.ts. */
export interface CfnptrFactsOut {
fnPtrTypedefs: string[];
fnTypeTypedefs: string[];
structs: { id: string; parsed: boolean; fields: { name: string; index: number; ptr: boolean; type: string }[] }[];
inlinePtr: boolean;
inlineTypes: string[];
inlineTags: string[];
initTokens: string[];
arrayElems: string[];
aliasNames: string[];
dPairs: string[];
dispatchFields: string[];
arrayDispatchNames: string[];
includes: string[];
}
export interface KernelModule { export interface KernelModule {
extractFile(filePath: string, content: string, language: string): KernelBuffers; extractFile(filePath: string, content: string, language: string): KernelBuffers;
contractInfo(): KernelContractInfo; contractInfo(): KernelContractInfo;
grammarInfo(language: string): KernelGrammarInfo | null; grammarInfo(language: string): KernelGrammarInfo | null;
/** Batched cFnPtr extraction sweep (task #5 step 2). OPTIONAL: absent on
* older binaries callers feature-detect and keep their JS path. */
cfnptrScanFiles?(files: CfnptrFileIn[]): CfnptrFactsOut[];
/** Native `stripCommentsForRegex(text, 'c')` — differential-oracle hook. */
cfnptrStripC?(text: string): string;
} }
const debugEnabled = () => process.env.CODEGRAPH_KERNEL_DEBUG === '1'; const debugEnabled = () => process.env.CODEGRAPH_KERNEL_DEBUG === '1';
+87
View File
@@ -88,6 +88,8 @@ import type { MaybeYield } from './cooperative-yield';
import { memoryBudgetBytes } from './memory-budget'; import { memoryBudgetBytes } from './memory-budget';
import { LRUCache } from './lru-cache'; import { LRUCache } from './lru-cache';
import { stripCommentsForRegex } from './strip-comments'; import { stripCommentsForRegex } from './strip-comments';
import { getKernel } from '../extraction/kernel/loader';
import type { CfnptrFactsOut, CfnptrFileIn } from '../extraction/kernel/loader';
const C_CPP_EXT = /\.(c|h|cc|cpp|cxx|hpp|hh|hxx|cppm|ipp|inl|tcc)$/i; const C_CPP_EXT = /\.(c|h|cc|cpp|cxx|hpp|hh|hxx|cppm|ipp|inl|tcc)$/i;
const FN_KINDS = new Set(['function', 'method']); const FN_KINDS = new Set(['function', 'method']);
@@ -627,8 +629,93 @@ export async function cFnPointerDispatchEdges(
}; };
// ---- Stage A: the extraction sweep — ONE read + strip per file ---- // ---- Stage A: the extraction sweep — ONE read + strip per file ----
//
// Two implementations, record-identical by the differential suite:
// • native (task #5 step 2): the kernel's `cfnptrScanFiles` strips and
// scans a BATCH of files per NAPI call (codegraph-kernel/src/cfnptr.rs —
// hand-rolled byte machines replicating the JS regex semantics), and the
// TS side only reads files, ships batches, and interns the returned
// facts. Include-path resolution stays here (it needs the filesystem).
// • JS: the original sweep, kept verbatim — the fallback for platforms
// without a kernel binary, older binaries (feature detection), the
// CODEGRAPH_KERNEL=0 kill switch, and CODEGRAPH_KERNEL_CFNPTR=0 (this
// scanner's own switch).
const kernel =
process.env.CODEGRAPH_KERNEL === '0' || process.env.CODEGRAPH_KERNEL_CFNPTR === '0'
? null
: getKernel();
const nativeSweep =
kernel && typeof kernel.cfnptrScanFiles === 'function' ? kernel.cfnptrScanFiles.bind(kernel) : null;
const mergeNativeFacts = (file: string, out: CfnptrFactsOut): void => {
for (const t of out.fnPtrTypedefs) fnPtrTypedefs.add(intern(t));
for (const t of out.fnTypeTypedefs) fnTypeTypedefs.add(intern(t));
for (const so of out.structs) {
if (!so.parsed) continue; // body never parsed — the JS sweep records nothing either
rawFieldsByNode.set(
so.id,
so.fields.map((f) => ({ name: f.name || null, index: f.index, ptr: f.ptr, type: f.type }))
);
}
for (const t of out.inlineTags) inlineTags.add(intern(t));
for (const t of out.aliasNames) aliasNames.add(intern(t));
const includes: string[] = [];
for (const cap of out.includes) {
if (!INCLUDABLE_EXT.test(cap)) continue;
const t = resolveInclude(file, cap);
if (t) includes.push(intern(t));
}
if (
out.initTokens.length || out.arrayElems.length || out.inlinePtr || out.inlineTypes.length ||
out.dPairs.length || out.dispatchFields.length || out.arrayDispatchNames.length || includes.length
) {
factsByFile.set(file, {
initTokens: out.initTokens.length ? out.initTokens.map(intern) : null,
arrayElems: out.arrayElems.length ? out.arrayElems.map(intern) : null,
inlinePtr: out.inlinePtr,
inlineTypes: out.inlineTypes.length ? out.inlineTypes.map(intern) : null,
dPairs: out.dPairs.length ? out.dPairs.map(intern) : null,
dispatchFields: out.dispatchFields.length ? out.dispatchFields.map(intern) : null,
arrayDispatchNames: out.arrayDispatchNames.length ? out.arrayDispatchNames.map(intern) : null,
includes: includes.length ? includes : NO_INCLUDES,
});
}
};
let tPass = Date.now(); let tPass = Date.now();
if (nativeSweep) {
// Batch of 16 = the tick/onFraction cadence, so yielding and progress
// reporting keep their shape while the boundary crossing amortizes.
const BATCH = 16;
let batch: { file: string; input: CfnptrFileIn }[] = [];
const flush = (): void => {
if (batch.length === 0) return;
const outs = nativeSweep(batch.map((b) => b.input));
for (let bi = 0; bi < batch.length; bi++) mergeNativeFacts(batch[bi]!.file, outs[bi]!);
batch = [];
};
for (const file of files) { for (const file of files) {
await tick();
const rawText = raw(file);
if (!rawText) continue; // unreadable or empty — the JS sweep skips these too
const tN = prof ? Date.now() : 0;
const fileNodes = ctx.getNodesInFile(file);
if (prof) { prof.nodesMs += Date.now() - tN; prof.nodesN++; }
const structs: CfnptrFileIn['structs'] = [];
for (const st of fileNodes) {
if (st.kind !== 'struct') continue;
// sliceLinesPre semantics ride along: falsy startLine never parses,
// and `endLine ?? startLine` is applied here so the kernel sees the
// exact slice bounds the JS sweep would use.
structs.push({ id: st.id, startLine: st.startLine ?? 0, endLine: st.endLine ?? st.startLine ?? 0 });
}
batch.push({ file, input: { text: rawText, structs } });
if (batch.length >= BATCH) flush();
}
flush();
}
// JS sweep (fallback path — see the stage comment above).
if (!nativeSweep) for (const file of files) {
await tick(); await tick();
const s = src(file); const s = src(file);
if (!s) continue; if (!s) continue;