feat(kernel): R7a C/C++ walker — dual-lang ccpp module, preParse hoist, 7 new blanks, c/cpp default-routed (#1346)
Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps byte-identical on all five + linux at kernel scale (10.4M dump lines, same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x). Deferral guard corrected by measurement (C/C++ error incidence 9-42%; --max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse cost deferred files paid. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Fable 5
Claude Fable 5
parent
44561b6aad
commit
2d72891b59
+2
-1
@@ -11,7 +11,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
### New Features
|
||||
|
||||
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, and Go projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, and django-scale codebases (Lombok-generated members included). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
|
||||
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, and C++ projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, django-, git-, and protobuf-scale codebases (Lombok-generated members, C function-pointer tables, and Unreal-Engine-style macro-heavy headers included; CUDA and Metal sources ride the C++ path). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
|
||||
- Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=<count>` to tune when it engages.
|
||||
- Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle.
|
||||
- The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical.
|
||||
@@ -34,6 +34,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
- Deleting a whole directory is now picked up by watch mode: the files inside it are removed from the index on the next auto-sync instead of lingering as stale records until an unrelated edit happened to trigger one. Operating systems often report a directory deletion as a single event on the directory itself (with no per-file events for its contents), which the watcher previously discarded. (#1285)
|
||||
- `codegraph sync` now gets the same slow-disk fix that made full indexing fast in 1.4.0: database checkpointing is deferred for the whole incremental run instead of firing every few megabytes of writes. On mechanical drives and other high-latency storage, a small sync on a large index no longer stalls for minutes at near-zero CPU — the cost of a sync scales with what changed, not with the size of the existing index. The same `CODEGRAPH_NO_WAL_DEFER=1` switch turns it off. (#1248)
|
||||
- C functions declared with a project-specific attribute macro in front of a typedef'd return type (`SEC_ATTR UINT32 MyFunc(VOID)` — common in embedded and kernel code) are now indexed under their real names. Previously the parser tripped over the unknown macro and stored the parameter list as the function name, leaving entries like `"(VOID)"` in the graph and making the real function unfindable. (#1211)
|
||||
- Macro-heavy C and C++ code indexes much more completely. Six ubiquitous idioms that previously tripped the parser into error recovery — dropping or garbling the surrounding symbols — now parse cleanly: the `#ifdef __cplusplus` / `extern "C" {` compatibility guard in C headers, iterator macros in statement position (`list_for_each_entry(pos, head, member) { … }` and the whole Linux-kernel/git/jemalloc family, braced or single-statement), the Linux/sparse declaration annotations (`static int __init foo(void)`, `void __user *buf`, `container_of(p, struct T, m)`), trailing parameter annotations (`int argc UNUSED`, git's house style), namespace-management macros alone on a line (`FMT_BEGIN_NAMESPACE`, Qt's `Q_OBJECT`), and function attribute macros in front of C++ return types. On git's own repository this nearly doubles the number of indexed symbols, and on the Linux kernel's `kernel/` and `mm/` directories it triples them; blast radius and callers get correspondingly more complete. A related fix stops the existing macro handling from corrupting `#define` lines that mention the same macro names, which removed a class of phantom parse errors in fmt-style headers.
|
||||
- C++ methods defined out-of-line inside a namespace (`namespace sim { Output MyClass::Apply(...) { ... } }`) now carry the namespace in their qualified name, matching their class. Fully-qualified call sites from other files (`sim::MyClass::Apply(...)`) resolve to the definition again, so `codegraph callers` and file impact no longer come up empty for this pattern. (#1291)
|
||||
- C++ methods defined out-of-line on a template class (`template <typename T> T Box<T>::get() { ... }`) no longer keep the template parameter list in their qualified name. They now index as `Box::get` — identical to an inline definition of the same method — so they link to their class and resolve from call sites again, and pathological multi-line template parameter lists can no longer blow the qualified name past filesystem name limits. (#1286)
|
||||
- Go route detection no longer misidentifies ordinary method calls that share HTTP verb names — `cache.Put("key", value)`, `store.Get("config", out)`, `bus.Handle("user.created", handler)` and the like were being indexed as HTTP routes, polluting route listings in cache-heavy codebases. A registration now has to look like one: its first argument must be a `/`-prefixed path (all routers) or a Go 1.22 `"METHOD /path"` pattern on `Handle`/`HandleFunc`, which now also extracts the method instead of listing the route as `ANY`. (#1259)
|
||||
|
||||
@@ -11162,3 +11162,139 @@ import DataStore from '../data/DataStore';
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// R7a preParse additions — the blanking passes added so macro-heavy C/C++
|
||||
// parses clean enough for the kernel route (each also improves the wasm
|
||||
// path's own graphs). Offset preservation is load-bearing everywhere.
|
||||
describe('C/C++ kernel-port preParse blanks (R7a)', () => {
|
||||
it('blankCCplusplusGuardBodies blanks extern-C guard bodies, keeps directives', async () => {
|
||||
const { blankCCplusplusGuardBodies } = await import('../src/extraction/languages/c-cpp');
|
||||
const src = [
|
||||
'#ifdef __cplusplus',
|
||||
'extern "C" {',
|
||||
'#endif',
|
||||
'int real_decl(void);',
|
||||
'#ifdef __cplusplus',
|
||||
'}',
|
||||
'#endif',
|
||||
'',
|
||||
].join('\n');
|
||||
const out = blankCCplusplusGuardBodies(src);
|
||||
expect(out.length).toBe(src.length);
|
||||
expect(out).not.toContain('extern "C"');
|
||||
expect(out).toContain('#ifdef __cplusplus'); // directives stay
|
||||
expect(out).toContain('int real_decl(void);');
|
||||
// A guard with a nested directive bails (needs real preprocessing).
|
||||
const nested = [
|
||||
'#ifdef __cplusplus',
|
||||
'#define EXTERNC extern "C"',
|
||||
'#endif',
|
||||
'',
|
||||
].join('\n');
|
||||
expect(blankCCplusplusGuardBodies(nested)).toBe(nested);
|
||||
// The `#ifndef` inverse guard is C-visible and must be untouched.
|
||||
const inverse = ['#ifndef __cplusplus', 'int c_only(void);', '#endif', ''].join('\n');
|
||||
expect(blankCCplusplusGuardBodies(inverse)).toBe(inverse);
|
||||
});
|
||||
|
||||
it('blankLoneMacroLines blanks namespace-management macros, spares expression operands', async () => {
|
||||
const { blankLoneMacroLines } = await import('../src/extraction/languages/c-cpp');
|
||||
const src = ['FMT_BEGIN_NAMESPACE', 'struct S { int x; };', 'FMT_END_NAMESPACE', ''].join('\n');
|
||||
const out = blankLoneMacroLines(src);
|
||||
expect(out.length).toBe(src.length);
|
||||
expect(out).not.toContain('FMT_BEGIN_NAMESPACE');
|
||||
expect(out).toContain('struct S { int x; };');
|
||||
// An ALL-CAPS operand alone on a line inside a multi-line expression is
|
||||
// NOT a lone macro — the next line starts with an operator.
|
||||
const expr = ['int x = 0', ' | FLAG_ONE', ' | FLAG_TWO;', ''].join('\n');
|
||||
expect(blankLoneMacroLines(expr)).toBe(expr);
|
||||
const cont = ['int y =', 'SOME_FLAG', '| OTHER;', ''].join('\n');
|
||||
expect(blankLoneMacroLines(cont)).toBe(cont);
|
||||
// Underscore-free solid words are too risky and stay.
|
||||
const bare = ['NDEBUG', 'int z;', ''].join('\n');
|
||||
expect(blankLoneMacroLines(bare)).toBe(bare);
|
||||
});
|
||||
|
||||
it('blankCStatementMacroCalls blanks indented iterator macros, keeps the block', async () => {
|
||||
const { blankCStatementMacroCalls } = await import('../src/extraction/languages/c-cpp');
|
||||
const src = [
|
||||
'static void walk(struct list *head) {',
|
||||
'\tlist_for_each_entry(pos, head, member) {',
|
||||
'\t\tuse(pos);',
|
||||
'\t}',
|
||||
'}',
|
||||
'',
|
||||
].join('\n');
|
||||
const out = blankCStatementMacroCalls(src);
|
||||
expect(out.length).toBe(src.length);
|
||||
expect(out).not.toContain('list_for_each_entry');
|
||||
expect(out).toContain('use(pos);');
|
||||
// A real call statement ends with `;` — untouched.
|
||||
expect(out).toContain('use(pos);');
|
||||
const call = ['void f(void) {', '\tdo_thing(a, b);', '}', ''].join('\n');
|
||||
expect(blankCStatementMacroCalls(call)).toBe(call);
|
||||
// Column-0 `name(args) {` is an implicit-int function definition — untouched.
|
||||
const kandr = ['main(argc, argv)', '{', '\treturn 0;', '}', ''].join('\n');
|
||||
expect(blankCStatementMacroCalls(kandr)).toBe(kandr);
|
||||
// Control-flow keywords are never macros.
|
||||
const ctrl = ['void g(int x) {', '\twhile (x) {', '\t\tx--;', '\t}', '}', ''].join('\n');
|
||||
expect(blankCStatementMacroCalls(ctrl)).toBe(ctrl);
|
||||
});
|
||||
|
||||
it('blankCTrailingParamAttrMacros blanks `name UNUSED` params, spares call args', async () => {
|
||||
const { blankCTrailingParamAttrMacros } = await import('../src/extraction/languages/c-cpp');
|
||||
const src = 'static int run(int argc UNUSED, const char **argv UNUSED)\n{\n\treturn 0;\n}\n';
|
||||
const out = blankCTrailingParamAttrMacros(src);
|
||||
expect(out.length).toBe(src.length);
|
||||
expect(out).not.toContain('UNUSED');
|
||||
expect(out).toContain('int argc ');
|
||||
// A macro CONSTANT as a call argument is preceded by `,`/`(`, never by a
|
||||
// bare identifier — untouched.
|
||||
const call = 'void f(void) {\n\tconnect(sock, DEFAULT_TIMEOUT);\n}\n';
|
||||
expect(blankCTrailingParamAttrMacros(call)).toBe(call);
|
||||
});
|
||||
|
||||
it('blankCKernelAnnotations blanks sparse/section dunders, spares parameterized ones and real types', async () => {
|
||||
const { blankCKernelAnnotations } = await import('../src/extraction/languages/c-cpp');
|
||||
const src = [
|
||||
'static int __init audit_init(void) { return 0; }',
|
||||
'void copy(void __user *dst, const char *src);',
|
||||
'__bpf_kfunc void bpf_iter_destroy(struct bpf_iter_num *it);',
|
||||
'__printf(1, 2) void log_fmt(const char *fmt, ...);',
|
||||
'struct e *entry = container_of(r, struct audit_entry, rule);',
|
||||
'__u32 count = 0;',
|
||||
'',
|
||||
].join('\n');
|
||||
const out = blankCKernelAnnotations(src);
|
||||
expect(out.length).toBe(src.length);
|
||||
expect(out).not.toContain('__init');
|
||||
expect(out).not.toContain('__user');
|
||||
expect(out).not.toContain('__bpf_kfunc');
|
||||
// Parameterized annotations keep their name — blanking it would strand
|
||||
// the argument list as a floating parenthesis.
|
||||
expect(out).toContain('__printf(1, 2)');
|
||||
// container_of's type-keyword argument blanks; other `struct` keywords stay.
|
||||
expect(out).toContain('container_of(r, audit_entry, rule)');
|
||||
expect(out).toContain('struct e *entry');
|
||||
// Real dunder TYPES are not annotations.
|
||||
expect(out).toContain('__u32 count');
|
||||
});
|
||||
|
||||
it('restoreDirectiveLines keeps #define lines out of the blanking blast radius', async () => {
|
||||
const { extractFromSource } = await import('../src/extraction');
|
||||
// FMT_API matches the _API-suffix member blank; without the directive
|
||||
// restore the #define loses its NAME and the file gains a parse error.
|
||||
const src = [
|
||||
'#define FMT_API FMT_VISIBILITY("default")',
|
||||
'class Widget {',
|
||||
' public:',
|
||||
' int size() const { return 1; }',
|
||||
'};',
|
||||
'',
|
||||
].join('\n');
|
||||
const result = extractFromSource('lib.hpp', src, 'cpp');
|
||||
expect(result.errors).toEqual([]);
|
||||
expect(result.nodes.some((n) => n.kind === 'class' && n.name === 'Widget')).toBe(true);
|
||||
expect(result.nodes.some((n) => n.kind === 'method' && n.name === 'size')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
/* Torture fixture for the C kernel walker (R7a) — exercises every c-path
|
||||
* branch of the checklist: fn-ptr tables, typedef enum/struct, file-scope
|
||||
* consts incl. multi-declarator, the macro-prototype misparse skip,
|
||||
* value-refs (+ shadow prune), the leading-attr-macro preParse blank, and
|
||||
* the C call shapes. Must parse ERROR-FREE post-preParse or the kernel arm
|
||||
* defers. */
|
||||
#include <stdio.h>
|
||||
#include <sys/socket.h>
|
||||
#include "local_ops.h"
|
||||
|
||||
/** Retry budget for the poller. */
|
||||
static const int MAX_RETRIES = 3;
|
||||
static const int LOW_WATER = 2, HIGH_WATER = 8;
|
||||
static int counter = 0;
|
||||
static const char *BANNER = "torture";
|
||||
|
||||
/* Bare identifier declarators are the macro-prototype misparse shape and are
|
||||
* skipped by design (uninit scalars are the accepted loss). */
|
||||
int bare_global;
|
||||
MYLIB_API config_handle;
|
||||
|
||||
/* Leading attribute macro — blanked by preParseCSource (#1211), so the real
|
||||
* name survives on both arms. */
|
||||
SEC_ATTR UINT32 masked_entry(VOID) { return 0; }
|
||||
|
||||
typedef enum { STATE_IDLE, STATE_RUNNING, STATE_DONE } run_state_t;
|
||||
|
||||
typedef struct {
|
||||
int fd;
|
||||
void (*on_recv)(int);
|
||||
} conn_t;
|
||||
|
||||
typedef struct conn_pool conn_pool_t;
|
||||
|
||||
typedef int (*cb_t)(int);
|
||||
|
||||
enum wire_flags { WIRE_A = 1, WIRE_B = 2 };
|
||||
|
||||
struct packet {
|
||||
int len;
|
||||
unsigned char body[64];
|
||||
struct packet *next;
|
||||
cb_t *async_cb;
|
||||
};
|
||||
|
||||
/** Sum helper (docstring). */
|
||||
static int add(int a, int b) { return a + b; }
|
||||
|
||||
static int cb_a(int x) { return add(x, 1); }
|
||||
static int cb_b(int x) { return add(x, 2); }
|
||||
|
||||
/* fn-ptr table at file scope — the ungated 'list' capture positions. */
|
||||
static cb_t DISPATCH_TABLE[] = { cb_a, cb_b };
|
||||
|
||||
static void handle_recv(int fd);
|
||||
|
||||
/* struct initializer — the ungated 'value' capture positions. */
|
||||
static const struct handler_ops OPS = { .recv = handle_recv, .flags = WIRE_A };
|
||||
|
||||
static void handle_recv(int fd) {
|
||||
struct packet pkt;
|
||||
pkt.len = fd;
|
||||
printf("fd=%d retries=%d\n", fd, MAX_RETRIES);
|
||||
}
|
||||
|
||||
/* Local shadow of a file-scope const — the shadow prune drops HIGH_WATER as a
|
||||
* value-ref target while LOW_WATER stays live. */
|
||||
static int shadowed_reader(void) {
|
||||
int HIGH_WATER = 99;
|
||||
return HIGH_WATER + LOW_WATER;
|
||||
}
|
||||
|
||||
static int use_table(int idx, int v) {
|
||||
cb_t fn = DISPATCH_TABLE[idx];
|
||||
int r = (*fn)(v);
|
||||
conn_t c = { 1, 0 };
|
||||
c.on_recv(r);
|
||||
return counter + r;
|
||||
}
|
||||
|
||||
static void spawn_workers(void) {
|
||||
register_handler(cb_a);
|
||||
signal_connect(&cb_b);
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
/// Torture fixture for the C++ kernel walker (R7a) — namespaces (incl. C++17
|
||||
/// nested + anonymous), out-of-line Cls::method defs, templates + template
|
||||
/// bases, operator definitions, stack construction, local fn-ptrs, UE-macro
|
||||
/// shapes THROUGH the hoisted preParse, using-aliases, access specifiers,
|
||||
/// static-member value reads, and the cpp call shapes. Must parse ERROR-FREE
|
||||
/// post-preParse or the kernel arm defers (spaced operator CALL SITES live in
|
||||
/// torture-defer.cpp — they produce ERROR nodes by design).
|
||||
#include <vector>
|
||||
#include "widget_base.hpp"
|
||||
|
||||
namespace app {
|
||||
|
||||
/** Engine config (docstring). */
|
||||
class Config {
|
||||
public:
|
||||
int retries;
|
||||
void apply();
|
||||
int helper_count() const { return 2; }
|
||||
|
||||
private:
|
||||
int secret;
|
||||
};
|
||||
|
||||
void Config::apply() { retries = helper_count(); }
|
||||
|
||||
namespace detail {
|
||||
struct Counter {
|
||||
int value;
|
||||
Counter *next;
|
||||
};
|
||||
} // namespace detail
|
||||
|
||||
int detail_probe() { return 1; }
|
||||
|
||||
} // namespace app
|
||||
|
||||
namespace app::net {
|
||||
class Session {
|
||||
public:
|
||||
void open();
|
||||
virtual ~Session() {}
|
||||
};
|
||||
void Session::open() {}
|
||||
} // namespace app::net
|
||||
|
||||
namespace {
|
||||
int hidden_helper() { return 3; }
|
||||
} // namespace
|
||||
|
||||
template <typename T>
|
||||
class Base {
|
||||
public:
|
||||
T item;
|
||||
};
|
||||
|
||||
template <typename T>
|
||||
class Box : public Base<T> {
|
||||
public:
|
||||
T get() const { return value_; }
|
||||
T unwrap();
|
||||
|
||||
private:
|
||||
T value_;
|
||||
};
|
||||
|
||||
template <typename T>
|
||||
T Box<T>::unwrap() {
|
||||
return value_;
|
||||
}
|
||||
|
||||
class Derived : public Base<int>, private app::Config {
|
||||
public:
|
||||
Derived() : total_(0) {}
|
||||
int total() const { return total_; }
|
||||
|
||||
private:
|
||||
int total_;
|
||||
};
|
||||
|
||||
struct Vec2 {
|
||||
float x, y;
|
||||
Vec2 operator+(const Vec2 &o) const { return {x + o.x, y + o.y}; }
|
||||
explicit operator bool() const { return x != 0 || y != 0; }
|
||||
Vec2 origin();
|
||||
};
|
||||
|
||||
enum class Mode : unsigned char { Off, On };
|
||||
enum Legacy { LEGACY_A, LEGACY_B };
|
||||
typedef struct {
|
||||
int id;
|
||||
} packet_t;
|
||||
using Handle = app::Config;
|
||||
|
||||
// UE-macro shapes — every one below is recovered by the hoisted preParse
|
||||
// (export macro, reflection markup, inline specifier, API member prefix).
|
||||
class MYMODULE_API Widget : public app::Config {
|
||||
public:
|
||||
UPROPERTY(EditAnywhere, Category = "State")
|
||||
float Health;
|
||||
FORCEINLINE float GetHealth() const { return Health; }
|
||||
ENGINE_API virtual void Tick(float Delta);
|
||||
};
|
||||
|
||||
void Widget::Tick(float Delta) { Health += Delta; }
|
||||
|
||||
Config GlobalConfig;
|
||||
int build_number = 7;
|
||||
|
||||
template <typename T>
|
||||
T compute_seed(T v) {
|
||||
return v + 1;
|
||||
}
|
||||
|
||||
float drive_helper(float v) { return v; }
|
||||
|
||||
Widget *make_widget() { return new Widget(); }
|
||||
|
||||
float drive() {
|
||||
Widget local;
|
||||
app::Config cfg;
|
||||
Vec2 a{1, 2};
|
||||
Vec2 b(a);
|
||||
Vec2 c2(1.5f, 2.5f);
|
||||
float f = a.x + b.y + c2.x;
|
||||
make_widget()->Tick(0.5f);
|
||||
auto kernel = &compute_seed<float>;
|
||||
if (f > 1) {
|
||||
kernel = &drive_helper;
|
||||
}
|
||||
float r = kernel(f);
|
||||
int flags = GlobalConfig.retries;
|
||||
Mode m = Mode::Off;
|
||||
int leg = LEGACY_A;
|
||||
app::detail_probe();
|
||||
compute_seed<int>(2);
|
||||
auto mp = &app::Config::apply;
|
||||
(void)mp;
|
||||
(void)m;
|
||||
return r + f + flags + leg;
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
// Torture header for the C++ kernel walker (R7a) — include guard, forward
|
||||
// declarations (skipped, #1093), extern "C" prototypes, header templates,
|
||||
// and a UE-reflection-shaped class recovered by the hoisted preParse.
|
||||
#ifndef TORTURE_HPP
|
||||
#define TORTURE_HPP
|
||||
|
||||
class Forward;
|
||||
struct Opaque;
|
||||
|
||||
extern "C" {
|
||||
int c_bridge(int value);
|
||||
}
|
||||
|
||||
/// Reusable clamp helper.
|
||||
template <typename T>
|
||||
T clamp_value(T v, T lo, T hi) {
|
||||
return v < lo ? lo : (v > hi ? hi : v);
|
||||
}
|
||||
|
||||
class MYLIB_API Meter : public Forward {
|
||||
public:
|
||||
UPROPERTY(BlueprintReadOnly)
|
||||
int Reading;
|
||||
FORCEINLINE int Peek() const { return Reading; }
|
||||
void Calibrate(int target);
|
||||
Forward *owner();
|
||||
};
|
||||
|
||||
inline void Meter::Calibrate(int target) { Reading = clamp_value(target, 0, 100); }
|
||||
|
||||
#endif
|
||||
@@ -0,0 +1,182 @@
|
||||
/**
|
||||
* Kernel↔wasm C/C++ extraction parity (R7a of the kernel migration).
|
||||
*
|
||||
* Asserts the native walker (codegraph-kernel/src/ccpp/) produces the SAME
|
||||
* ExtractionResult as the wasm TreeSitterExtractor — nodes, edges, and
|
||||
* unresolved refs compared as canonicalized multisets — over:
|
||||
* - the checked-in torture fixtures (torture.c / torture.cpp / torture.hpp:
|
||||
* fn-ptr tables, typedef enum/struct, multi-declarator consts, namespaces
|
||||
* incl. C++17 nested, out-of-line Cls::method defs, templates + template
|
||||
* bases, operators, stack construction, local fn-ptrs, UE-macro shapes
|
||||
* through the hoisted preParse, using-aliases, value-ref shadowing), and
|
||||
* - Metal/CUDA-shaped sources arriving as language 'cpp' — pinning that the
|
||||
* route point applies the SAME extension/content-gated preParse blanks to
|
||||
* the kernel arm (docs/design/ccpp-kernel-port-checklist.md, decision 1/2).
|
||||
*
|
||||
* Files with parse errors — including the spaced explicit-operator CALL-SITE
|
||||
* shape (#1247), which rides an ERROR node — must DEFER to wasm (`defer:`),
|
||||
* asserted below. The full-repo sweep lives in scripts/kernel-parity.mjs
|
||||
* (redis/git/fmt et al., run for the §5 gate); this suite keeps the invariant
|
||||
* alive in `npm test`. Skips when no kernel binary is staged;
|
||||
* CODEGRAPH_KERNEL_EXPECT=1 turns that into a failure (kernel-scaffold.test.ts).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { extractFromSource } from '../src/extraction';
|
||||
import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars';
|
||||
import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel';
|
||||
import type { ExtractionResult, Language } from '../src/types';
|
||||
|
||||
const KERNEL_PATH = path.join(
|
||||
__dirname,
|
||||
'..',
|
||||
'codegraph-kernel',
|
||||
'prebuilds',
|
||||
`${process.platform}-${process.arch}`,
|
||||
'codegraph-kernel.node'
|
||||
);
|
||||
const kernelBuilt = fs.existsSync(KERNEL_PATH);
|
||||
|
||||
const FIXTURE_DIR = path.join(__dirname, 'fixtures', 'kernel-parity');
|
||||
|
||||
function canon(result: ExtractionResult): { nodes: string[]; edges: string[]; refs: string[] } {
|
||||
return {
|
||||
nodes: result.nodes
|
||||
.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort()))
|
||||
.sort(),
|
||||
edges: result.edges.map((e) => JSON.stringify(e, Object.keys(e).sort())).sort(),
|
||||
refs: result.unresolvedReferences
|
||||
.map((r) => JSON.stringify(r, Object.keys(r).sort()))
|
||||
.sort(),
|
||||
};
|
||||
}
|
||||
|
||||
const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const;
|
||||
let savedEnv: Record<string, string | undefined>;
|
||||
|
||||
describe.skipIf(!kernelBuilt)('kernel C/C++ extraction parity', () => {
|
||||
beforeAll(async () => {
|
||||
await initGrammars();
|
||||
await loadGrammarsForLanguages(['c', 'cpp']);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]]));
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
for (const k of ENV_KEYS) {
|
||||
if (savedEnv[k] === undefined) delete process.env[k];
|
||||
else process.env[k] = savedEnv[k];
|
||||
}
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
function assertParity(filePath: string, source: string, language: Language, minNodes = 3): void {
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
const viaKernel = tryKernelExtract(filePath, source, language);
|
||||
expect(viaKernel, `kernel extraction failed for ${filePath}`).not.toBeNull();
|
||||
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource(filePath, source, language);
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
|
||||
const k = canon(viaKernel!);
|
||||
const w = canon(viaWasm);
|
||||
expect(k.nodes, `${filePath}: nodes`).toEqual(w.nodes);
|
||||
expect(k.edges, `${filePath}: edges`).toEqual(w.edges);
|
||||
expect(k.refs, `${filePath}: refs`).toEqual(w.refs);
|
||||
// Meaningful comparison, not empty-vs-empty (the inline Metal/CUDA
|
||||
// sources are deliberately small — they pass their exact node count).
|
||||
expect(viaWasm.nodes.length).toBeGreaterThanOrEqual(minNodes);
|
||||
}
|
||||
|
||||
it('torture fixture (c): fn-ptr tables, typedefs, file-scope consts, value-refs', () => {
|
||||
const file = path.join(FIXTURE_DIR, 'torture.c');
|
||||
assertParity('fixtures/torture.c', fs.readFileSync(file, 'utf8'), 'c');
|
||||
});
|
||||
|
||||
it('torture fixture (cpp): namespaces, out-of-line methods, templates, fn-ptrs, UE macros', () => {
|
||||
const file = path.join(FIXTURE_DIR, 'torture.cpp');
|
||||
assertParity('fixtures/torture.cpp', fs.readFileSync(file, 'utf8'), 'cpp');
|
||||
});
|
||||
|
||||
it('torture fixture (hpp): fwd decls, extern "C", header templates, reflection markup', () => {
|
||||
const file = path.join(FIXTURE_DIR, 'torture.hpp');
|
||||
assertParity('fixtures/torture.hpp', fs.readFileSync(file, 'utf8'), 'cpp');
|
||||
});
|
||||
|
||||
// Metal rides the cpp route: `.metal` maps to language 'cpp' and the
|
||||
// extension-gated `[[attribute]]` blank must reach the kernel arm through
|
||||
// the route-point preParse hoist (filePath rides along for the gate).
|
||||
it('metal-shaped source (.metal → cpp): attribute blanks applied on both arms', () => {
|
||||
const metal = [
|
||||
'struct VertexIn {',
|
||||
' float3 position [[attribute(0)]];',
|
||||
' float2 uv [[attribute(1)]];',
|
||||
'};',
|
||||
'static float2 scale_uv(float2 uv) { return uv; }',
|
||||
'',
|
||||
].join('\n');
|
||||
assertParity('fixtures/shader.metal', metal, 'cpp');
|
||||
});
|
||||
|
||||
// CUDA rides the cpp route too: specifier + launch-config blanks are gated
|
||||
// by extension OR content, and both fire before the kernel call.
|
||||
it('cuda-shaped source (.cu → cpp): specifier + launch blanks applied on both arms', () => {
|
||||
const cuda = [
|
||||
'__global__ void step_kernel(float *data) { data[0] += 1.0f; }',
|
||||
'void launch(float *data) { step_kernel<<<1, 256>>>(data); }',
|
||||
'',
|
||||
].join('\n');
|
||||
assertParity('fixtures/kern.cu', cuda, 'cpp');
|
||||
});
|
||||
|
||||
// Every torture fixture again with CRLF line endings — the shape every
|
||||
// Windows autocrlf checkout has. Derived in memory (not a checked-in CRLF
|
||||
// file) so no platform or editor can silently normalize it away. Pins the
|
||||
// JS-multiline-^ docstring semantics for the C comment markers (#1329).
|
||||
it.each([
|
||||
['torture.c', 'c'],
|
||||
['torture.cpp', 'cpp'],
|
||||
['torture.hpp', 'cpp'],
|
||||
] as const)('torture fixture CRLF parity: %s', (name, lang) => {
|
||||
const file = path.join(FIXTURE_DIR, name);
|
||||
const crlf = fs.readFileSync(file, 'utf8').replace(/(?<!\r)\n/g, '\r\n');
|
||||
assertParity(`fixtures/${name} (crlf)`, crlf, lang);
|
||||
});
|
||||
|
||||
it('spaced explicit-operator call sites defer to the wasm extractor (#1247 rides an ERROR node)', () => {
|
||||
const source = [
|
||||
'struct It { int operator*() const { return 1; } };',
|
||||
'int read_it(const It &it) { return it.operator *(); }',
|
||||
'',
|
||||
].join('\n');
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(tryKernelExtract('src/op.cpp', source, 'cpp')).toBeNull();
|
||||
// The seam still serves the file — through the wasm path, where the
|
||||
// operator-call recovery emits the `it.operator*` ref.
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource('src/op.cpp', source, 'cpp');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(
|
||||
viaWasm.unresolvedReferences.some((r) => r.referenceName === 'it.operator*')
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it('files with parse errors defer to the wasm extractor (recovery is encoding-dependent)', () => {
|
||||
const broken = 'void f( {\n return }} 12 (\n';
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(tryKernelExtract('src/broken.c', broken, 'c')).toBeNull();
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource('src/broken.c', broken, 'c');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(viaWasm.nodes.some((n) => n.kind === 'file')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -36,7 +36,7 @@ const kernelBuilt = fs.existsSync(KERNEL_PATH);
|
||||
|
||||
// Every kernel-capable language. `jsx` shares the javascript grammar on BOTH
|
||||
// paths (langs.rs mirrors WASM_GRAMMAR_FILES), so the distinct grammars are:
|
||||
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java', 'python', 'go'];
|
||||
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java', 'python', 'go', 'c', 'cpp'];
|
||||
|
||||
describe.skipIf(!kernelBuilt)('kernel↔wasm grammar parity', () => {
|
||||
beforeAll(async () => {
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
target/
|
||||
# Cross-compile scratch dirs (e.g. target-linux for the cg1212 envelope runs)
|
||||
target-*/
|
||||
prebuilds/
|
||||
*.node
|
||||
|
||||
Generated
+22
@@ -52,6 +52,8 @@ dependencies = [
|
||||
"regex",
|
||||
"sha2",
|
||||
"tree-sitter",
|
||||
"tree-sitter-c",
|
||||
"tree-sitter-cpp",
|
||||
"tree-sitter-go",
|
||||
"tree-sitter-java",
|
||||
"tree-sitter-javascript",
|
||||
@@ -482,6 +484,26 @@ dependencies = [
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-c"
|
||||
version = "0.24.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a9b2eb57a55fed6b00812912e730b7a275cf4fe98bfd6a5d76263d4438371728"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-cpp"
|
||||
version = "0.23.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "df2196ea9d47b4ab4a31b9297eaa5a5d19a0b121dceb9f118f6790ad0ab94743"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-go"
|
||||
version = "0.23.4"
|
||||
|
||||
@@ -25,6 +25,11 @@ tree-sitter-javascript = "0.25"
|
||||
tree-sitter-java = "0.23"
|
||||
tree-sitter-python = "0.23"
|
||||
tree-sitter-go = "0.23"
|
||||
# Pinned exact: the vendored wasm (src/extraction/wasm/) was built from these
|
||||
# tags' checked-in parser.c, sha-matched against these registry tarballs
|
||||
# (R7a prep, #1345). A patch bump here without re-vendoring breaks the match.
|
||||
tree-sitter-c = "=0.24.2"
|
||||
tree-sitter-cpp = "=0.23.4"
|
||||
|
||||
[build-dependencies]
|
||||
napi-build = "2"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -15,8 +15,8 @@ use tree_sitter::Language;
|
||||
|
||||
/// Languages this kernel binary can extract (reported by contractInfo;
|
||||
/// TS-side routing policy decides what actually routes).
|
||||
pub const LANGUAGES: [&str; 7] =
|
||||
["typescript", "tsx", "javascript", "jsx", "java", "python", "go"];
|
||||
pub const LANGUAGES: [&str; 9] =
|
||||
["typescript", "tsx", "javascript", "jsx", "java", "python", "go", "c", "cpp"];
|
||||
|
||||
pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
match language {
|
||||
@@ -26,6 +26,11 @@ pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
"java" => Some(tree_sitter_java::LANGUAGE.into()),
|
||||
"python" => Some(tree_sitter_python::LANGUAGE.into()),
|
||||
"go" => Some(tree_sitter_go::LANGUAGE.into()),
|
||||
// `.metal`/`.cu`/`.cuh` map to language 'cpp' at detectLanguage, so the
|
||||
// dialects ride this grammar too (their blanking pre-passes stay
|
||||
// TS-side — the route point applies preParse before the kernel call).
|
||||
"c" => Some(tree_sitter_c::LANGUAGE.into()),
|
||||
"cpp" => Some(tree_sitter_cpp::LANGUAGE.into()),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
#![deny(clippy::all)]
|
||||
|
||||
mod buffers;
|
||||
mod ccpp;
|
||||
mod docstring;
|
||||
mod ids;
|
||||
mod go;
|
||||
@@ -103,6 +104,7 @@ pub fn extract_file(file_path: String, content: String, language: String) -> Res
|
||||
"java" => java::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"python" => python::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"go" => go::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"c" | "cpp" => ccpp::extract(&file_path, &content, &language).map_err(Error::from_reason)?,
|
||||
_ => tsjs::extract(&file_path, &content, &language).map_err(Error::from_reason)?,
|
||||
};
|
||||
Ok(ExtractBuffers {
|
||||
|
||||
@@ -1,17 +1,70 @@
|
||||
# C/C++ kernel port (R7a) — the bug-for-bug checklist
|
||||
|
||||
**Status:** survey COMPLETE; grammars VENDORED + suite-green (2026-07-17):
|
||||
tree-sitter-c v0.24.2 (`b780e47`, parser.c `f2883ff9…`) + tree-sitter-cpp
|
||||
v0.23.4 (`f41e1a0`, parser.c `2a35a43b…`, scanner.c `cf60387d…`), built with
|
||||
ts-cli 0.25.10 from checked-in parser.c, in `src/extraction/wasm/` +
|
||||
VENDORED_WASM_LANGS. The walker PR adds the SAME-version crates + kernel
|
||||
grammar registry (kernel-grammar-parity then pins the alignment). Walker +
|
||||
gates not started.
|
||||
This is §0a-recipe step 1's output for c/cpp: every TS-side branch the walker
|
||||
must mirror, with file:line anchors into the reference implementation. Read it
|
||||
WITH `docs/design/rust-kernel-migration-plan.md` (§0a recipe, §5 gates).
|
||||
Companion walkers to crib structure from: `codegraph-kernel/src/tsjs/` (multi-
|
||||
dialect module), `java.rs`, `go.rs` (receiver QNs), `python.rs`.
|
||||
**Status: COMPLETE — walker SHIPPED + gates PASSED, c/cpp DEFAULT-ROUTED
|
||||
(2026-07-17).** Walker: `codegraph-kernel/src/ccpp/mod.rs` (one dual-language
|
||||
module, every branch below mirrored; its header comment lists the quirks).
|
||||
Grammars: tree-sitter-c v0.24.2 (`b780e47`, parser.c `f2883ff9…`) +
|
||||
tree-sitter-cpp v0.23.4 (`f41e1a0`, parser.c `2a35a43b…`, scanner.c
|
||||
`cf60387d…`), crates pinned `=exact` in Cargo.toml, wasm vendored from the
|
||||
same tags, kernel-grammar-parity green. preParse HOISTED to the route point
|
||||
(`preParsedSource` in src/extraction/kernel/index.ts — both tryKernelExtract
|
||||
and the raw bulk path), so no blanking ported to Rust.
|
||||
|
||||
**Gate results (2026-07-17):**
|
||||
- Parity sweeps — **0 diffs on every compared file**: redis 592, git 790,
|
||||
fmt 42, protobuf 925, ALS-Community (UE spot-check) 40 files byte-parity.
|
||||
- Full-init dump-diffs — **byte-identical** kernel-arm vs wasm-arm: redis
|
||||
(131,921 dump lines), git (160,844), fmt (35,433), protobuf (780,620),
|
||||
ALS (3,694).
|
||||
- Torture fixtures torture.c/.cpp/.hpp + CRLF variants + Metal/CUDA
|
||||
hoist-parity + defer tests in `__tests__/kernel-ccpp-parity.test.ts`; new
|
||||
preParse blanks unit-tested in extraction.test.ts; full suite green with
|
||||
`CODEGRAPH_KERNEL_EXPECT=1`.
|
||||
- **Deferral-rate guard — CORRECTED BY MEASUREMENT** (the §4f pattern): the
|
||||
<10% bar was calibrated on ts/java/py/go (0–0.42% parse-error incidence).
|
||||
Macro-heavy C/C++ genuinely parses with errors at double-digit file rates
|
||||
(final sweeps: als 9%, git 16.1%, redis 25.3%, protobuf 25.8%, fmt 42% —
|
||||
fmt's template metaprogramming + `.operator[]`-in-decltype shapes are
|
||||
grammar-inherent), and every erroring file defers BY POLICY. Measured with
|
||||
the defer disabled (`CODEGRAPH_KERNEL_CCPP_ERROR_EXTRACT=1`, sweep-only
|
||||
hatch): recovery-divergence is real (21/207 redis, 8/382 git, 9/31 fmt
|
||||
erroring files extract differently across UTF-8/UTF-16), so the defer
|
||||
stays. The sweep harness now takes `--max-deferral` (default 0.1; use 0.5
|
||||
for c/cpp — a broken walker still trips it by deferring ~everything).
|
||||
- **Seven NEW/extended preParse blanks** cut real incidence (from 32%/52%
|
||||
starting points; linux `kernel/`+`mm/` subtrees 79% → 58%), each
|
||||
offset-preserving, TS-side, shared by both arms, and a graph-quality win
|
||||
for the wasm path itself (git 7.1k → 13.3k nodes; the linux subtrees
|
||||
2.4k → 7.2k): `#ifdef __cplusplus` guard bodies, lone macro lines
|
||||
(`FMT_BEGIN_NAMESPACE`, `Q_OBJECT`), C statement iterator macros
|
||||
(`list_for_each_entry(…) { }` and brace-less bodies), C trailing param
|
||||
attrs (`int argc UNUSED`), the curated Linux/sparse annotation list
|
||||
(`__init`/`__user`/… — structural matching is impossible there: `__u32
|
||||
count` is shape-identical; parameterized `__printf(1,2)` guarded out) +
|
||||
`container_of`'s type-keyword argument, leading attr macros extended to
|
||||
cpp, and directive-line restore (stops the older blanks corrupting
|
||||
`#define` lines).
|
||||
- **Defer-reuse (the linux-economics fix):** a deferred file used to pay the
|
||||
pipeline three times — the worker's raw kernel try, extractFromSource's
|
||||
kernel RE-try, then wasm with a third preParse. A one-slot defer memo in
|
||||
the route point short-circuits the repeat kernel attempt and hands the
|
||||
already-blanked source to the wasm fallback (`sourceIsPreParsed`). On
|
||||
linux this + the annotation blanks took the kernel-arm parse-loop from
|
||||
560s (WORSE than the 426s wasm arm) to **356s vs 435s wasm-arm (−18%)**,
|
||||
and the 2c/6GB envelope to **19.1 min kernel-arm vs 22.9 min wasm-arm
|
||||
(−17%)** with a RICHER graph (2,048,295 nodes / 6,406,933 edges; two
|
||||
independent kernel-arm runs byte-same counts).
|
||||
- **Linux-scale dump gate:** kernel-arm and wasm-arm full graphs are
|
||||
**byte-identical** — `dump-graph.mjs` over both 5.2GB DBs:
|
||||
10,444,551 dump lines each, sha256 `cd4182e6…` on both. (Dumping a 2M-node
|
||||
DB needs a big-heap host run — `node --max-old-space-size=16000 …
|
||||
> file` then hash the FILE; the in-container 6GB heap OOMs and a straight
|
||||
pipe at GB scale dies with ENOBUFS, both of which silently hash truncated
|
||||
output as the empty stream.)
|
||||
|
||||
The sections below are §0a-recipe step 1's output — every TS-side branch the
|
||||
walker mirrors, with file:line anchors (as of `705e501`). Read WITH
|
||||
`docs/design/rust-kernel-migration-plan.md` (§0a recipe, §5 gates).
|
||||
|
||||
## Architecture decisions (already made by the plan)
|
||||
|
||||
|
||||
@@ -62,26 +62,44 @@ them are the ORIGINAL plan and carry expectations that measurement later correct
|
||||
Record runs DONE (§7a.2): 2c/6GB 20.4min, 8c/7GB 18.3min NO-OOM — byte-exact.
|
||||
Batch-loop profile round DONE (§7a.3, #1339): countGuard quadratic killed,
|
||||
19.3min. cFnPtr round DONE (§7a.4, #1341): 2.07× standalone, edge set
|
||||
hash-identical, envelope **17.6min (R6 −33%)**. The <10min-on-8c target
|
||||
remains open; levers left, ranked: **R7a C/C++ port (parse 338s — the
|
||||
last big rock)** > backpressure ~120s (checkpoint I/O floor) > E-scan/
|
||||
settle/read-mapping (~70–90s each, approaching honest work) > the 8c
|
||||
re-run formality (est. ~15.5min).
|
||||
- [ ] **R7a. C/C++ port** — STARTED 2026-07-17: the §0a-recipe step-1 survey is
|
||||
COMPLETE — every TS branch, quirk, and helper is enumerated with line
|
||||
anchors in **`docs/design/ccpp-kernel-port-checklist.md`** (read it FIRST;
|
||||
it also fixes the architecture: preParse hoisted to the route point so no
|
||||
blanking ports to Rust, Metal/CUDA stay wasm this round, one dual-lang
|
||||
walker module). Next: grammars (upgrade+vendor from matched tags, suite
|
||||
green BEFORE the walker), then the walker, then the §5 gate ladder.
|
||||
Original scope note: biggest single-language effort; unlocks cg1212's parse
|
||||
expectation (6.2m → ~1.5–2m, 23% of that wall) + CARLA/UE/llvm-class repos;
|
||||
Metal + CUDA ride along (their blanking pre-passes stay TS-side — `preParse`
|
||||
is offset-preserving and the route point can apply it before the kernel call;
|
||||
see the T2 note in `src/extraction/kernel/index.ts`). Largest per-language
|
||||
surface in tree-sitter.ts: namespace prefix stacks (#1291), local fn-pointer
|
||||
tables (#932), operator calls (#1247), stack construction (#1035), macro
|
||||
salvage + `.h` content detection (stays at detectLanguage, upstream — free).
|
||||
hash-identical, envelope **17.6min (R6 −33%)**. R7a landed 2026-07-17:
|
||||
envelope now **19.1min on a substantially RICHER graph** (the new
|
||||
preParse blanks recover previously-error-swallowed code; wasm-arm on
|
||||
the same graph is 22.9min — the 17.6 record was the old smaller graph
|
||||
and isn't directly comparable). The <10min-on-8c target remains open;
|
||||
levers left, ranked: **C/C++ deferral cuts (58% of linux files still
|
||||
defer to wasm — each recovered idiom moves parse toward the native
|
||||
floor)** > backpressure ~120s (checkpoint I/O floor) > E-scan/settle/
|
||||
read-mapping (~70–90s each, approaching honest work) > the 8c re-run
|
||||
formality.
|
||||
- [x] **R7a. C/C++ port** — DONE 2026-07-17, same-day walker+gates after the
|
||||
survey (#1344) and grammar vendoring (#1345). One dual-language walker
|
||||
(`codegraph-kernel/src/ccpp/`), preParse HOISTED to the route point
|
||||
(both tryKernelExtract and the raw bulk path — no blanking ported to
|
||||
Rust; Metal/CUDA ride the cpp route through the same hoist). Gates:
|
||||
parity sweeps **0 diffs** on redis/git/fmt/protobuf/ALS (2,389 files
|
||||
compared); full-init dump-diffs **byte-identical** on all five;
|
||||
DEFAULT_ROUTED += c, cpp. Three measurement corrections recorded in
|
||||
the checklist doc: (1) C/C++ parse-error incidence is 9–42% per repo
|
||||
(vs 0–0.42% for prior languages), so erroring-file deferral is
|
||||
routine, not a broken-kernel signal — the sweep gained
|
||||
`--max-deferral` (0.5 for c/cpp) after confirming recovery-divergence
|
||||
is real with the sweep-only no-defer hatch; (2) seven new/extended
|
||||
TS-side preParse blanks (extern-C guard bodies, lone macro lines,
|
||||
statement iterator macros, trailing `UNUSED` params, the curated
|
||||
Linux/sparse `__init`-family annotations + `container_of` type args,
|
||||
cpp leading-attr, directive-line restore) cut real incidence (linux
|
||||
subtrees 79% → 58%) AND grew the wasm path's own graphs (git
|
||||
7.1k → 13.3k nodes) — so cg1212's "counts must stay
|
||||
2,048,664/6,405,964" expectation is superseded: the graph legitimately
|
||||
changes with the blanks; the invariant is kernel-arm == wasm-arm at
|
||||
every scale (held: five byte-identical dumps + the linux dump-hash
|
||||
pair); (3) at high deferral the kernel arm initially LOST arm-vs-arm
|
||||
on linux (deferred files ran the pipeline 3×) — fixed with the
|
||||
one-slot defer memo + blanked-source reuse; final cg1212 envelope
|
||||
**19.1 min kernel-arm** (parse-loop 560 → 356s; R6 26.4 → P1 17.6 on
|
||||
the old smaller graph → 19.1 on the new richer one:
|
||||
2,048,295 nodes / 6,406,933 edges, two runs byte-same).
|
||||
- [ ] **R7b. Remaining long tail** per the tracker (§4) — ruby/php/csharp/rust/… T1s
|
||||
are now ~1-day-each with the walker pattern; T3 may stay TS forever (fine).
|
||||
- [ ] **P2. Arc 3, graph richness** (§7b) — product-priority call, standard gates.
|
||||
@@ -98,7 +116,8 @@ and has the current build deployed at `/app` (tree at `/work/linux`).
|
||||
|
||||
**What exists:**
|
||||
- `codegraph-kernel/` — napi-rs crate. One WALKER MODULE per language
|
||||
(`tsjs/`, `java.rs`, `python.rs`, `go.rs`) mirroring `TreeSitterExtractor`'s
|
||||
(`tsjs/`, `java.rs`, `python.rs`, `go.rs`, `ccpp/` for c+cpp) mirroring
|
||||
`TreeSitterExtractor`'s
|
||||
per-language paths bug-for-bug; shared `buffers.rs` (wire contract — twin of
|
||||
`src/extraction/kernel/layout.ts`, byte-matched, ABI-versioned), `ids.rs`
|
||||
(sha node ids, test-pinned to `generateNodeId`), `docstring.rs`, `textutil.rs`
|
||||
@@ -106,10 +125,13 @@ and has the current build deployed at `/app` (tree at `/work/linux`).
|
||||
(grammar registry).
|
||||
- `src/extraction/kernel/` — loader (contract-verifies before routing; a stale
|
||||
.node silently degrades to wasm; `CODEGRAPH_KERNEL_DEBUG=1` explains), decode,
|
||||
routing (`DEFAULT_ROUTED` = ts/tsx/js/jsx/java/python/go;
|
||||
`CODEGRAPH_KERNEL_LANGS` REPLACES the set; `CODEGRAPH_KERNEL=0` kills), and the
|
||||
routing (`DEFAULT_ROUTED` = ts/tsx/js/jsx/java/python/go/c/cpp;
|
||||
`CODEGRAPH_KERNEL_LANGS` REPLACES the set; `CODEGRAPH_KERNEL=0` kills), the
|
||||
deferred-decode transport (`tryKernelExtractRaw` → buffers ride to the store
|
||||
worker; files with applicable framework `extract()` hooks keep the decoded path).
|
||||
worker; files with applicable framework `extract()` hooks keep the decoded
|
||||
path), and the **preParse hoist** (`preParsedSource` — a language's
|
||||
offset-preserving `preParse` hook runs before BOTH kernel entry points, so
|
||||
c/cpp/metal/cuda blanking stays TS-side and both arms parse identical bytes).
|
||||
- Gates in-repo: `scripts/kernel-parity.mjs` (per-file kernel↔wasm diff,
|
||||
ORDER-sensitive, full-object; deferral-rate guard), `scripts/dump-graph.mjs`
|
||||
(natural-key full-DB dump for the byte-identical diff),
|
||||
@@ -490,8 +512,8 @@ parity before porting the language.
|
||||
| rust, dart, scala, lua, luau, r | dedicated files | T1 | crates.io (luau/r/scala: verify crate freshness vs our wasm) | Long-tail T1; port opportunistically after the big five. | ☐ |
|
||||
| kotlin | `languages/kotlin.ts` | T1½ | crates.io | Expect/actual pairing is synthesis-side (fine); extraction is clean but validate against a KMP repo. | ☐ |
|
||||
| swift | shared + dedicated branch | T1½ | crates.io | **Trap:** in-class property extraction lives in `tree-sitter.ts`'s DEDICATED branch, not `swift.ts` (#1020 — Alamofire went 0→348 props). Gate on Alamofire. | ☐ |
|
||||
| c, cpp | `languages/c-cpp.ts` | **T2** | crates.io | Keep as TS pre-passes: `blankCppExportMacros`/`blankCppInlineMacros` (UE `class MACRO Name` phantom-function misparse, #1096–#1102, CARLA 440→6), in-body reflection collapse guard (#1206), content-based `.h` C-vs-C++ detection. | ☐ |
|
||||
| metal, cuda | dialects over the cpp grammar | **T2** (rides c/cpp) | crates.io (cpp) | README-listed as first-class languages. Both are dialect-gated cpp: Metal = specifier/`[[attribute]]` blanking (#1121, the preParse-takes-filePath pattern); CUDA = `<<<>>>` blanking + content-gated `.h` (#1172). Their pre-passes must run before the kernel parse or stay TS-side; gate them WITH the c/cpp port, not separately. | ☐ |
|
||||
| c, cpp | `languages/c-cpp.ts` | **T2** | crates.io | **DONE (R7a, 2026-07-17)** — `ccpp/` walker; ALL pre-passes stayed TS-side via the route-point preParse hoist (+6 new blanks added during gating — see the checklist doc); content-based `.h` C-vs-C++ detection stays upstream at detectLanguage. Parity 0-diff + dump byte-identical on redis/git/fmt/protobuf/ALS. | ☑ |
|
||||
| metal, cuda | dialects over the cpp grammar | **T2** (rides c/cpp) | crates.io (cpp) | **DONE (rides R7a)** — `.metal`/`.cu`/`.cuh` map to 'cpp' and their blanks run in the hoisted preParse (filePath rides along for the extension gates); hoist-parity pinned in kernel-ccpp-parity.test.ts + the metal/cuda suites. | ☑ |
|
||||
| objc | `languages/objc.ts` | T2 | crates.io | Rides the c-cpp trap family; RN bridge extraction feeds `rnCrossPlatformEdges` (synthesis-side, fine). | ☐ |
|
||||
| arkts | `languages/arkts.ts` | T2 | **vendored** (harmony-contrib) | Dot-prefixed refs + decorator-gated matching fixed 36,840 wrong edges — that logic must port exactly or stay TS-side. Compile our grammar fork natively. | ☐ |
|
||||
| pascal | `languages/pascal.ts` | T2 | **vendored** | Paired with dfm-extractor (T3); `extractPascalDefProc` indexed lookups. | ☐ |
|
||||
|
||||
@@ -78,5 +78,10 @@ esac
|
||||
|
||||
DEST="$CRATE/prebuilds/$PLATFORM"
|
||||
mkdir -p "$DEST"
|
||||
# rm first so the copy lands on a FRESH inode: overwriting a signed dylib in
|
||||
# place leaves macOS's per-inode signature cache stale, and every process
|
||||
# that then dlopens the staged .node is SIGKILLed at load (the on-disk
|
||||
# signature still verifies, which makes it maddening to diagnose).
|
||||
rm -f "$DEST/codegraph-kernel.node"
|
||||
cp "$LIB" "$DEST/codegraph-kernel.node"
|
||||
echo "[kernel] staged $DEST/codegraph-kernel.node ($(du -h "$DEST/codegraph-kernel.node" | cut -f1))"
|
||||
|
||||
+40
-11
@@ -10,7 +10,14 @@
|
||||
*
|
||||
* Usage:
|
||||
* node scripts/kernel-parity.mjs <file-or-dir>... [--lang typescript,tsx]
|
||||
* [--max-samples N] [--list-files]
|
||||
* [--max-samples N] [--list-files] [--max-deferral 0.1]
|
||||
*
|
||||
* --max-deferral: the broken-kernel backstop (default 0.1). For C/C++ pass
|
||||
* 0.5: macro-heavy C/C++ trees genuinely parse with errors at 10–40% file
|
||||
* rates even after the preParse blanking family (git 19%, protobuf 26%, fmt
|
||||
* 42% — measured 2026-07-17), and every erroring file defers BY POLICY, so
|
||||
* the 10% bar calibrated on the 0–0.4% incidence of ts/java/py/go would fail
|
||||
* healthy sweeps. A broken walker still trips 0.5 (it defers ~everything).
|
||||
*
|
||||
* Requires: npm run build (dist/) and a staged kernel (npm run build:kernel).
|
||||
* Exit code: 0 = parity, 1 = diffs found, 2 = setup error.
|
||||
@@ -28,10 +35,12 @@ const paths = [];
|
||||
let langFilter = null;
|
||||
let maxSamples = 5;
|
||||
let listFiles = false;
|
||||
let maxDeferral = 0.1;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === '--lang') langFilter = new Set(args[++i].split(','));
|
||||
else if (args[i] === '--max-samples') maxSamples = Number(args[++i]);
|
||||
else if (args[i] === '--list-files') listFiles = true;
|
||||
else if (args[i] === '--max-deferral') maxDeferral = Number(args[++i]);
|
||||
else paths.push(args[i]);
|
||||
}
|
||||
if (paths.length === 0) {
|
||||
@@ -39,11 +48,16 @@ if (paths.length === 0) {
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go']);
|
||||
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'c', 'cpp']);
|
||||
const EXTS = new Map([
|
||||
['.ts', 'typescript'], ['.mts', 'typescript'], ['.cts', 'typescript'],
|
||||
['.tsx', 'tsx'], ['.js', 'javascript'], ['.mjs', 'javascript'],
|
||||
['.cjs', 'javascript'], ['.jsx', 'jsx'], ['.java', 'java'], ['.py', 'python'], ['.pyw', 'python'], ['.go', 'go'],
|
||||
// C/C++ (R7a). `.h` needs CONTENT sniffing (C vs C++) — resolved per file
|
||||
// in the run loop via detectLanguage, matching the real indexer's routing.
|
||||
['.c', 'c'], ['.h', 'detect'],
|
||||
['.cpp', 'cpp'], ['.cc', 'cpp'], ['.cxx', 'cpp'], ['.hpp', 'cpp'], ['.hxx', 'cpp'],
|
||||
['.metal', 'cpp'], ['.cu', 'cpp'], ['.cuh', 'cpp'],
|
||||
]);
|
||||
|
||||
/** Collect candidate files. */
|
||||
@@ -60,7 +74,12 @@ function collect(p, out) {
|
||||
for (const e of fs.readdirSync(p)) collect(path.join(p, e), out);
|
||||
} else if (EXTS.has(path.extname(p))) {
|
||||
const lang = EXTS.get(path.extname(p));
|
||||
if (!langFilter || langFilter.has(lang)) out.push({ file: p, lang });
|
||||
// 'detect' (.h) resolves per file in the run loop; under --lang it rides
|
||||
// along whenever either C-family language is requested.
|
||||
const passes =
|
||||
!langFilter ||
|
||||
(lang === 'detect' ? langFilter.has('c') || langFilter.has('cpp') : langFilter.has(lang));
|
||||
if (passes) out.push({ file: p, lang });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -73,7 +92,7 @@ if (files.length === 0) {
|
||||
|
||||
// --- load the built engine ---------------------------------------------------
|
||||
const { extractFromSource } = await import(dist('extraction/tree-sitter.js'));
|
||||
const { initGrammars, loadGrammarsForLanguages } = await import(dist('extraction/grammars.js'));
|
||||
const { initGrammars, loadGrammarsForLanguages, detectLanguage } = await import(dist('extraction/grammars.js'));
|
||||
const kernel = await import(dist('extraction/kernel/index.js'));
|
||||
|
||||
await initGrammars();
|
||||
@@ -156,13 +175,19 @@ function report(category, sample) {
|
||||
let filesWithDiffs = 0;
|
||||
let filesOk = 0;
|
||||
let deferred = 0;
|
||||
let processed = 0; // collected files minus content-detect skips
|
||||
let totals = { nodes: 0, edges: 0, refs: 0 };
|
||||
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
|
||||
for (const { file, lang } of files) {
|
||||
for (const { file, lang: extLang } of files) {
|
||||
const source = fs.readFileSync(file, 'utf8');
|
||||
const rel = path.relative(ROOT, file);
|
||||
// `.h` resolves C vs C++ by content — the same call the indexer makes.
|
||||
const lang = extLang === 'detect' ? detectLanguage(rel, source) : extLang;
|
||||
if (!KERNEL_LANGS.has(lang)) continue;
|
||||
if (langFilter && !langFilter.has(lang)) continue;
|
||||
processed++;
|
||||
|
||||
delete process.env.CODEGRAPH_KERNEL; // kernel path on
|
||||
const kres = kernel.tryKernelExtract(rel, source, lang);
|
||||
@@ -222,7 +247,7 @@ for (const { file, lang } of files) {
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n=== kernel parity: ${filesOk}/${files.length} files byte-parity` +
|
||||
console.log(`\n=== kernel parity: ${filesOk}/${processed} files byte-parity` +
|
||||
` (${filesWithDiffs} with diffs, ${deferred} deferred-to-wasm)` +
|
||||
` | wasm totals: ${totals.nodes} nodes / ${totals.edges} edges / ${totals.refs} refs ===\n`);
|
||||
|
||||
@@ -232,11 +257,15 @@ for (const [cat, { count, samples }] of sorted) {
|
||||
for (const s of samples) console.log(` ${s.length > 400 ? s.slice(0, 400) + '…' : s}`);
|
||||
}
|
||||
|
||||
// Deferrals are per-file parse-error routing (expected, rare). A high rate
|
||||
// means the kernel is broken and hiding behind the fallback — fail loudly.
|
||||
const deferralRate = deferred / files.length;
|
||||
if (deferralRate > 0.1) {
|
||||
console.error(`deferral rate ${(deferralRate * 100).toFixed(1)}% exceeds 10% — kernel likely broken`);
|
||||
// Deferrals are per-file parse-error routing (expected; rare for most
|
||||
// languages, routine for macro-heavy C/C++ — see --max-deferral above). A
|
||||
// rate past the threshold means the kernel is broken and hiding behind the
|
||||
// fallback — fail loudly.
|
||||
const deferralRate = deferred / Math.max(processed, 1);
|
||||
if (deferralRate > maxDeferral) {
|
||||
console.error(
|
||||
`deferral rate ${(deferralRate * 100).toFixed(1)}% exceeds ${(maxDeferral * 100).toFixed(0)}% — kernel likely broken`
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
process.exit(filesWithDiffs > 0 ? 1 : 0);
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
*/
|
||||
|
||||
import type { ExtractionResult, Language } from '../../types';
|
||||
import { EXTRACTORS } from '../languages';
|
||||
import { getKernel, kernelSupports } from './loader';
|
||||
import { decodeExtractBuffers } from './decode';
|
||||
import {
|
||||
@@ -41,6 +42,12 @@ const DEFAULT_ROUTED: ReadonlySet<Language> = new Set<Language>([
|
||||
'java',
|
||||
'python',
|
||||
'go',
|
||||
// R7a (2026-07-17): parity swept 0-diff on redis/git/fmt/protobuf/ALS
|
||||
// (2,389 files compared) + full-init dump-diffs byte-identical; erroring
|
||||
// files defer per-file to wasm (routine for macro-heavy C/C++ — see
|
||||
// scripts/kernel-parity.mjs --max-deferral).
|
||||
'c',
|
||||
'cpp',
|
||||
]);
|
||||
|
||||
/**
|
||||
@@ -55,6 +62,22 @@ const POST_PASSES: Partial<Record<Language, KernelPostPass>> = {
|
||||
// (none yet — R2+)
|
||||
};
|
||||
|
||||
/**
|
||||
* The preParse hoist (checklist §arch-1): languages with an offset-preserving
|
||||
* `preParse` hook (c/cpp macro blanking, csharp #237, metal #1121, cuda #1172)
|
||||
* apply it HERE, before the kernel call, so both arms parse identical blanked
|
||||
* bytes and none of the blanking logic needs a Rust port. The wasm fallback
|
||||
* path is untouched — TreeSitterExtractor applies the same hook itself on the
|
||||
* RAW source it receives, so a kernel error/defer still extracts identically.
|
||||
* Every blank is an equal-length-space replacement, so offsets, lines, and
|
||||
* columns survive; `filePath` rides along for the extension-gated dialect
|
||||
* blanks (`.metal` attributes; `.cu`/`.cuh` + content-gated CUDA).
|
||||
*/
|
||||
function preParsedSource(filePath: string, source: string, language: Language): string {
|
||||
const pre = EXTRACTORS[language]?.preParse;
|
||||
return pre ? pre(source, filePath) : source;
|
||||
}
|
||||
|
||||
function isRouted(language: Language): boolean {
|
||||
const env = process.env.CODEGRAPH_KERNEL_LANGS;
|
||||
if (env === undefined || env === '') return DEFAULT_ROUTED.has(language);
|
||||
@@ -73,6 +96,37 @@ export function kernelRoutes(language: Language): boolean {
|
||||
/** Warned-once registry so a broken language logs a single line, not one per file. */
|
||||
const warned = new Set<string>();
|
||||
|
||||
/**
|
||||
* One-slot defer memo. A file the kernel defers (parse errors → wasm) used to
|
||||
* pay the full pipeline again at every seam: the worker's raw try blanked +
|
||||
* native-parsed it, extractFromSource's kernel try blanked + native-parsed it
|
||||
* AGAIN, and the wasm extractor then re-applied preParse a third time. On a
|
||||
* high-deferral tree (the Linux kernel defers ~79% of files) that waste
|
||||
* dominated the arm's parse phase. The slot remembers the LAST deferred
|
||||
* (file, source, language) so (a) a repeat kernel attempt for the same file
|
||||
* short-circuits to null, and (b) the wasm fallback can reuse the
|
||||
* already-blanked source instead of re-running preParse. Source is matched by
|
||||
* string identity — the worker passes the same string through every seam.
|
||||
*/
|
||||
let deferSlot: { filePath: string; source: string; language: Language; pre: string } | null = null;
|
||||
|
||||
/** The hoisted preParse output for a just-deferred file, if it matches. */
|
||||
export function takeDeferredPreParse(
|
||||
filePath: string,
|
||||
source: string,
|
||||
language: Language
|
||||
): string | null {
|
||||
if (
|
||||
deferSlot &&
|
||||
deferSlot.filePath === filePath &&
|
||||
deferSlot.source === source &&
|
||||
deferSlot.language === language
|
||||
) {
|
||||
return deferSlot.pre;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** The raw table buffers + the cheap facts the orchestrator needs pre-decode. */
|
||||
export interface KernelRawResult {
|
||||
buffers: NonNullable<ExtractionResult['kernelBuffers']>;
|
||||
@@ -96,8 +150,10 @@ export function tryKernelExtractRaw(
|
||||
if (!kernelRoutes(language) || POST_PASSES[language]) return null;
|
||||
const kernel = getKernel();
|
||||
if (!kernel) return null;
|
||||
if (takeDeferredPreParse(filePath, source, language) !== null) return null; // already deferred
|
||||
const pre = preParsedSource(filePath, source, language);
|
||||
try {
|
||||
const buffers = kernel.extractFile(filePath, source, language);
|
||||
const buffers = kernel.extractFile(filePath, pre, language);
|
||||
const meta = buffers.meta;
|
||||
if (meta.readUInt8(LAYOUT_META.version) !== LAYOUT_ABI) {
|
||||
throw new Error(`kernel buffer ABI ${meta.readUInt8(0)} != expected ${LAYOUT_ABI}`);
|
||||
@@ -118,7 +174,10 @@ export function tryKernelExtractRaw(
|
||||
return { buffers, counts, errors };
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
if (message.includes('defer:')) return null;
|
||||
if (message.includes('defer:')) {
|
||||
deferSlot = { filePath, source, language, pre };
|
||||
return null;
|
||||
}
|
||||
if (!warned.has(language)) {
|
||||
warned.add(language);
|
||||
process.stderr.write(
|
||||
@@ -166,13 +225,11 @@ export function tryKernelExtract(
|
||||
if (!kernelRoutes(language)) return null;
|
||||
const kernel = getKernel();
|
||||
if (!kernel) return null;
|
||||
if (takeDeferredPreParse(filePath, source, language) !== null) return null; // already deferred
|
||||
const t0 = Date.now();
|
||||
const pre = preParsedSource(filePath, source, language);
|
||||
try {
|
||||
// NOTE(T2 languages): when a preParse-carrying language (csharp #237,
|
||||
// metal #1121, cuda #1172, c/cpp macro blanking) routes here, its
|
||||
// offset-preserving preParse hook must be applied to `source` first —
|
||||
// wire that alongside the language's port, gated WITH its equivalence run.
|
||||
const buffers = kernel.extractFile(filePath, source, language);
|
||||
const buffers = kernel.extractFile(filePath, pre, language);
|
||||
const result = decodeExtractBuffers(buffers, filePath, language);
|
||||
POST_PASSES[language]?.(result, source);
|
||||
result.durationMs = Date.now() - t0;
|
||||
@@ -182,7 +239,10 @@ export function tryKernelExtract(
|
||||
// `defer:` is the kernel's expected-routing signal (files with parse
|
||||
// errors take the wasm path — its error RECOVERY is the canonical one;
|
||||
// recovery differs between UTF-8 and UTF-16 parsing). Silent by design.
|
||||
if (message.includes('defer:')) return null;
|
||||
if (message.includes('defer:')) {
|
||||
deferSlot = { filePath, source, language, pre };
|
||||
return null;
|
||||
}
|
||||
if (!warned.has(language)) {
|
||||
warned.add(language);
|
||||
process.stderr.write(
|
||||
|
||||
@@ -520,6 +520,63 @@ export function blankCppAnnotationMacroCalls(source: string): string {
|
||||
return chars.join('');
|
||||
}
|
||||
|
||||
/**
|
||||
* Blank a macro that is the ONLY token on its line — no parens, no semicolon:
|
||||
* namespace-management macros (`FMT_BEGIN_NAMESPACE`, `FMT_END_EXPORT`,
|
||||
* `JEMALLOC_DIAGNOSTIC_DISABLE_SPURIOUS`), Qt's `Q_OBJECT`, and friends. A
|
||||
* bare identifier is not a statement or declaration in C or C++, so
|
||||
* tree-sitter drops into error recovery at every one — and since the kernel
|
||||
* path defers ANY erroring file to wasm, this single idiom deferred 13/73 fmt
|
||||
* files and a comparable share of jemalloc, forfeiting the native-parse win
|
||||
* on exactly the header-heavy trees it targets (the wasm path also mis-nests
|
||||
* scopes around them today). Replacing the token with equal-length spaces
|
||||
* preserves every byte offset and the surrounding declarations parse clean.
|
||||
*
|
||||
* Matched tightly so a real identifier can never be touched — ALL of:
|
||||
* - the line consists of ONE ALL-CAPS token (≥4 chars, with `_`), optionally
|
||||
* followed by a same-line comment — a lone lowercase identifier or any
|
||||
* second token disqualifies;
|
||||
* - the PREVIOUS non-blank line does not end in a continuation character
|
||||
* (`=`, an operator, `,`, `(`, `?`, `:`, or a `\` macro-definition
|
||||
* continuation) — so an ALL-CAPS operand split onto its own line inside a
|
||||
* multi-line expression (`int x =\n SOME_CONST\n | OTHER;`) is left
|
||||
* alone; and
|
||||
* - the NEXT non-blank line starts like a declaration/scope token
|
||||
* (letter, `_`, `#`, `{`, `}`, or `~`) or the file ends — an operator,
|
||||
* string literal, or `;` continuation rejects the match.
|
||||
* Shared by C and C++ (the idiom is identical in both).
|
||||
*/
|
||||
const LONE_MACRO_LINE_RE = /^[ \t]*([A-Z][A-Z0-9_]{3,})[ \t]*(?:\/\/[^\n\r]*|\/\*[^\n\r]*\*\/[ \t]*)?\r?$/;
|
||||
const LONE_MACRO_CONTINUATION_END_RE = /[=+\-*/%&|^<>?:,(\\]$/;
|
||||
export function blankLoneMacroLines(source: string): string {
|
||||
if (!/^[ \t]*[A-Z][A-Z0-9_]{3,}[ \t]*\r?$/m.test(source)) return source;
|
||||
const lines = source.split('\n');
|
||||
const content = (l: string): string => l.replace(/\r$/, '').trim();
|
||||
let changed = false;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i] as string;
|
||||
const m = LONE_MACRO_LINE_RE.exec(line);
|
||||
if (!m) continue;
|
||||
// Underscore requirement rides the macro convention (FMT_BEGIN_NAMESPACE,
|
||||
// Q_OBJECT); a solid all-caps word (`NDEBUG`-style) alone is too risky.
|
||||
if (!(m[1] as string).includes('_')) continue;
|
||||
let prev = i - 1;
|
||||
while (prev >= 0 && content(lines[prev] as string) === '') prev--;
|
||||
if (prev >= 0 && LONE_MACRO_CONTINUATION_END_RE.test(content(lines[prev] as string))) continue;
|
||||
let next = i + 1;
|
||||
while (next < lines.length && content(lines[next] as string) === '') next++;
|
||||
if (next < lines.length) {
|
||||
const first = content(lines[next] as string)[0];
|
||||
if (!first || !/[A-Za-z_#{}~]/.test(first)) continue;
|
||||
}
|
||||
const start = line.indexOf(m[1] as string);
|
||||
lines[i] =
|
||||
line.slice(0, start) + ' '.repeat((m[1] as string).length) + line.slice(start + (m[1] as string).length);
|
||||
changed = true;
|
||||
}
|
||||
return changed ? lines.join('\n') : source;
|
||||
}
|
||||
|
||||
/**
|
||||
* Blank an export/visibility macro sitting in front of a *member* or *method*
|
||||
* declaration inside a class/namespace (`ENGINE_API virtual void Tick(…)`,
|
||||
@@ -689,24 +746,64 @@ function looksLikeCudaSource(source: string): boolean {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore preprocessor-directive lines to their original bytes after the
|
||||
* blanking passes ran. The token-level blanks match on shape, not context, so
|
||||
* a macro name that happens to sit inside a DIRECTIVE gets blanked too — and
|
||||
* blanking the name position of `#define FMT_API FMT_VISIBILITY("default")`
|
||||
* leaves a nameless `# define FMT_VISIBILITY(…)`, which is a parse
|
||||
* ERROR (fmt's base.h carries several). Inside a directive the blanks were
|
||||
* never useful anyway: tree-sitter stores `#define` bodies as raw
|
||||
* preproc_arg text it doesn't parse, so blanking there can only ever break
|
||||
* the directive itself. Copying the original directive lines back (including
|
||||
* `\`-continuation lines of multi-line defines) is offset-preserving by
|
||||
* construction and strictly reduces parse errors on both extraction arms.
|
||||
*/
|
||||
function restoreDirectiveLines(original: string, blanked: string): string {
|
||||
if (blanked === original || original.indexOf('#') === -1) return blanked;
|
||||
const o = original.split('\n');
|
||||
const b = blanked.split('\n');
|
||||
let changed = false;
|
||||
let continuation: boolean = false;
|
||||
for (let i = 0; i < o.length && i < b.length; i++) {
|
||||
const line = o[i] as string;
|
||||
const isDirective: boolean = continuation || /^[ \t]*#/.test(line);
|
||||
if (isDirective && b[i] !== line) {
|
||||
b[i] = line;
|
||||
changed = true;
|
||||
}
|
||||
continuation = isDirective && /\\\s*$/.test(line.replace(/\r$/, ''));
|
||||
}
|
||||
return changed ? b.join('\n') : blanked;
|
||||
}
|
||||
|
||||
/** C/C++ source pre-processing before tree-sitter: recover macro-annotated class
|
||||
* definitions, macro-prefixed function definitions, macro-prefixed members, and
|
||||
* macro-decorated members (Unreal-Engine reflection markup) — plus the non-C++
|
||||
* surface of the dialects parsed with the C++ grammar: `.metal` MSL attribute
|
||||
* annotations, and CUDA specifiers + launch syntax (by `.cu`/`.cuh` extension
|
||||
* or by content, for CUDA living in `.h`/`.hpp` headers). Offset-preserving. */
|
||||
* or by content, for CUDA living in `.h`/`.hpp` headers). Offset-preserving;
|
||||
* directive lines are restored at the end (see restoreDirectiveLines). */
|
||||
function preParseCppSource(source: string, filePath?: string): string {
|
||||
const blanked = blankCppAnnotationMacroCalls(
|
||||
// blankCLeadingAttrMacros runs AFTER the api-prefix blank so a stacked
|
||||
// `FMT_NORETURN FMT_API void f(…)` reduces to the `MACRO Ret name(` shape
|
||||
// it matches (the _API token is already spaces by then).
|
||||
let blanked = blankLoneMacroLines(
|
||||
blankCLeadingAttrMacros(
|
||||
blankCppAnnotationMacroCalls(
|
||||
blankCppInlineAnnotationMacros(
|
||||
blankCppApiPrefixMacros(blankCppInlineMacros(blankCppExportMacros(source)))
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
const lower = filePath ? filePath.toLowerCase() : '';
|
||||
if (lower.endsWith('.metal')) return blankMetalAttributes(blanked);
|
||||
if (lower.endsWith('.cu') || lower.endsWith('.cuh') || looksLikeCudaSource(source)) {
|
||||
return blankCudaConstructs(blanked);
|
||||
if (lower.endsWith('.metal')) {
|
||||
blanked = blankMetalAttributes(blanked);
|
||||
} else if (lower.endsWith('.cu') || lower.endsWith('.cuh') || looksLikeCudaSource(source)) {
|
||||
blanked = blankCudaConstructs(blanked);
|
||||
}
|
||||
return blanked;
|
||||
return restoreDirectiveLines(source, blanked);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -743,13 +840,259 @@ export function blankCLeadingAttrMacros(source: string): string {
|
||||
);
|
||||
}
|
||||
|
||||
/** C source pre-processing: recover functions hidden behind a leading
|
||||
* attribute macro (#1211), then — for C-detected headers in CUDA projects
|
||||
* (llm.c keeps `__device__` helpers and kernel prototypes in plain `.h`) —
|
||||
* the same content-gated CUDA blank as C++. Offset-preserving. */
|
||||
/**
|
||||
* Blank the body of `#ifdef __cplusplus … #endif` guard regions in C sources.
|
||||
* The ubiquitous C-header compatibility idiom
|
||||
*
|
||||
* #ifdef __cplusplus
|
||||
* extern "C" {
|
||||
* #endif
|
||||
*
|
||||
* is NOT valid C — `extern "C" {` (and any other C++-only line under the
|
||||
* guard) drops tree-sitter-c into error recovery, so effectively every public
|
||||
* C header carries parse errors. The wasm path shrugs (recovery keeps the
|
||||
* rest); the kernel path defers EVERY erroring file to wasm by policy — so
|
||||
* this one idiom pushed C-header deferral to ~32% on redis (vs the <10%
|
||||
* gate) and forfeited the native-parse win exactly where C repos have the
|
||||
* most files. A C compiler never sees the guarded lines (`__cplusplus` is
|
||||
* only defined for C++), so blanking the region BODY mirrors the
|
||||
* preprocessor's own view of the file.
|
||||
*
|
||||
* Matched conservatively, line-based and offset-preserving:
|
||||
* - the opener must be `#ifdef __cplusplus` / `#if defined(__cplusplus)`;
|
||||
* - the body may contain NO other preprocessor directive (a nested `#if`,
|
||||
* `#else`, or `#define` bails the whole region — those need real
|
||||
* preprocessing, so the file keeps its current behavior);
|
||||
* - the region must close with `#endif` within a few lines (guards are
|
||||
* tiny; a giant region is something else).
|
||||
* The `#ifdef`/`#endif` directive lines themselves are kept — an empty
|
||||
* preproc_ifdef parses clean — and every blanked byte becomes a space with
|
||||
* `\r` preserved, so offsets, lines, and columns survive on CRLF checkouts.
|
||||
*/
|
||||
const C_CPLUSPLUS_GUARD_OPEN_RE =
|
||||
/^[ \t]*#[ \t]*(?:ifdef[ \t]+__cplusplus\b|if[ \t]+defined[ \t]*\(?[ \t]*__cplusplus[ \t]*\)?)/;
|
||||
const C_PREPROC_DIRECTIVE_RE = /^[ \t]*#/;
|
||||
const C_PREPROC_ENDIF_RE = /^[ \t]*#[ \t]*endif\b/;
|
||||
const C_CPLUSPLUS_GUARD_MAX_BODY_LINES = 40;
|
||||
export function blankCCplusplusGuardBodies(source: string): string {
|
||||
if (source.indexOf('__cplusplus') === -1) return source;
|
||||
const lines = source.split('\n');
|
||||
const stripCr = (l: string): string => (l.endsWith('\r') ? l.slice(0, -1) : l);
|
||||
let changed = false;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (!C_CPLUSPLUS_GUARD_OPEN_RE.test(stripCr(lines[i] as string))) continue;
|
||||
let end = -1;
|
||||
for (let j = i + 1; j < lines.length && j - i - 1 <= C_CPLUSPLUS_GUARD_MAX_BODY_LINES; j++) {
|
||||
const line = stripCr(lines[j] as string);
|
||||
if (C_PREPROC_ENDIF_RE.test(line)) {
|
||||
end = j;
|
||||
break;
|
||||
}
|
||||
if (C_PREPROC_DIRECTIVE_RE.test(line)) break; // nested directive — bail
|
||||
}
|
||||
if (end < 0) continue;
|
||||
for (let k = i + 1; k < end; k++) {
|
||||
lines[k] = (lines[k] as string).replace(/[^\r]/g, ' ');
|
||||
}
|
||||
changed = true;
|
||||
i = end;
|
||||
}
|
||||
return changed ? lines.join('\n') : source;
|
||||
}
|
||||
|
||||
/**
|
||||
* Blank a C iterator-macro call in STATEMENT position — `ql_foreach(iter,
|
||||
* &arena->tcache_ql, link) { … }` (jemalloc), `for_each_string_list_item(item,
|
||||
* &list) { … }` (git), `list_for_each_entry(pos, head, member) { … }` (the
|
||||
* Linux kernel's core iteration idiom). A call followed by a brace block is
|
||||
* not a C statement, so tree-sitter-c drops into error recovery at every use —
|
||||
* these macros are the single largest source of parse errors in macro-heavy C
|
||||
* trees (git: ~39% of files error; the kernel path defers each one to wasm).
|
||||
* Blanking JUST the macro call leaves the brace block as a bare compound
|
||||
* statement — valid C — so the body's calls/locals extract normally on both
|
||||
* arms instead of riding error recovery.
|
||||
*
|
||||
* C-ONLY, and matched tightly:
|
||||
* - the call must be INDENTED (statement position; file-scope definitions
|
||||
* start at column 0, and an unbraced file-scope `name(args) { }` is a
|
||||
* valid implicit-int function definition that must not be touched);
|
||||
* - lowercase-led identifier (iterator macros are lowercase by convention;
|
||||
* this also excludes constructors if the file is really C++) that is not a
|
||||
* control keyword;
|
||||
* - the parens balance ON the line (string literals skipped), and after
|
||||
* them only `{` or end-of-line may follow — a `;` (a real call statement),
|
||||
* an operator, or any other token disqualifies;
|
||||
* - when the line ends at `)`, the NEXT non-blank line must begin with `{`.
|
||||
* C++ deliberately does NOT get this pass: an indented snake_case
|
||||
* constructor (`basic_string_view(const Char* s) : … {`) is exactly this
|
||||
* shape, and blanking it would corrupt every STL-style class.
|
||||
*/
|
||||
const C_STMT_MACRO_KEYWORDS = new Set([
|
||||
'if', 'while', 'for', 'switch', 'return', 'do', 'else', 'sizeof',
|
||||
]);
|
||||
export function blankCStatementMacroCalls(source: string): string {
|
||||
const lines = source.split('\n');
|
||||
let changed = false;
|
||||
const content = (l: string): string => l.replace(/\r$/, '').trim();
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i] as string;
|
||||
const m = /^[ \t]+([a-z_][a-z0-9_]*)[ \t]*\(/.exec(line);
|
||||
if (!m || C_STMT_MACRO_KEYWORDS.has(m[1] as string)) continue;
|
||||
const open = line.indexOf('(', m[0].length - 1);
|
||||
let depth = 0;
|
||||
let close = -1;
|
||||
for (let k = open; k < line.length; k++) {
|
||||
const ch = line[k];
|
||||
if (ch === '"' || ch === "'") {
|
||||
const quote = ch;
|
||||
k++;
|
||||
while (k < line.length && line[k] !== quote) {
|
||||
if (line[k] === '\\') k++;
|
||||
k++;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (ch === '(') depth++;
|
||||
else if (ch === ')') {
|
||||
depth--;
|
||||
if (depth === 0) {
|
||||
close = k;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (close < 0) continue; // parens don't balance on the line
|
||||
const after = line.slice(close + 1).replace(/\r$/, '').trim();
|
||||
if (after === '') {
|
||||
// Brace on the next line (`ql_foreach(…)\n{`) or a brace-less
|
||||
// single-statement body (`for_each_subsys(ss, i)\n\tstmt;` — blanking
|
||||
// leaves the bare statement, valid C). A next line starting with an
|
||||
// operator/string/`;` is an expression continuation — bail.
|
||||
let next = i + 1;
|
||||
while (next < lines.length && content(lines[next] as string) === '') next++;
|
||||
if (next >= lines.length) continue;
|
||||
const first = content(lines[next] as string)[0];
|
||||
if (!first || !/[A-Za-z_{]/.test(first)) continue;
|
||||
} else if (after !== '{') {
|
||||
continue;
|
||||
}
|
||||
const identStart = line.indexOf(m[1] as string);
|
||||
lines[i] =
|
||||
line.slice(0, identStart) +
|
||||
' '.repeat(close + 1 - identStart) +
|
||||
line.slice(close + 1);
|
||||
changed = true;
|
||||
}
|
||||
return changed ? lines.join('\n') : source;
|
||||
}
|
||||
|
||||
/**
|
||||
* Blank a trailing parameter-attribute macro — `int argc UNUSED,` /
|
||||
* `struct repository *repo UNUSED)` — git's house style for
|
||||
* `__attribute__((unused))` on nearly every callback parameter (and the same
|
||||
* shape as `MAYBE_UNUSED`/`G_GNUC_UNUSED` elsewhere). tree-sitter-c can't
|
||||
* parse a second identifier after the parameter name, so every such
|
||||
* SIGNATURE drops into error recovery — the single largest deferral bucket
|
||||
* on git (~150 files). Blanking the macro leaves an ordinary parameter.
|
||||
*
|
||||
* Matched tightly: an identifier, whitespace, then an ALL-CAPS ≥3-char token
|
||||
* immediately before `,` or `)`. Two juxtaposed identifiers in that position
|
||||
* have no other valid-C reading — in a CALL the would-be macro is preceded
|
||||
* by `,`/`(`, an operator, or a literal, never by a bare identifier. C-only:
|
||||
* C++ grammars accept more juxtapositions (user-defined suffixes, macro'd
|
||||
* `final`/`override`), so cpp keeps its existing recovery there.
|
||||
*/
|
||||
const C_TRAILING_PARAM_ATTR_RE = /\b([A-Za-z_]\w*)([ \t]+)([A-Z][A-Z0-9_]{2,})(?=[ \t]*[,)])/g;
|
||||
export function blankCTrailingParamAttrMacros(source: string): string {
|
||||
if (!C_TRAILING_PARAM_ATTR_RE.test(source)) {
|
||||
C_TRAILING_PARAM_ATTR_RE.lastIndex = 0;
|
||||
return source;
|
||||
}
|
||||
C_TRAILING_PARAM_ATTR_RE.lastIndex = 0;
|
||||
return source.replace(
|
||||
C_TRAILING_PARAM_ATTR_RE,
|
||||
(_m, name: string, ws: string, macro: string) => name + ws + ' '.repeat(macro.length)
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Blank the Linux-kernel/sparse declaration-annotation macros — `static int
|
||||
* __init audit_init(void)`, `void __user *buf`, `__bpf_kfunc void f(…)`,
|
||||
* `int x __ro_after_init;`. These lowercase double-underscore annotations sit
|
||||
* between storage/type tokens and the declarator, a position tree-sitter-c
|
||||
* can't reconcile, and they blanket the Linux tree: measured on the kernel's
|
||||
* own `kernel/` + `mm/` subtrees, they are the largest single deferral cause
|
||||
* (the `__init` family alone heads ~37% of erroring files).
|
||||
*
|
||||
* A structural match is IMPOSSIBLE here: `__u32 count` (a real typedef) and
|
||||
* `__init foo` (an annotation) are byte-shape identical — so unlike the
|
||||
* shape-keyed blanks above, this is a CURATED list (the CPP_INLINE_MACROS
|
||||
* precedent) of well-known sparse/section/compiler annotations that are
|
||||
* reserved-namespace macros in every codebase that spells them. Whole-word,
|
||||
* equal-length spaces, C-only (the C++ grammar's kernel exposure is
|
||||
* negligible and cpp keeps its narrower blank set).
|
||||
*/
|
||||
const C_KERNEL_ANNOTATIONS = [
|
||||
'__init', '__exit', '__initdata', '__initconst', '__exitdata',
|
||||
'__devinit', '__devexit', '__cpuinit', '__meminit', '__meminitdata',
|
||||
'__net_init', '__net_exit', '__init_or_module',
|
||||
'__user', '__kernel', '__iomem', '__percpu', '__rcu', '__force', '__nocast',
|
||||
'__must_check', '__maybe_unused', '__always_unused', '__used', '__cold',
|
||||
'__hot', '__weak', '__pure', '__sched', '__malloc', '__visible',
|
||||
'__deprecated', '__ro_after_init', '__read_mostly', '__refdata',
|
||||
'__latent_entropy', '__randomize_layout', '__no_randomize_layout',
|
||||
'__bpf_kfunc', '__function_aligned', '__always_inline', '__noreturn',
|
||||
] as const;
|
||||
// `(?!\s*\()` keeps the parameterized annotations (`__printf(1, 2)`,
|
||||
// `__aligned(8)`, `__section("x")`) intact — blanking just their name would
|
||||
// strand the argument list as a floating parenthesis and CREATE an error.
|
||||
const C_KERNEL_ANNOTATION_RE = new RegExp(
|
||||
`\\b(${[...C_KERNEL_ANNOTATIONS].sort((a, b) => b.length - a.length).join('|')})\\b(?!\\s*\\()`,
|
||||
'g'
|
||||
);
|
||||
export function blankCKernelAnnotations(source: string): string {
|
||||
if (source.indexOf('__') === -1) return source;
|
||||
C_KERNEL_ANNOTATION_RE.lastIndex = 0;
|
||||
if (!C_KERNEL_ANNOTATION_RE.test(source)) return source;
|
||||
let out = source.replace(C_KERNEL_ANNOTATION_RE, (m) => ' '.repeat(m.length));
|
||||
// `container_of(ptr, struct T, member)` — the type-keyword argument is the
|
||||
// one call shape tree-sitter-c cannot read (a macro taking a TYPE), and it
|
||||
// is pervasive across the Linux tree. Blanking just the `struct`/`union`
|
||||
// keyword leaves `container_of(ptr, T, member)` — a plain
|
||||
// identifier argument the grammar parses natively. Keyed to the macro name
|
||||
// so no other `struct` keyword anywhere is ever touched.
|
||||
if (out.indexOf('container_of') !== -1) {
|
||||
out = out.replace(
|
||||
/(\bcontainer_of\s*\([^;()]*?,\s*)(struct|union)(\s+)/g,
|
||||
(_m, head: string, kw: string, ws: string) => head + ' '.repeat(kw.length) + ws
|
||||
);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** C source pre-processing: neutralize `#ifdef __cplusplus` compat-guard
|
||||
* bodies (invisible to a C compiler; `extern "C" {` otherwise errors every
|
||||
* public header), blank declaration-markup macro calls and lone macro lines
|
||||
* (`REDIS_NO_SANITIZE("bounds")` before a definition, jemalloc's diagnostic
|
||||
* toggles — the same structural shapes the C++ side already blanks), recover
|
||||
* functions hidden behind a leading attribute macro (#1211), then — for
|
||||
* C-detected headers in CUDA projects (llm.c keeps `__device__` helpers and
|
||||
* kernel prototypes in plain `.h`) — the same content-gated CUDA blank as
|
||||
* C++. Offset-preserving. */
|
||||
function preParseCSource(source: string): string {
|
||||
const blanked = blankCLeadingAttrMacros(source);
|
||||
return looksLikeCudaSource(blanked) ? blankCudaConstructs(blanked) : blanked;
|
||||
let blanked = blankCLeadingAttrMacros(
|
||||
blankLoneMacroLines(
|
||||
blankCStatementMacroCalls(
|
||||
blankCTrailingParamAttrMacros(
|
||||
blankCppAnnotationMacroCalls(
|
||||
blankCKernelAnnotations(blankCCplusplusGuardBodies(source))
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
if (looksLikeCudaSource(blanked)) blanked = blankCudaConstructs(blanked);
|
||||
return restoreDirectiveLines(source, blanked);
|
||||
}
|
||||
|
||||
export const cppExtractor: LanguageExtractor = {
|
||||
|
||||
@@ -30,7 +30,7 @@ import { DfmExtractor } from './dfm-extractor';
|
||||
import { VueExtractor } from './vue-extractor';
|
||||
import { MyBatisExtractor } from './mybatis-extractor';
|
||||
import { CfmlExtractor } from './cfml-extractor';
|
||||
import { tryKernelExtract } from './kernel';
|
||||
import { tryKernelExtract, takeDeferredPreParse } from './kernel';
|
||||
import {
|
||||
getAllFrameworkResolvers,
|
||||
getApplicableFrameworks,
|
||||
@@ -429,13 +429,23 @@ export class TreeSitterExtractor {
|
||||
private fnRefCandidates: Array<FnRefCandidate & { fromNodeId: string }> = [];
|
||||
// Memoized "is this a Vue store file" verdict (per-extractor = per-file).
|
||||
private vueStoreFile: boolean | null = null;
|
||||
// Source already went through the extractor's preParse at the kernel route
|
||||
// point (this instance is the wasm fallback for a kernel-deferred file) —
|
||||
// don't blank it a second time.
|
||||
private sourceIsPreParsed = false;
|
||||
|
||||
constructor(filePath: string, source: string, language?: Language) {
|
||||
constructor(
|
||||
filePath: string,
|
||||
source: string,
|
||||
language?: Language,
|
||||
options?: { sourceIsPreParsed?: boolean }
|
||||
) {
|
||||
this.filePath = filePath;
|
||||
this.source = source;
|
||||
this.language = language || detectLanguage(filePath, source);
|
||||
this.extractor = EXTRACTORS[this.language] || null;
|
||||
this.fnRefSpec = FN_REF_SPECS[this.language];
|
||||
this.sourceIsPreParsed = options?.sourceIsPreParsed === true;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -484,8 +494,9 @@ export class TreeSitterExtractor {
|
||||
// grammar gaps — e.g. C# blanks conditional-compilation directive lines
|
||||
// the grammar mis-parses inside enum bodies (#237). We reassign
|
||||
// this.source so downstream getNodeText reads the same bytes the parser
|
||||
// saw (identical outside the blanked directive lines).
|
||||
if (this.extractor?.preParse) {
|
||||
// saw (identical outside the blanked directive lines). Skipped when the
|
||||
// kernel route point already applied it (sourceIsPreParsed).
|
||||
if (this.extractor?.preParse && !this.sourceIsPreParsed) {
|
||||
this.source = this.extractor.preParse(this.source, this.filePath);
|
||||
}
|
||||
this.tree = parser.parse(this.source) ?? null;
|
||||
@@ -6708,7 +6719,15 @@ export function extractFromSource(
|
||||
if (kernelResult) {
|
||||
result = kernelResult;
|
||||
} else {
|
||||
const extractor = new TreeSitterExtractor(filePath, source, detectedLanguage);
|
||||
// A kernel-deferred file already paid the (offset-preserving) preParse
|
||||
// at the route point — reuse those bytes instead of blanking again.
|
||||
const deferredPre = takeDeferredPreParse(filePath, source, detectedLanguage);
|
||||
const extractor = new TreeSitterExtractor(
|
||||
filePath,
|
||||
deferredPre ?? source,
|
||||
detectedLanguage,
|
||||
{ sourceIsPreParsed: deferredPre != null }
|
||||
);
|
||||
result = extractor.extract();
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user