feat(kernel): R1 scaffold — napi-rs extraction kernel, buffer contract, routing + fallback, grammar-parity CI
Phase 0 of the Rust extraction-kernel migration (docs/design/ rust-kernel-migration-plan.md, now checked in with §3a recording the shipped state): - codegraph-kernel/ napi-rs crate: extractFile(path, content, language) → five flat buffers (meta/nodes/edges/refs/arena), one JS boundary crossing per file. Node ids computed Rust-side, byte-identical to generateNodeId (pinned by test vector). Reserved per-node metrics slot for the Arc 3.2 code-metrics work. - Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures, byte-range scope stack → ::-joined qualified names, contains edges, refs attributed to the innermost enclosing symbol). Seed TS/JS queries are smoke-level; R2 replaces them with the full port. - Routing seam in extractFromSource with per-file wasm fallback. DEFAULT_ROUTED is empty — no behavior change until a language passes its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before routing; EDGE_KINDS became a runtime array because kind order is now wire contract. - Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the exact crate revisions (tree-sitter-typescript v0.23.2, tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) — the tree-sitter-wasms builds were 2023-era, which the new kernel-grammar-parity test caught on day one. Production TS/JS parsing gets 2.5 years of grammar fixes; full suite green (2456 tests). - Build/release wiring: scripts/build-kernel.sh + npm run build:kernel; release.yml kernel prebuild matrix (continue-on-error — the kernel is optional everywhere, bundles fall back to the wasm path); bundles stage lib/kernel/codegraph-kernel.node; release job runs the kernel suites with CODEGRAPH_KERNEL_EXPECT=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
4efc6c70e2
commit
c5eebe6beb
@@ -0,0 +1,104 @@
|
||||
//! codegraph-kernel — native extraction kernel (napi-rs).
|
||||
//!
|
||||
//! Replaces ONLY the parse+extract walk inside the parse workers, behind the
|
||||
//! existing `ExtractionResult` contract. Input `(filePath, content, language)`
|
||||
//! per file; output flat typed buffers — one boundary crossing per file.
|
||||
//! Everything downstream (resolution, synthesis, frameworks, MCP) is
|
||||
//! untouched and consumes the decoded result exactly as before.
|
||||
//!
|
||||
//! Calls are synchronous by design: the existing `ParseWorkerPool` workers
|
||||
//! already parallelize per-file, so each worker thread drives its own kernel
|
||||
//! call (do NOT rebuild the pool on the Rust side — see the migration plan §3).
|
||||
|
||||
#![deny(clippy::all)]
|
||||
|
||||
mod buffers;
|
||||
mod emitter;
|
||||
mod ids;
|
||||
mod langs;
|
||||
|
||||
use napi::bindgen_prelude::*;
|
||||
use napi_derive::napi;
|
||||
|
||||
/// The five flat tables for one file. See buffers.rs for the byte layout;
|
||||
/// `src/extraction/kernel/layout.ts` is the TS mirror.
|
||||
#[napi(object)]
|
||||
pub struct ExtractBuffers {
|
||||
pub meta: Buffer,
|
||||
pub nodes: Buffer,
|
||||
pub edges: Buffer,
|
||||
pub refs: Buffer,
|
||||
pub arena: Buffer,
|
||||
}
|
||||
|
||||
/// Wire-contract description — the TS loader verifies this against
|
||||
/// src/types.ts before routing anything to the kernel, so an out-of-date
|
||||
/// `.node` degrades to the wasm path instead of mis-decoding.
|
||||
#[napi(object)]
|
||||
pub struct ContractInfo {
|
||||
pub abi_version: u32,
|
||||
pub kernel_version: String,
|
||||
pub node_kinds: Vec<String>,
|
||||
pub edge_kinds: Vec<String>,
|
||||
/// Languages this binary can extract (routing is still TS-side policy).
|
||||
pub languages: Vec<String>,
|
||||
}
|
||||
|
||||
/// Grammar identity for the grammar-source-parity gate: the wasm grammar and
|
||||
/// the native grammar must expose identical node-kind/field tables, or
|
||||
/// kernel-vs-fallback routing would be non-deterministic.
|
||||
#[napi(object)]
|
||||
pub struct GrammarInfo {
|
||||
pub abi_version: u32,
|
||||
pub node_kind_count: u32,
|
||||
pub field_count: u32,
|
||||
pub node_kinds: Vec<String>,
|
||||
pub field_names: Vec<String>,
|
||||
}
|
||||
|
||||
#[napi]
|
||||
pub fn contract_info() -> ContractInfo {
|
||||
ContractInfo {
|
||||
abi_version: buffers::KERNEL_ABI_VERSION as u32,
|
||||
kernel_version: env!("CARGO_PKG_VERSION").to_string(),
|
||||
node_kinds: buffers::NODE_KINDS.iter().map(|s| s.to_string()).collect(),
|
||||
edge_kinds: buffers::EDGE_KINDS.iter().map(|s| s.to_string()).collect(),
|
||||
languages: langs::ALL.iter().map(|s| s.name.to_string()).collect(),
|
||||
}
|
||||
}
|
||||
|
||||
#[napi]
|
||||
pub fn grammar_info(language: String) -> Option<GrammarInfo> {
|
||||
let spec = langs::spec_for(&language)?;
|
||||
let lang = spec.language();
|
||||
let node_kind_count = lang.node_kind_count();
|
||||
let field_count = lang.field_count();
|
||||
let node_kinds = (0..node_kind_count)
|
||||
.map(|i| lang.node_kind_for_id(i as u16).unwrap_or("").to_string())
|
||||
.collect();
|
||||
// Field ids are 1-based in tree-sitter.
|
||||
let field_names = (1..=field_count)
|
||||
.map(|i| lang.field_name_for_id(i as u16).unwrap_or("").to_string())
|
||||
.collect();
|
||||
Some(GrammarInfo {
|
||||
abi_version: lang.abi_version() as u32,
|
||||
node_kind_count: node_kind_count as u32,
|
||||
field_count: field_count as u32,
|
||||
node_kinds,
|
||||
field_names,
|
||||
})
|
||||
}
|
||||
|
||||
#[napi]
|
||||
pub fn extract_file(file_path: String, content: String, language: String) -> Result<ExtractBuffers> {
|
||||
let spec = langs::spec_for(&language)
|
||||
.ok_or_else(|| Error::from_reason(format!("kernel does not support language: {language}")))?;
|
||||
let out = emitter::extract(&file_path, &content, spec).map_err(Error::from_reason)?;
|
||||
Ok(ExtractBuffers {
|
||||
meta: out.meta.into(),
|
||||
nodes: out.nodes.into(),
|
||||
edges: out.edges.into(),
|
||||
refs: out.refs.into(),
|
||||
arena: out.arena.into(),
|
||||
})
|
||||
}
|
||||
Reference in New Issue
Block a user