feat(kernel): R1 scaffold — napi-rs extraction kernel, buffer contract, routing + fallback, grammar-parity CI

Phase 0 of the Rust extraction-kernel migration (docs/design/
rust-kernel-migration-plan.md, now checked in with §3a recording the
shipped state):

- codegraph-kernel/ napi-rs crate: extractFile(path, content, language)
  → five flat buffers (meta/nodes/edges/refs/arena), one JS boundary
  crossing per file. Node ids computed Rust-side, byte-identical to
  generateNodeId (pinned by test vector). Reserved per-node metrics slot
  for the Arc 3.2 code-metrics work.
- Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures,
  byte-range scope stack → ::-joined qualified names, contains edges,
  refs attributed to the innermost enclosing symbol). Seed TS/JS queries
  are smoke-level; R2 replaces them with the full port.
- Routing seam in extractFromSource with per-file wasm fallback.
  DEFAULT_ROUTED is empty — no behavior change until a language passes
  its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill
  switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before
  routing; EDGE_KINDS became a runtime array because kind order is now
  wire contract.
- Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the
  exact crate revisions (tree-sitter-typescript v0.23.2,
  tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) —
  the tree-sitter-wasms builds were 2023-era, which the new
  kernel-grammar-parity test caught on day one. Production TS/JS parsing
  gets 2.5 years of grammar fixes; full suite green (2456 tests).
- Build/release wiring: scripts/build-kernel.sh + npm run build:kernel;
  release.yml kernel prebuild matrix (continue-on-error — the kernel is
  optional everywhere, bundles fall back to the wasm path); bundles stage
  lib/kernel/codegraph-kernel.node; release job runs the kernel suites
  with CODEGRAPH_KERNEL_EXPECT=1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-07-16 20:13:51 -05:00
co-authored by Claude Fable 5
parent 4efc6c70e2
commit c5eebe6beb
29 changed files with 2804 additions and 16 deletions
+104
View File
@@ -0,0 +1,104 @@
//! codegraph-kernel — native extraction kernel (napi-rs).
//!
//! Replaces ONLY the parse+extract walk inside the parse workers, behind the
//! existing `ExtractionResult` contract. Input `(filePath, content, language)`
//! per file; output flat typed buffers — one boundary crossing per file.
//! Everything downstream (resolution, synthesis, frameworks, MCP) is
//! untouched and consumes the decoded result exactly as before.
//!
//! Calls are synchronous by design: the existing `ParseWorkerPool` workers
//! already parallelize per-file, so each worker thread drives its own kernel
//! call (do NOT rebuild the pool on the Rust side — see the migration plan §3).
#![deny(clippy::all)]
mod buffers;
mod emitter;
mod ids;
mod langs;
use napi::bindgen_prelude::*;
use napi_derive::napi;
/// The five flat tables for one file. See buffers.rs for the byte layout;
/// `src/extraction/kernel/layout.ts` is the TS mirror.
#[napi(object)]
pub struct ExtractBuffers {
pub meta: Buffer,
pub nodes: Buffer,
pub edges: Buffer,
pub refs: Buffer,
pub arena: Buffer,
}
/// Wire-contract description — the TS loader verifies this against
/// src/types.ts before routing anything to the kernel, so an out-of-date
/// `.node` degrades to the wasm path instead of mis-decoding.
#[napi(object)]
pub struct ContractInfo {
pub abi_version: u32,
pub kernel_version: String,
pub node_kinds: Vec<String>,
pub edge_kinds: Vec<String>,
/// Languages this binary can extract (routing is still TS-side policy).
pub languages: Vec<String>,
}
/// Grammar identity for the grammar-source-parity gate: the wasm grammar and
/// the native grammar must expose identical node-kind/field tables, or
/// kernel-vs-fallback routing would be non-deterministic.
#[napi(object)]
pub struct GrammarInfo {
pub abi_version: u32,
pub node_kind_count: u32,
pub field_count: u32,
pub node_kinds: Vec<String>,
pub field_names: Vec<String>,
}
#[napi]
pub fn contract_info() -> ContractInfo {
ContractInfo {
abi_version: buffers::KERNEL_ABI_VERSION as u32,
kernel_version: env!("CARGO_PKG_VERSION").to_string(),
node_kinds: buffers::NODE_KINDS.iter().map(|s| s.to_string()).collect(),
edge_kinds: buffers::EDGE_KINDS.iter().map(|s| s.to_string()).collect(),
languages: langs::ALL.iter().map(|s| s.name.to_string()).collect(),
}
}
#[napi]
pub fn grammar_info(language: String) -> Option<GrammarInfo> {
let spec = langs::spec_for(&language)?;
let lang = spec.language();
let node_kind_count = lang.node_kind_count();
let field_count = lang.field_count();
let node_kinds = (0..node_kind_count)
.map(|i| lang.node_kind_for_id(i as u16).unwrap_or("").to_string())
.collect();
// Field ids are 1-based in tree-sitter.
let field_names = (1..=field_count)
.map(|i| lang.field_name_for_id(i as u16).unwrap_or("").to_string())
.collect();
Some(GrammarInfo {
abi_version: lang.abi_version() as u32,
node_kind_count: node_kind_count as u32,
field_count: field_count as u32,
node_kinds,
field_names,
})
}
#[napi]
pub fn extract_file(file_path: String, content: String, language: String) -> Result<ExtractBuffers> {
let spec = langs::spec_for(&language)
.ok_or_else(|| Error::from_reason(format!("kernel does not support language: {language}")))?;
let out = emitter::extract(&file_path, &content, spec).map_err(Error::from_reason)?;
Ok(ExtractBuffers {
meta: out.meta.into(),
nodes: out.nodes.into(),
edges: out.edges.into(),
refs: out.refs.into(),
arena: out.arena.into(),
})
}