github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(kernel): R2 — full TypeScript/JavaScript extraction port, byte-parity with the wasm path

Browse Source 
Replaces the R1 seed .scm emitter with a bespoke Rust walker
(codegraph-kernel/src/tsjs/) that mirrors TreeSitterExtractor's TS/JS
paths function-for-function: declarations (incl. #808 field/property
classification), qualified names, docstrings (#780 wrapper climbs),
signatures, imports/re-exports + per-binding refs, calls with
receiver-qualified callees (#1230 literal-receiver skip), instantiations,
decorators, inheritance, type annotations (#381), type-alias members +
tuple contracts (#359/#634), React component recognition (#841
forwardRef/memo/styled), object-of-functions / zustand-through-middleware
/ RTK Query endpoints + generated hooks / vuex + pinia store shapes,
function-as-value capture with the flush gate (#756), and value-reference
edges with the shadow prune (#895/#897). The generic query emitter is
deleted — extraction parity needs logic .scm can't express; future
languages get walkers too (migration plan §4a).

Positions and JS string-slice semantics are emitted in UTF-16 code units
natively, so kernel output is byte-identical to web-tree-sitter's — no
column diff class exists.

Parity evidence (macOS): scripts/kernel-parity.mjs (full-object multiset
diff per file) — this repo 353/353 files, excalidraw 643/643 (10,650
nodes / 10,726 edges / 68,307 refs), plus torture fixtures checked into
__tests__/fixtures/kernel-parity/ and enforced in npm test by
kernel-tsjs-parity.test.ts. The strict compare caught one real decoder
bug the loose harness missed: refs must NOT carry denormalized
filePath/language at the extractFromSource seam (the store fills them).

Perf: extraction 2.6× single-thread on excalidraw (487ms vs 1,255ms,
identical outputs). Routing stays opt-in (CODEGRAPH_KERNEL_LANGS) until
the R3 equivalence gate (large repo, DB dump-diff, retrieval invariants,
agent A/B, Linux/Windows) passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-07-16 22:14:40 -05:00
co-authored by Claude Fable 5
parent c5eebe6beb
commit 9ad5cd7ba2
20 changed files with 3382 additions and 419 deletions
Download Patch File Download Diff File Expand all files Collapse all files
codegraph-kernel/src/langs.rs
+20 -71
View File
@@ -1,78 +1,27 @@
//! Per-language specs: grammar + `.scm` query + (later) per-language config.
//! Grammar registry: codegraph `Language` string → native tree-sitter grammar.
//!
//! Tier-1 languages are meant to be *mostly* a query file plus a small config
//! here; logic queries can't express stays TS-side as a per-language `post()`
//! hook over the returned buffers (see `src/extraction/kernel/route.ts`).
//! Mirrors the wasm side's `WASM_GRAMMAR_FILES` mapping (src/extraction/
//! grammars.ts): `tsx` and `jsx` reuse another language's grammar exactly the
//! way the wasm map does. The kernel-grammar-parity test asserts each entry is
//! built from the SAME grammar revision as the vendored wasm — bump the crate
//! and the wasm together.
//!
//! Language strings are codegraph `Language` values (src/types.ts), not
//! grammar names — `tsx` and `jsx` are separate entries that reuse another
//! entry's grammar exactly like `WASM_GRAMMAR_FILES` does on the wasm path.
//! (R1 shipped a generic `.scm`-query emitter here; R2 replaced it with the
//! bespoke per-language walker — see tsjs/ and the migration plan §3a — because
//! extraction parity needs logic queries can't express. New languages add a
//! grammar entry + a walker module.)
use std::sync::OnceLock;
use tree_sitter::{Language, Query};
use tree_sitter::Language;
pub struct LangSpec {
/// codegraph Language string (src/types.ts).
pub name: &'static str,
get_language: fn() -> Language,
query_src: &'static str,
language: OnceLock<Language>,
query: OnceLock<Result<Query, String>>,
}
/// Languages this kernel binary can extract (reported by contractInfo;
/// TS-side routing policy decides what actually routes).
pub const LANGUAGES: [&str; 4] = ["typescript", "tsx", "javascript", "jsx"];
impl LangSpec {
const fn new(name: &'static str, get_language: fn() -> Language, query_src: &'static str) -> Self {
LangSpec {
name,
get_language,
query_src,
language: OnceLock::new(),
query: OnceLock::new(),
}
}
pub fn language(&self) -> &Language {
self.language.get_or_init(self.get_language)
}
pub fn query(&self) -> Result<&Query, String> {
self.query
.get_or_init(|| {
Query::new(self.language(), self.query_src)
.map_err(|e| format!("query compile failed for {}: {e}", self.name))
})
.as_ref()
.map_err(|e| e.clone())
pub fn grammar_for(language: &str) -> Option<Language> {
match language {
"typescript" => Some(tree_sitter_typescript::LANGUAGE_TYPESCRIPT.into()),
"tsx" => Some(tree_sitter_typescript::LANGUAGE_TSX.into()),
"javascript" | "jsx" => Some(tree_sitter_javascript::LANGUAGE.into()),
_ => None,
}
}
fn ts_language() -> Language {
tree_sitter_typescript::LANGUAGE_TYPESCRIPT.into()
}
fn tsx_language() -> Language {
tree_sitter_typescript::LANGUAGE_TSX.into()
}
fn js_language() -> Language {
tree_sitter_javascript::LANGUAGE.into()
}
static TYPESCRIPT: LangSpec = LangSpec::new(
"typescript",
ts_language,
include_str!("../queries/typescript.scm"),
);
static TSX: LangSpec = LangSpec::new("tsx", tsx_language, include_str!("../queries/typescript.scm"));
static JAVASCRIPT: LangSpec = LangSpec::new(
"javascript",
js_language,
include_str!("../queries/javascript.scm"),
);
static JSX: LangSpec = LangSpec::new("jsx", js_language, include_str!("../queries/javascript.scm"));
pub static ALL: [&LangSpec; 4] = [&TYPESCRIPT, &TSX, &JAVASCRIPT, &JSX];
pub fn spec_for(language: &str) -> Option<&'static LangSpec> {
ALL.iter().find(|s| s.name == language).copied()
}