feat(kernel): R4 — Java port with Lombok synthesis, gate passed, default-on

Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring
the wasm extractor's Java paths bug-for-bug: package namespaces,
imports, javadoc, annotations→decorates, type_list inheritance,
static-final constants, enum constants, anonymous classes (including
the TS side's 0-based-line quirk on the extends ref), method_invocation
calls with the this.field unwrap and the Foo.getInstance().bar() chain
encoding (#645/#608), static-member value reads, method-reference
fn-refs (#756), value-reference edges, and the full Lombok member
synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/
EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared
docstring/textutil modules moved to crate level. Grammar:
tree-sitter-java 0.23.5, with the wasm grammar vendored from the same
tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build.

Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit
341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test;
full-init dump-diffs byte-identical on gson (49,766 rows), retrofit
(62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux
container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1.

The gate caught a real cross-language bug: fn-ref dedupe and value-ref
self-target checks must compare node ID STRINGS, not node-table rows —
ids collide for same-(kind, name, line) nodes, which minified one-line
bundles hit routinely (retrofit's website JS exposed it; latent in the
TS/JS walker since R2, never released). Fixed in both walkers.

Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat
(parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that
wall is main-thread-bound (reads + store), not worker-CPU-bound — the
§6 expectation assumed otherwise. Where worker CPU binds the kernel
delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s
(~1.25×). The identified lever for the many-core headline is decoding
kernel buffers directly into store rows (skipping per-node JS object
materialization); the buffer contract already carries everything.

DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-07-16 23:25:29 -05:00
co-authored by Claude Fable 5
parent c8cca9a601
commit 03d54e47a1
19 changed files with 1828 additions and 30 deletions
+1 -1
View File
@@ -11,7 +11,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
### New Features ### New Features
- Indexing TypeScript, TSX, JavaScript, and JSX projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-scale codebases. The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely. - Indexing TypeScript, TSX, JavaScript, JSX, and Java projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode- and dubbo-scale codebases (Lombok-generated members included). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
- Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=<count>` to tune when it engages. - Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=<count>` to tune when it engages.
- Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle. - Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle.
- The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical. - The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical.
@@ -0,0 +1,102 @@
/**
* Java torture fixture exercises every Java extraction path the kernel
* ports: package namespace, imports, javadoc, annotations, inheritance,
* fields/constants, enums, anonymous classes, method references, static
* member reads, fluent chains, Lombok synthesis, value refs + shadowing.
*/
package com.example.torture;
import java.util.List;
import java.util.Map;
import static java.util.Objects.requireNonNull;
import com.example.other.OtherClass;
import lombok.Data;
/** Javadoc for the service. */
@Service
@Component("torture")
public class TortureService extends BaseService implements Runnable, AutoCloseable {
/** A shared constant table (value-ref target). */
public static final Map<String, Integer> RETRY_LIMITS = Map.of("a", 1);
private static final String API_BASE = "https://example.test";
protected int count = 0;
private final List<String> names;
int packagePrivate, secondDeclarator;
/** Ctor javadoc. */
public TortureService(List<String> names) {
this.names = requireNonNull(names);
register(this::onEvent);
queue(TortureService::compute);
queue(OtherClass::handle);
Runnable r = () -> helper(RETRY_LIMITS);
executor.submit(new Runnable() {
@Override
public void run() {
helper(RETRY_LIMITS);
}
});
}
@Override
@Deprecated
public void run() {
Config cfg = ConfigLoader.getInstance().load();
this.registry.lookup("x");
helper(Direction.UP);
String base = API_BASE;
int max = Limits.MAX_VALUE;
new StringBuilder(16).append(base);
}
private static Config helper(Object arg) {
return new Config();
}
private void onEvent() {}
private static void compute() {}
public void shadowed() {
String API_BASE = "local"; // shadows the class constant
log(API_BASE);
}
enum Direction {
UP,
DOWN;
Direction opposite() {
return this == UP ? DOWN : UP;
}
}
interface Listener {
void onChange(TortureService svc);
}
}
@Data
class LombokBean {
private String name;
private boolean isActive;
private final int id;
private static int counter;
private String toString; // taken: no synthetic toString field collision
public String getName() { return name; } // explicit getter never overridden
}
@lombok.Getter
@lombok.extern.slf4j.Slf4j
@Builder
class LombokBuilderBean {
private List<String> items;
}
interface Shape extends Comparable<Shape>, Cloneable {
double area();
}
@interface Marker {
String value() default "";
}
+1 -1
View File
@@ -36,7 +36,7 @@ const kernelBuilt = fs.existsSync(KERNEL_PATH);
// Every kernel-capable language. `jsx` shares the javascript grammar on BOTH // Every kernel-capable language. `jsx` shares the javascript grammar on BOTH
// paths (langs.rs mirrors WASM_GRAMMAR_FILES), so the distinct grammars are: // paths (langs.rs mirrors WASM_GRAMMAR_FILES), so the distinct grammars are:
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript']; const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java'];
describe.skipIf(!kernelBuilt)('kernel↔wasm grammar parity', () => { describe.skipIf(!kernelBuilt)('kernel↔wasm grammar parity', () => {
beforeAll(async () => { beforeAll(async () => {
+2 -2
View File
@@ -72,8 +72,8 @@ describe.skipIf(!kernelBuilt)('kernel scaffold', () => {
expect(info.languages).toContain('javascript'); expect(info.languages).toContain('javascript');
}); });
it('TS/JS family routes to the kernel by default (R3 default-on); others stay wasm', () => { it('TS/JS family + Java route to the kernel by default; others stay wasm', () => {
for (const lang of ['typescript', 'tsx', 'javascript', 'jsx'] as const) { for (const lang of ['typescript', 'tsx', 'javascript', 'jsx', 'java'] as const) {
expect(kernelRoutes(lang), lang).toBe(true); expect(kernelRoutes(lang), lang).toBe(true);
} }
expect(kernelRoutes('python')).toBe(false); expect(kernelRoutes('python')).toBe(false);
+6 -1
View File
@@ -60,7 +60,7 @@ let savedEnv: Record<string, string | undefined>;
describe.skipIf(!kernelBuilt)('kernel TS/JS extraction parity', () => { describe.skipIf(!kernelBuilt)('kernel TS/JS extraction parity', () => {
beforeAll(async () => { beforeAll(async () => {
await initGrammars(); await initGrammars();
await loadGrammarsForLanguages(['typescript', 'tsx', 'javascript', 'jsx']); await loadGrammarsForLanguages(['typescript', 'tsx', 'javascript', 'jsx', 'java']);
}); });
beforeEach(() => { beforeEach(() => {
@@ -105,6 +105,11 @@ describe.skipIf(!kernelBuilt)('kernel TS/JS extraction parity', () => {
assertParity('fixtures/torture.js', fs.readFileSync(file, 'utf8'), 'javascript'); assertParity('fixtures/torture.js', fs.readFileSync(file, 'utf8'), 'javascript');
}); });
it('torture fixture (java): Lombok, anonymous classes, method refs, chains', () => {
const file = path.join(FIXTURE_DIR, 'Torture.java');
assertParity('fixtures/Torture.java', fs.readFileSync(file, 'utf8'), 'java');
});
it.each(REAL_SOURCES)('real source parity: %s', (rel) => { it.each(REAL_SOURCES)('real source parity: %s', (rel) => {
const file = path.join(__dirname, '..', rel); const file = path.join(__dirname, '..', rel);
assertParity(rel, fs.readFileSync(file, 'utf8'), 'typescript'); assertParity(rel, fs.readFileSync(file, 'utf8'), 'typescript');
+11
View File
@@ -52,6 +52,7 @@ dependencies = [
"regex", "regex",
"sha2", "sha2",
"tree-sitter", "tree-sitter",
"tree-sitter-java",
"tree-sitter-javascript", "tree-sitter-javascript",
"tree-sitter-typescript", "tree-sitter-typescript",
] ]
@@ -479,6 +480,16 @@ dependencies = [
"tree-sitter-language", "tree-sitter-language",
] ]
[[package]]
name = "tree-sitter-java"
version = "0.23.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0aa6cbcdc8c679b214e616fd3300da67da0e492e066df01bcf5a5921a71e90d6"
dependencies = [
"cc",
"tree-sitter-language",
]
[[package]] [[package]]
name = "tree-sitter-javascript" name = "tree-sitter-javascript"
version = "0.25.0" version = "0.25.0"
+1
View File
@@ -22,6 +22,7 @@ regex = "1"
# bump these together with the wasm side or that gate fails. # bump these together with the wasm side or that gate fails.
tree-sitter-typescript = "0.23" tree-sitter-typescript = "0.23"
tree-sitter-javascript = "0.25" tree-sitter-javascript = "0.25"
tree-sitter-java = "0.23"
[build-dependencies] [build-dependencies]
napi-build = "2" napi-build = "2"
File diff suppressed because it is too large Load Diff
+2 -1
View File
@@ -15,13 +15,14 @@ use tree_sitter::Language;
/// Languages this kernel binary can extract (reported by contractInfo; /// Languages this kernel binary can extract (reported by contractInfo;
/// TS-side routing policy decides what actually routes). /// TS-side routing policy decides what actually routes).
pub const LANGUAGES: [&str; 4] = ["typescript", "tsx", "javascript", "jsx"]; pub const LANGUAGES: [&str; 5] = ["typescript", "tsx", "javascript", "jsx", "java"];
pub fn grammar_for(language: &str) -> Option<Language> { pub fn grammar_for(language: &str) -> Option<Language> {
match language { match language {
"typescript" => Some(tree_sitter_typescript::LANGUAGE_TYPESCRIPT.into()), "typescript" => Some(tree_sitter_typescript::LANGUAGE_TYPESCRIPT.into()),
"tsx" => Some(tree_sitter_typescript::LANGUAGE_TSX.into()), "tsx" => Some(tree_sitter_typescript::LANGUAGE_TSX.into()),
"javascript" | "jsx" => Some(tree_sitter_javascript::LANGUAGE.into()), "javascript" | "jsx" => Some(tree_sitter_javascript::LANGUAGE.into()),
"java" => Some(tree_sitter_java::LANGUAGE.into()),
_ => None, _ => None,
} }
} }
+7 -1
View File
@@ -17,8 +17,11 @@
#![deny(clippy::all)] #![deny(clippy::all)]
mod buffers; mod buffers;
mod docstring;
mod ids; mod ids;
mod java;
mod langs; mod langs;
mod textutil;
mod tsjs; mod tsjs;
use napi::bindgen_prelude::*; use napi::bindgen_prelude::*;
@@ -94,7 +97,10 @@ pub fn grammar_info(language: String) -> Option<GrammarInfo> {
#[napi] #[napi]
pub fn extract_file(file_path: String, content: String, language: String) -> Result<ExtractBuffers> { pub fn extract_file(file_path: String, content: String, language: String) -> Result<ExtractBuffers> {
let out = tsjs::extract(&file_path, &content, &language).map_err(Error::from_reason)?; let out = match language.as_str() {
"java" => java::extract(&file_path, &content).map_err(Error::from_reason)?,
_ => tsjs::extract(&file_path, &content, &language).map_err(Error::from_reason)?,
};
Ok(ExtractBuffers { Ok(ExtractBuffers {
meta: out.meta.into(), meta: out.meta.into(),
nodes: out.nodes.into(), nodes: out.nodes.into(),
+10 -10
View File
@@ -3,7 +3,7 @@
//! src/extraction/tree-sitter.ts; TS-file line references are as of the R2 //! src/extraction/tree-sitter.ts; TS-file line references are as of the R2
//! port. Bug-for-bug fidelity is deliberate — fix the TS side first. //! port. Bug-for-bug fidelity is deliberate — fix the TS side first.
use super::util; use crate::textutil as util;
use super::{ use super::{
body_of, is_builtin_type, is_literal_receiver, is_react_hoc, is_variable_type, body_of, is_builtin_type, is_literal_receiver, is_react_hoc, is_variable_type,
is_vue_collection_name, Extra, Scope, Walker, is_vue_collection_name, Extra, Scope, Walker,
@@ -43,7 +43,7 @@ impl<'t> Walker<'t> {
} }
let extra = Extra { let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src), docstring: crate::docstring::preceding_docstring(node, self.src),
signature: self.signature_of(node), signature: self.signature_of(node),
visibility: self.visibility_of(node), visibility: self.visibility_of(node),
is_exported: Some(self.is_exported(node)), is_exported: Some(self.is_exported(node)),
@@ -120,7 +120,7 @@ impl<'t> Walker<'t> {
let resolved_body = body_of(node); // skipBodilessClass unset for TS/JS let resolved_body = body_of(node); // skipBodilessClass unset for TS/JS
let name = self.extract_name(node); let name = self.extract_name(node);
let extra = Extra { let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src), docstring: crate::docstring::preceding_docstring(node, self.src),
visibility: self.visibility_of(node), visibility: self.visibility_of(node),
is_exported: Some(self.is_exported(node)), is_exported: Some(self.is_exported(node)),
..Extra::default() ..Extra::default()
@@ -161,7 +161,7 @@ impl<'t> Walker<'t> {
let name = self.extract_name(node); let name = self.extract_name(node);
let extra = Extra { let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src), docstring: crate::docstring::preceding_docstring(node, self.src),
signature: self.signature_of(node), signature: self.signature_of(node),
visibility: self.visibility_of(node), visibility: self.visibility_of(node),
is_async: Some(self.is_async(node)), is_async: Some(self.is_async(node)),
@@ -187,7 +187,7 @@ impl<'t> Walker<'t> {
pub(super) fn extract_interface(&mut self, node: Node<'t>) { pub(super) fn extract_interface(&mut self, node: Node<'t>) {
let name = self.extract_name(node); let name = self.extract_name(node);
let extra = Extra { let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src), docstring: crate::docstring::preceding_docstring(node, self.src),
is_exported: Some(self.is_exported(node)), is_exported: Some(self.is_exported(node)),
..Extra::default() ..Extra::default()
}; };
@@ -209,7 +209,7 @@ impl<'t> Walker<'t> {
let Some(body) = body_of(node) else { return }; let Some(body) = body_of(node) else { return };
let name = self.extract_name(node); let name = self.extract_name(node);
let extra = Extra { let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src), docstring: crate::docstring::preceding_docstring(node, self.src),
visibility: self.visibility_of(node), visibility: self.visibility_of(node),
is_exported: Some(self.is_exported(node)), is_exported: Some(self.is_exported(node)),
..Extra::default() ..Extra::default()
@@ -255,7 +255,7 @@ impl<'t> Walker<'t> {
// --- extractProperty (#808 property-classified class fields) --------------------- // --- extractProperty (#808 property-classified class fields) ---------------------
pub(super) fn extract_property(&mut self, node: Node<'t>) -> Option<(u32, String)> { pub(super) fn extract_property(&mut self, node: Node<'t>) -> Option<(u32, String)> {
let docstring = super::docstring::preceding_docstring(node, self.src); let docstring = crate::docstring::preceding_docstring(node, self.src);
let visibility = self.visibility_of(node); let visibility = self.visibility_of(node);
let is_static = Some(self.is_static(node).unwrap_or(false)); // `?? false` — always present let is_static = Some(self.is_static(node).unwrap_or(false)); // `?? false` — always present
@@ -296,7 +296,7 @@ impl<'t> Walker<'t> {
pub(super) fn extract_variable(&mut self, node: Node<'t>) { pub(super) fn extract_variable(&mut self, node: Node<'t>) {
let is_const = self.is_const_decl(node); let is_const = self.is_const_decl(node);
let kind: &'static str = if is_const { "constant" } else { "variable" }; let kind: &'static str = if is_const { "constant" } else { "variable" };
let docstring = super::docstring::preceding_docstring(node, self.src); let docstring = crate::docstring::preceding_docstring(node, self.src);
let is_exported = self.is_exported(node); // `?? false` — always present let is_exported = self.is_exported(node); // `?? false` — always present
for i in 0..node.named_child_count() { for i in 0..node.named_child_count() {
@@ -807,7 +807,7 @@ impl<'t> Walker<'t> {
return false; return false;
} }
let extra = Extra { let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src), docstring: crate::docstring::preceding_docstring(node, self.src),
is_exported: Some(self.is_exported(node)), is_exported: Some(self.is_exported(node)),
..Extra::default() ..Extra::default()
}; };
@@ -858,7 +858,7 @@ impl<'t> Walker<'t> {
"property" "property"
}; };
let extra = Extra { let extra = Extra {
docstring: super::docstring::preceding_docstring(child, self.src), docstring: crate::docstring::preceding_docstring(child, self.src),
signature: Some(self.text(child).to_string()), signature: Some(self.text(child).to_string()),
qualified_name: Some(format!("{alias_name}::{member_name}")), qualified_name: Some(format!("{alias_name}::{member_name}")),
..Extra::default() ..Extra::default()
+23 -7
View File
@@ -9,10 +9,9 @@
//! parity gate fails. Positions are emitted in UTF-16 code units (what //! parity gate fails. Positions are emitted in UTF-16 code units (what
//! web-tree-sitter reports), see util::col16. //! web-tree-sitter reports), see util::col16.
mod docstring;
mod extractors; mod extractors;
mod fnref; mod fnref;
pub(crate) mod util; use crate::textutil as util;
use crate::buffers::{ use crate::buffers::{
build_meta, edge_kind_index, node_kind_index, Arena, BoolFlags, EdgeRow, EmitOut, NodeRow, build_meta, edge_kind_index, node_kind_index, Arena, BoolFlags, EdgeRow, EmitOut, NodeRow,
@@ -148,6 +147,11 @@ pub struct Walker<'t> {
arena: Arena, arena: Arena,
tables: Tables, tables: Tables,
stack: Vec<Scope>, stack: Vec<Scope>,
/// Node id string per row. Rows are unique but IDS COLLIDE for same
/// (kind, name, line) nodes — routine in minified one-line files — and the
/// TS extractor's fn-ref dedupe and value-ref self-checks key on the ID,
/// so parity requires comparing ids, not rows.
node_ids: Vec<String>,
/// Function/method names defined in this file (fn-ref flush gate). /// Function/method names defined in this file (fn-ref flush gate).
defined_fn_names: HashSet<String>, defined_fn_names: HashSet<String>,
/// Simple names from `imports` refs (fn-ref flush gate). /// Simple names from `imports` refs (fn-ref flush gate).
@@ -198,6 +202,7 @@ pub fn extract(file_path: &str, source: &str, language: &str) -> Result<EmitOut,
arena: Arena::default(), arena: Arena::default(),
tables: Tables::default(), tables: Tables::default(),
stack: Vec::new(), stack: Vec::new(),
node_ids: Vec::new(),
defined_fn_names: HashSet::new(), defined_fn_names: HashSet::new(),
imported_names: HashSet::new(), imported_names: HashSet::new(),
fn_ref_cands: Vec::new(), fn_ref_cands: Vec::new(),
@@ -234,6 +239,7 @@ pub fn extract(file_path: &str, source: &str, language: &str) -> Result<EmitOut,
return_type: NONE_STR, return_type: NONE_STR,
extra_json: NONE_STR, extra_json: NONE_STR,
}); });
w.node_ids.push(ids::file_node_id(file_path));
w.stack.push(Scope { row: 0, kind: "file", name: base_name.to_string() }); w.stack.push(Scope { row: 0, kind: "file", name: base_name.to_string() });
w.visit_node(tree.root_node()); w.visit_node(tree.root_node());
@@ -399,6 +405,7 @@ impl<'t> Walker<'t> {
target_id_str: NONE_STR, target_id_str: NONE_STR,
}); });
self.node_ids.push(id);
if kind == "function" || kind == "method" { if kind == "function" || kind == "method" {
self.defined_fn_names.insert(name.to_string()); self.defined_fn_names.insert(name.to_string());
} }
@@ -485,7 +492,9 @@ impl<'t> Walker<'t> {
let refs_kind = edge_kind_index("references").unwrap(); let refs_kind = edge_kind_index("references").unwrap();
for scope in &scopes { for scope in &scopes {
let mut seen: HashSet<u32> = HashSet::new(); // Self-skip and per-scope dedupe compare node ID STRINGS (which
// collide for same-(kind, name, line) nodes), matching the TS side.
let mut seen: HashSet<&str> = HashSet::new();
let mut stack: Vec<Node> = vec![scope.node]; let mut stack: Vec<Node> = vec![scope.node];
let mut visited = 0usize; let mut visited = 0usize;
while let Some(n) = stack.pop() { while let Some(n) = stack.pop() {
@@ -496,8 +505,12 @@ impl<'t> Walker<'t> {
if matches!(n.kind(), "identifier" | "constant" | "name" | "simple_identifier") { if matches!(n.kind(), "identifier" | "constant" | "name" | "simple_identifier") {
let ref_name = self.text(n); let ref_name = self.text(n);
if let Some(&target_row) = targets.get(ref_name) { if let Some(&target_row) = targets.get(ref_name) {
if target_row != scope.row && ref_name != scope.name && !seen.contains(&target_row) { let target_id = self.node_ids[target_row as usize].as_str();
seen.insert(target_row); if target_id != self.node_ids[scope.row as usize]
&& ref_name != scope.name
&& !seen.contains(&target_id)
{
seen.insert(target_id);
let meta = self.arena.put(r#"{"valueRef":true}"#); let meta = self.arena.put(r#"{"valueRef":true}"#);
self.tables.push_edge(&EdgeRow { self.tables.push_edge(&EdgeRow {
source_idx: scope.row, source_idx: scope.row,
@@ -559,7 +572,7 @@ impl<'t> Walker<'t> {
if cands.is_empty() || util::is_generated_file(self.file_path) { if cands.is_empty() || util::is_generated_file(self.file_path) {
return; return;
} }
let mut seen: HashSet<(u32, String)> = HashSet::new(); let mut seen: HashSet<(String, String)> = HashSet::new();
for (from, c) in cands { for (from, c) in cands {
// Gate: `this.<member>` always flushes; everything else must match // Gate: `this.<member>` always flushes; everything else must match
// a same-file function/method or an imported name. (The `::` and // a same-file function/method or an imported name. (The `::` and
@@ -571,7 +584,10 @@ impl<'t> Walker<'t> {
{ {
continue; continue;
} }
if !seen.insert((from, c.name.clone())) { // Dedupe on the node ID STRING, not the row — ids collide for
// same-(kind, name, line) nodes (minified one-liners) and the TS
// side keys its dedupe on `${fromNodeId}|${name}`.
if !seen.insert((self.node_ids[from as usize].clone(), c.name.clone())) {
continue; continue;
} }
let column = util::col16(self.src, &self.line_starts, c.row, c.column_byte); let column = util::col16(self.src, &self.line_starts, c.row, c.column_byte);
+45 -2
View File
@@ -24,7 +24,12 @@ Work top to bottom; each step has a section below with the detail.
**passed + DEFAULT-ON 2026-07-16, see §4b.** One deferred leg: Windows-VM run **passed + DEFAULT-ON 2026-07-16, see §4b.** One deferred leg: Windows-VM run
(VM stopped, `prlctl start` needs Parallels Pro — benign: no .node ⇒ wasm fallback; (VM stopped, `prlctl start` needs Parallels Pro — benign: no .node ⇒ wasm fallback;
the release matrix builds + gates win32 prebuilds). the release matrix builds + gates win32 prebuilds).
- [ ] **R4. Port Java** → re-run the dubbo benchmark → the cbm-parity headline. (§4, §6) - [x] **R4. Port Java** → re-run the dubbo benchmark → the cbm-parity headline. (§4, §6)
**ported + gate passed + DEFAULT-ON 2026-07-16, see §4c.** Benchmark reality
check: dubbo's parse-loop WALL on many-core machines is main-thread-bound (store/
dispatch), so the Mac headline barely moves (~11.3→11.1s; parse-loop 5.0→4.4s);
the win shows where worker CPU binds (dubbo on 2-CPU: 28→22.5s, ~1.25×). The
identified follow-up lever for the Mac number is decode-direct-to-store (§4c).
- [ ] **R5. Port Python, Go.** (§4) - [ ] **R5. Port Python, Go.** (§4)
- [ ] **R6. Kernel-scale re-validation** in the cg1212 container (expect parse 6m → ~2m). (§6) - [ ] **R6. Kernel-scale re-validation** in the cg1212 container (expect parse 6m → ~2m). (§6)
- [ ] **R7. Long-tail languages opportunistically** per the tracker; T3 may stay TS forever. (§4) - [ ] **R7. Long-tail languages opportunistically** per the tracker; T3 may stay TS forever. (§4)
@@ -221,6 +226,44 @@ Default routing: `DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}` in
`src/extraction/kernel/index.ts`. `CODEGRAPH_KERNEL_LANGS` REPLACES the set; `src/extraction/kernel/index.ts`. `CODEGRAPH_KERNEL_LANGS` REPLACES the set;
`CODEGRAPH_KERNEL=0` kills. Changelog entry added under [Unreleased]. `CODEGRAPH_KERNEL=0` kills. Changelog entry added under [Unreleased].
### 4c. R4 — Java PORTED + gate PASSED + DEFAULT-ON (2026-07-16)
- **Walker:** `codegraph-kernel/src/java.rs` (self-contained, sharing the crate-level
docstring/textutil modules) — package namespaces, imports, javadoc, annotations →
decorates, type_list inheritance, fields/constants (static-final → constant),
enum_constant members, anonymous classes (`<T$anon@line>` incl. the TS side's
0-based-line quirk, mirrored bug-for-bug), method_invocation calls with the
`this.field` unwrap + the `Foo.getInstance().bar()` chain encoding, static-member
value reads, method_reference fn-refs (`this::x` / `Type::m`), value refs, and the
**full Lombok member synthesizer** (#912: @Getter/@Setter/@Data/@Value/@Builder/
@ToString/@EqualsAndHashCode/@Slf4j-family, taken-member dedup by exact
`classQN::name`). Grammar: tree-sitter-java crate 0.23.5; wasm vendored from the
SAME tag (94703d5, parser.c sha-matched), replacing tree-sitter-wasms' ^0.20.2 build.
- **Parity:** extraction sweeps — gson 262/262, retrofit 341/341, dubbo 4,048/4,048,
torture fixture (`__tests__/fixtures/kernel-parity/Torture.java`, in `npm test`).
Full-init dump-diffs byte-identical: gson (49,766 rows), retrofit (62,735),
**dubbo (441,266 rows)**. All R2/R3 repos re-verified after the fix below.
- **The gate caught a REAL cross-language bug:** retrofit's minified website JS
exposed that fn-ref dedupe and value-ref self-checks must compare node **ID
strings**, not table rows — IDs collide for same-(kind,name,line) nodes (routine in
minified one-liners: many `function e` on line 3) and the TS side keys on
`${fromNodeId}|${name}`. Fixed in BOTH walkers (`node_ids` per row); this affected
tsjs too (latent since R2, never released).
- **Benchmark honesty (the §6 expectation was wrong about WHERE the win lands):**
dubbo fresh-init on the 11-core M3 Pro is ~FLAT end-to-end (11.311.5 wasm →
11.011.6 kernel; parse-loop wall 5,020→4,394ms) because that phase's wall is
**main-thread-bound** (file reads + result store + SQLite), not worker-CPU-bound —
8 wasm workers already hide extraction CPU behind the main thread on big-core
machines. Where worker CPU binds, the kernel delivers: **dubbo on 2-CPU/6GB Linux
27.828.6s → 22.322.8s (~1.25×)**; excalidraw same envelope ~1.5×; vscode-on-Mac
1.28×. The **cbm-parity Mac headline therefore needs the next lever: decode the
kernel's buffers DIRECTLY into store rows** (skip per-node JS object
materialization on the main thread) — buffer contract already carries everything;
tracked as the top §7a-adjacent follow-up.
- **Platforms:** Linux container (arm64): all 23 kernel tests green EXPECT=1;
Windows VM still deferred (same fallback rationale as §4b).
- Default routing now includes `java`.
## 4. Per-language tracker ## 4. Per-language tracker
Tiers: **T1** = mostly `.scm` + mapping config. **T2** = needs bespoke pre/post passes kept Tiers: **T1** = mostly `.scm` + mapping config. **T2** = needs bespoke pre/post passes kept
@@ -239,7 +282,7 @@ parity before porting the language.
| Language(s) | Today | Tier | Grammar source | Migration notes / known traps | Status | | Language(s) | Today | Tier | Grammar source | Migration notes / known traps | Status |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| typescript, tsx, javascript, jsx | `languages/typescript.ts`, `javascript.ts` + shared branches | T1 | crates.io | First target. Value-reference edges (#895/#897) and component recognition (#841 forwardRef/memo/styled) must survive — they're extraction-side. Largest test surface; gate is strictest here. **PORTED + GATE PASSED + DEFAULT-ON (§4a/§4b); erroring files defer to wasm per-file.** | ✅ | | typescript, tsx, javascript, jsx | `languages/typescript.ts`, `javascript.ts` + shared branches | T1 | crates.io | First target. Value-reference edges (#895/#897) and component recognition (#841 forwardRef/memo/styled) must survive — they're extraction-side. Largest test surface; gate is strictest here. **PORTED + GATE PASSED + DEFAULT-ON (§4a/§4b); erroring files defer to wasm per-file.** | ✅ |
| java | `languages/java.ts` | T1 | crates.io | Second target; unlocks the dubbo-parity claim. Lombok member synthesis (#912) is a NODE synthesizer hook in extraction (`synthesizeMembers`) — port or keep as TS post-pass. | | | java | `languages/java.ts` | T1 | crates.io | Second target; unlocks the dubbo-parity claim. Lombok member synthesis (#912) is a NODE synthesizer hook in extraction (`synthesizeMembers`) — port or keep as TS post-pass. **PORTED incl. Lombok + gate passed + DEFAULT-ON (§4c).** | |
| python | `languages/python.ts` | T1 | crates.io | Third. Decorator extraction feeds framework route detection — parity required. | ☐ | | python | `languages/python.ts` | T1 | crates.io | Third. Decorator extraction feeds framework route detection — parity required. | ☐ |
| go | `languages/go.ts` | T1 | crates.io | Third (tie). Value-reference edges ship here too (#897). | ☐ | | go | `languages/go.ts` | T1 | crates.io | Third (tie). Value-reference edges ship here too (#897). | ☐ |
| ruby, php | dedicated files | T1 | crates.io | Straightforward; PHP property-receiver shapes (#1220/#1251) are RESOLUTION-side, unaffected. | ☐ | | ruby, php | dedicated files | T1 | crates.io | Straightforward; PHP property-receiver shapes (#1220/#1251) are RESOLUTION-side, unaffected. | ☐ |
+2 -2
View File
@@ -39,11 +39,11 @@ if (paths.length === 0) {
process.exit(2); process.exit(2);
} }
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx']); const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java']);
const EXTS = new Map([ const EXTS = new Map([
['.ts', 'typescript'], ['.mts', 'typescript'], ['.cts', 'typescript'], ['.ts', 'typescript'], ['.mts', 'typescript'], ['.cts', 'typescript'],
['.tsx', 'tsx'], ['.js', 'javascript'], ['.mjs', 'javascript'], ['.tsx', 'tsx'], ['.js', 'javascript'], ['.mjs', 'javascript'],
['.cjs', 'javascript'], ['.jsx', 'jsx'], ['.cjs', 'javascript'], ['.jsx', 'jsx'], ['.java', 'java'],
]); ]);
/** Collect candidate files. */ /** Collect candidate files. */
+4 -2
View File
@@ -279,15 +279,17 @@ export async function initGrammars(): Promise<void> {
* parse identically and per-language routing stays graph-neutral: * parse identically and per-language routing stays graph-neutral:
* - tree-sitter/tree-sitter-typescript v0.23.2 (f975a62) typescript + tsx * - tree-sitter/tree-sitter-typescript v0.23.2 (f975a62) typescript + tsx
* - tree-sitter/tree-sitter-javascript v0.25.0 (44c892e) javascript + jsx * - tree-sitter/tree-sitter-javascript v0.25.0 (44c892e) javascript + jsx
* - tree-sitter/tree-sitter-java v0.23.5 (94703d5) java
* Built from each repo's CHECKED-IN parser.c (no `generate`) with * Built from each repo's CHECKED-IN parser.c (no `generate`) with
* tree-sitter-cli 0.25.10 `build --wasm` the same tables crates.io compiles. * tree-sitter-cli 0.25.10 `build --wasm` the same tables crates.io compiles
* (parser.c sha-matched against the crates.io tarball).
* The kernel-grammar-parity test asserts this alignment; bump the crate and * The kernel-grammar-parity test asserts this alignment; bump the crate and
* the vendored wasm together. * the vendored wasm together.
*/ */
const VENDORED_WASM_LANGS: ReadonlySet<GrammarLanguage> = new Set([ const VENDORED_WASM_LANGS: ReadonlySet<GrammarLanguage> = new Set([
'pascal', 'scala', 'lua', 'luau', 'csharp', 'r', 'cfml', 'cfscript', 'cfquery', 'pascal', 'scala', 'lua', 'luau', 'csharp', 'r', 'cfml', 'cfscript', 'cfquery',
'cobol', 'vbnet', 'erlang', 'terraform', 'arkts', 'nix', 'cobol', 'vbnet', 'erlang', 'terraform', 'arkts', 'nix',
'typescript', 'tsx', 'javascript', 'jsx', 'typescript', 'tsx', 'javascript', 'jsx', 'java',
]); ]);
/** Absolute path of a language's grammar WASM (vendored or tree-sitter-wasms). */ /** Absolute path of a language's grammar WASM (vendored or tree-sitter-wasms). */
+1
View File
@@ -33,6 +33,7 @@ const DEFAULT_ROUTED: ReadonlySet<Language> = new Set<Language>([
'tsx', 'tsx',
'javascript', 'javascript',
'jsx', 'jsx',
'java',
]); ]);
/** /**
Binary file not shown.