github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(kernel): R4 — Java port with Lombok synthesis, gate passed, default-on

Browse Source 
Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring
the wasm extractor's Java paths bug-for-bug: package namespaces,
imports, javadoc, annotations→decorates, type_list inheritance,
static-final constants, enum constants, anonymous classes (including
the TS side's 0-based-line quirk on the extends ref), method_invocation
calls with the this.field unwrap and the Foo.getInstance().bar() chain
encoding (#645/#608), static-member value reads, method-reference
fn-refs (#756), value-reference edges, and the full Lombok member
synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/
EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared
docstring/textutil modules moved to crate level. Grammar:
tree-sitter-java 0.23.5, with the wasm grammar vendored from the same
tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build.

Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit
341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test;
full-init dump-diffs byte-identical on gson (49,766 rows), retrofit
(62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux
container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1.

The gate caught a real cross-language bug: fn-ref dedupe and value-ref
self-target checks must compare node ID STRINGS, not node-table rows —
ids collide for same-(kind, name, line) nodes, which minified one-line
bundles hit routinely (retrofit's website JS exposed it; latent in the
TS/JS walker since R2, never released). Fixed in both walkers.

Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat
(parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that
wall is main-thread-bound (reads + store), not worker-CPU-bound — the
§6 expectation assumed otherwise. Where worker CPU binds the kernel
delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s
(~1.25×). The identified lever for the many-core headline is decoding
kernel buffers directly into store rows (skipping per-node JS object
materialization); the buffer contract already carries everything.

DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-07-16 23:25:29 -05:00
co-authored by Claude Fable 5
parent c8cca9a601
commit 03d54e47a1
19 changed files with 1828 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
codegraph-kernel/src/tsjs/docstring.rs
-140
View File
@@ -1,140 +0,0 @@
//! getPrecedingDocstring / cleanCommentMarkers — faithful port of
//! src/extraction/tree-sitter-helpers.ts (#780 wrapper-climb semantics).
use regex::Regex;
use std::sync::OnceLock;
use tree_sitter::Node;
/// DOCSTRING_WRAPPER_TYPES (tree-sitter-helpers.ts).
fn is_wrapper(kind: &str) -> bool {
matches!(
kind,
"export_statement"
| "decorated_definition"
| "lexical_declaration"
| "variable_declaration"
| "variable_declarator"
| "ambient_declaration"
)
}
fn is_comment(kind: &str) -> bool {
matches!(
kind,
"comment" | "line_comment" | "block_comment" | "documentation_comment"
)
}
struct Cleaners {
block_open: Regex,
block_close: Regex,
lua_open: Regex,
lua_close: Regex,
paren_star_open: Regex,
paren_star_close: Regex,
brace_open: Regex,
brace_close: Regex,
slashes: Regex,
dashes: Regex,
hash: Regex,
percent: Regex,
star_cont: Regex,
}
fn cleaners() -> &'static Cleaners {
static C: OnceLock<Cleaners> = OnceLock::new();
C.get_or_init(|| Cleaners {
block_open: Regex::new(r"^/\*+!?").unwrap(),
block_close: Regex::new(r"\*+/$").unwrap(),
lua_open: Regex::new(r"^--\[=*\[").unwrap(),
lua_close: Regex::new(r"\]=*\]$").unwrap(),
paren_star_open: Regex::new(r"^\(\*").unwrap(),
paren_star_close: Regex::new(r"\*\)$").unwrap(),
brace_open: Regex::new(r"^\{").unwrap(),
brace_close: Regex::new(r"\}$").unwrap(),
slashes: Regex::new(r"(?m)^//[/!]?\s?").unwrap(),
dashes: Regex::new(r"(?m)^--\s?").unwrap(),
hash: Regex::new(r"(?m)^#\s?").unwrap(),
percent: Regex::new(r"(?m)^%+\s?").unwrap(),
star_cont: Regex::new(r"(?m)^\s*\*\s?").unwrap(),
})
}
/// cleanCommentMarkers — strip comment syntax, keep the prose.
pub fn clean_comment_markers(comment: &str) -> String {
let c = cleaners();
let mut s = comment.trim().to_string();
if s.starts_with("/*") {
s = c.block_open.replace(&s, "").into_owned();
s = c.block_close.replace(&s, "").into_owned();
} else if s.starts_with("--[") {
s = c.lua_open.replace(&s, "").into_owned();
s = c.lua_close.replace(&s, "").into_owned();
} else if s.starts_with("(*") {
s = c.paren_star_open.replace(&s, "").into_owned();
s = c.paren_star_close.replace(&s, "").into_owned();
} else if s.starts_with('{') {
s = c.brace_open.replace(&s, "").into_owned();
s = c.brace_close.replace(&s, "").into_owned();
}
s = c.slashes.replace_all(&s, "").into_owned();
s = c.dashes.replace_all(&s, "").into_owned();
s = c.hash.replace_all(&s, "").into_owned();
s = c.percent.replace_all(&s, "").into_owned();
s = c.star_cont.replace_all(&s, "").into_owned();
s.trim().to_string()
}
/// getPrecedingDocstring — collect the comment run immediately preceding the
/// node (climbing out of declaration wrappers first), cleaned and joined.
/// Returns None when there is no preceding comment (a PRESENT-but-empty
/// docstring after cleaning still returns Some(""), matching the TS helper).
pub fn preceding_docstring(node: Node, src: &str) -> Option<String> {
let mut anchor = node;
while let Some(parent) = anchor.parent() {
if is_wrapper(parent.kind()) {
anchor = parent;
} else {
break;
}
}
let mut comments: Vec<&str> = Vec::new();
let mut sibling = anchor.prev_named_sibling();
while let Some(s) = sibling {
if is_comment(s.kind()) {
comments.push(&src[s.byte_range()]);
sibling = s.prev_named_sibling();
} else {
break;
}
}
if comments.is_empty() {
return None;
}
comments.reverse(); // collected nearest-first; TS unshifts to keep source order
Some(
comments
.iter()
.map(|c| clean_comment_markers(c))
.collect::<Vec<_>>()
.join("\n")
.trim()
.to_string(),
)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn strips_line_and_block_markers() {
assert_eq!(clean_comment_markers("// hello"), "hello");
assert_eq!(clean_comment_markers("/// doc line"), "doc line");
assert_eq!(
clean_comment_markers("/**\n * Adds things.\n * @param a first\n */"),
"Adds things.\n@param a first"
);
}
}
codegraph-kernel/src/tsjs/extractors.rs
+10 -10
View File
@@ -3,7 +3,7 @@
//! src/extraction/tree-sitter.ts; TS-file line references are as of the R2
//! port. Bug-for-bug fidelity is deliberate — fix the TS side first.
use super::util;
use crate::textutil as util;
use super::{
body_of, is_builtin_type, is_literal_receiver, is_react_hoc, is_variable_type,
is_vue_collection_name, Extra, Scope, Walker,
@@ -43,7 +43,7 @@ impl<'t> Walker<'t> {
}
let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src),
docstring: crate::docstring::preceding_docstring(node, self.src),
signature: self.signature_of(node),
visibility: self.visibility_of(node),
is_exported: Some(self.is_exported(node)),
@@ -120,7 +120,7 @@ impl<'t> Walker<'t> {
let resolved_body = body_of(node); // skipBodilessClass unset for TS/JS
let name = self.extract_name(node);
let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src),
docstring: crate::docstring::preceding_docstring(node, self.src),
visibility: self.visibility_of(node),
is_exported: Some(self.is_exported(node)),
..Extra::default()
@@ -161,7 +161,7 @@ impl<'t> Walker<'t> {
let name = self.extract_name(node);
let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src),
docstring: crate::docstring::preceding_docstring(node, self.src),
signature: self.signature_of(node),
visibility: self.visibility_of(node),
is_async: Some(self.is_async(node)),
@@ -187,7 +187,7 @@ impl<'t> Walker<'t> {
pub(super) fn extract_interface(&mut self, node: Node<'t>) {
let name = self.extract_name(node);
let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src),
docstring: crate::docstring::preceding_docstring(node, self.src),
is_exported: Some(self.is_exported(node)),
..Extra::default()
};
@@ -209,7 +209,7 @@ impl<'t> Walker<'t> {
let Some(body) = body_of(node) else { return };
let name = self.extract_name(node);
let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src),
docstring: crate::docstring::preceding_docstring(node, self.src),
visibility: self.visibility_of(node),
is_exported: Some(self.is_exported(node)),
..Extra::default()
@@ -255,7 +255,7 @@ impl<'t> Walker<'t> {
// --- extractProperty (#808 property-classified class fields) ---------------------
pub(super) fn extract_property(&mut self, node: Node<'t>) -> Option<(u32, String)> {
let docstring = super::docstring::preceding_docstring(node, self.src);
let docstring = crate::docstring::preceding_docstring(node, self.src);
let visibility = self.visibility_of(node);
let is_static = Some(self.is_static(node).unwrap_or(false)); // `?? false` — always present
@@ -296,7 +296,7 @@ impl<'t> Walker<'t> {
pub(super) fn extract_variable(&mut self, node: Node<'t>) {
let is_const = self.is_const_decl(node);
let kind: &'static str = if is_const { "constant" } else { "variable" };
let docstring = super::docstring::preceding_docstring(node, self.src);
let docstring = crate::docstring::preceding_docstring(node, self.src);
let is_exported = self.is_exported(node); // `?? false` — always present
for i in 0..node.named_child_count() {
@@ -807,7 +807,7 @@ impl<'t> Walker<'t> {
return false;
}
let extra = Extra {
docstring: super::docstring::preceding_docstring(node, self.src),
docstring: crate::docstring::preceding_docstring(node, self.src),
is_exported: Some(self.is_exported(node)),
..Extra::default()
};
@@ -858,7 +858,7 @@ impl<'t> Walker<'t> {
"property"
};
let extra = Extra {
docstring: super::docstring::preceding_docstring(child, self.src),
docstring: crate::docstring::preceding_docstring(child, self.src),
signature: Some(self.text(child).to_string()),
qualified_name: Some(format!("{alias_name}::{member_name}")),
..Extra::default()
codegraph-kernel/src/tsjs/mod.rs
+23 -7
View File
@@ -9,10 +9,9 @@
//! parity gate fails. Positions are emitted in UTF-16 code units (what
//! web-tree-sitter reports), see util::col16.
mod docstring;
mod extractors;
mod fnref;
pub(crate) mod util;
use crate::textutil as util;
use crate::buffers::{
build_meta, edge_kind_index, node_kind_index, Arena, BoolFlags, EdgeRow, EmitOut, NodeRow,
@@ -148,6 +147,11 @@ pub struct Walker<'t> {
arena: Arena,
tables: Tables,
stack: Vec<Scope>,
/// Node id string per row. Rows are unique but IDS COLLIDE for same
/// (kind, name, line) nodes — routine in minified one-line files — and the
/// TS extractor's fn-ref dedupe and value-ref self-checks key on the ID,
/// so parity requires comparing ids, not rows.
node_ids: Vec<String>,
/// Function/method names defined in this file (fn-ref flush gate).
defined_fn_names: HashSet<String>,
/// Simple names from `imports` refs (fn-ref flush gate).
@@ -198,6 +202,7 @@ pub fn extract(file_path: &str, source: &str, language: &str) -> Result<EmitOut,
arena: Arena::default(),
tables: Tables::default(),
stack: Vec::new(),
node_ids: Vec::new(),
defined_fn_names: HashSet::new(),
imported_names: HashSet::new(),
fn_ref_cands: Vec::new(),
@@ -234,6 +239,7 @@ pub fn extract(file_path: &str, source: &str, language: &str) -> Result<EmitOut,
return_type: NONE_STR,
extra_json: NONE_STR,
});
w.node_ids.push(ids::file_node_id(file_path));
w.stack.push(Scope { row: 0, kind: "file", name: base_name.to_string() });
w.visit_node(tree.root_node());
@@ -399,6 +405,7 @@ impl<'t> Walker<'t> {
target_id_str: NONE_STR,
});
self.node_ids.push(id);
if kind == "function" || kind == "method" {
self.defined_fn_names.insert(name.to_string());
}
@@ -485,7 +492,9 @@ impl<'t> Walker<'t> {
let refs_kind = edge_kind_index("references").unwrap();
for scope in &scopes {
let mut seen: HashSet<u32> = HashSet::new();
// Self-skip and per-scope dedupe compare node ID STRINGS (which
// collide for same-(kind, name, line) nodes), matching the TS side.
let mut seen: HashSet<&str> = HashSet::new();
let mut stack: Vec<Node> = vec![scope.node];
let mut visited = 0usize;
while let Some(n) = stack.pop() {
@@ -496,8 +505,12 @@ impl<'t> Walker<'t> {
if matches!(n.kind(), "identifier" | "constant" | "name" | "simple_identifier") {
let ref_name = self.text(n);
if let Some(&target_row) = targets.get(ref_name) {
if target_row != scope.row && ref_name != scope.name && !seen.contains(&target_row) {
seen.insert(target_row);
let target_id = self.node_ids[target_row as usize].as_str();
if target_id != self.node_ids[scope.row as usize]
&& ref_name != scope.name
&& !seen.contains(&target_id)
{
seen.insert(target_id);
let meta = self.arena.put(r#"{"valueRef":true}"#);
self.tables.push_edge(&EdgeRow {
source_idx: scope.row,
@@ -559,7 +572,7 @@ impl<'t> Walker<'t> {
if cands.is_empty() || util::is_generated_file(self.file_path) {
return;
}
let mut seen: HashSet<(u32, String)> = HashSet::new();
let mut seen: HashSet<(String, String)> = HashSet::new();
for (from, c) in cands {
// Gate: `this.<member>` always flushes; everything else must match
// a same-file function/method or an imported name. (The `::` and
@@ -571,7 +584,10 @@ impl<'t> Walker<'t> {
{
continue;
}
if !seen.insert((from, c.name.clone())) {
// Dedupe on the node ID STRING, not the row — ids collide for
// same-(kind, name, line) nodes (minified one-liners) and the TS
// side keys its dedupe on `${fromNodeId}|${name}`.
if !seen.insert((self.node_ids[from as usize].clone(), c.name.clone())) {
continue;
}
let column = util::col16(self.src, &self.line_starts, c.row, c.column_byte);
codegraph-kernel/src/tsjs/util.rs
-190
View File
@@ -1,190 +0,0 @@
//! Shared utilities for the TS/JS walker: compiled regexes, UTF-16 position
//! conversion, generated-file detection, and small text helpers — each
//! mirroring a specific helper in src/extraction/tree-sitter.ts (noted inline).
use regex::Regex;
use std::sync::OnceLock;
macro_rules! re {
($name:ident, $pat:expr) => {
pub fn $name() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new($pat).expect(concat!("regex ", stringify!($name))))
}
};
}
// RTK_HOOK_NAME_RE (tree-sitter.ts)
re!(rtk_hook_name, r"^use[A-Z][A-Za-z0-9]*(?:Query|Mutation)$");
// reactComponentHoc's styled test
re!(styled_callee, r"^styled\b");
// PascalCase component gate (#841)
re!(pascal_case, r"^[A-Z]");
// extractCall parenthesized-conversion normalization
re!(paren_conversion, r"^\(\s*\*?\s*([A-Za-z_][\w.]*)\s*\)$");
// flushFnRefCandidates SIMPLE_NAME
re!(simple_name, r"^[A-Za-z_$][A-Za-z0-9_$]*$");
// flushFnRefCandidates QUALIFIED_IMPORT
re!(qualified_import, r"^[A-Za-z_$][A-Za-z0-9_$.\\]*[.\\]([A-Za-z_$][A-Za-z0-9_$]*)$");
// captureFnRefCandidates rhs param-storage skip — trailing identifier of LHS
re!(lhs_last_name, r"([A-Za-z_$][A-Za-z0-9_$]*)\s*$");
// extractTsTupleContractNames identifier test
re!(ident_dollar, r"^[A-Za-z_$][A-Za-z0-9_$]*$");
// looksLikeVueStoreFile signal (VUE_STORE_FILE_SIGNAL)
re!(
vue_store_signal,
r"\bdefineStore\b|\bcreateStore\b|\bVuex\b|\bmutations\b|\bactions\b|\bgetters\b|\bnamespaced\b"
);
// value-ref target-name distinctiveness: /[A-Z_]/
re!(has_upper_or_underscore, r"[A-Z_]");
/// isGeneratedFile (src/extraction/generated-detection.ts) — full pattern list
/// ported so future language walkers share it.
pub fn is_generated_file(file_path: &str) -> bool {
static RES: OnceLock<Vec<Regex>> = OnceLock::new();
let patterns = RES.get_or_init(|| {
[
r"\.pb\.go$",
r"\.pulsar\.go$",
r"_grpc\.pb\.go$",
r"_mock\.go$",
r"_mocks\.go$",
r"^mock_[^/]+\.go$",
r"\.generated\.[jt]sx?$",
r"\.gen\.[jt]sx?$",
r"\.pb\.[jt]s$",
r"_pb\.[jt]s$",
r"_grpc_pb\.[jt]s$",
r"\.min\.m?js$",
r"_pb2(_grpc)?\.py$",
r"_pb2\.pyi$",
r"\.pb\.(cc|h)$",
r"\.g\.cs$",
r"Grpc\.cs$",
r"OuterClass\.java$",
r"Grpc\.java$",
r"\.pb\.swift$",
r"\.g\.dart$",
r"\.freezed\.dart$",
r"\.pb\.dart$",
r"\.pbgrpc\.dart$",
r"\.chopper\.dart$",
r"\.generated\.rs$",
]
.iter()
.map(|p| Regex::new(p).expect("generated pattern"))
.collect()
});
patterns.iter().any(|p| p.is_match(file_path))
}
/// Byte offsets of each line start, for UTF-16 column conversion.
pub fn line_starts(src: &str) -> Vec<usize> {
let mut out = vec![0usize];
for (i, b) in src.bytes().enumerate() {
if b == b'\n' {
out.push(i + 1);
}
}
out
}
/// UTF-16 code units in `s` — what web-tree-sitter (and JS string ops)
/// count, so kernel-emitted columns are byte-identical to the wasm path's.
pub fn utf16_len(s: &str) -> usize {
s.chars().map(|c| c.len_utf16()).sum()
}
/// Column (UTF-16 units) of `byte_pos` on line `row`, given `line_starts`.
pub fn col16(src: &str, starts: &[usize], row: usize, byte_pos: usize) -> u32 {
let ls = starts.get(row).copied().unwrap_or(0);
if byte_pos <= ls {
return 0;
}
utf16_len(&src[ls..byte_pos]) as u32
}
/// JS `String.prototype.slice(0, n)` in UTF-16 units, without splitting a
/// surrogate pair (when the cut would split one, we stop one code unit short —
/// a lone surrogate isn't representable in Rust and never round-trips through
/// SQLite anyway). Returns (sliced, was_truncated_at_or_beyond_n).
pub fn slice_utf16(s: &str, n: usize) -> (String, bool) {
let mut used = 0usize;
let mut out = String::new();
for c in s.chars() {
let w = c.len_utf16();
if used + w > n {
return (out, true);
}
used += w;
out.push(c);
if used == n {
// Exactly at the limit: truncated iff any source remains.
let truncated = out.len() < s.len();
return (out, truncated);
}
}
(out, false)
}
/// objectKeyName (tree-sitter.ts): strip ONE leading and ONE trailing quote
/// character (`'`, `"`, or backtick).
pub fn object_key_name(s: &str) -> String {
let mut out = s;
if let Some(first) = out.chars().next() {
if first == '\'' || first == '"' || first == '`' {
out = &out[first.len_utf8()..];
}
}
if let Some(last) = out.chars().last() {
if last == '\'' || last == '"' || last == '`' {
out = &out[..out.len() - last.len_utf8()];
}
}
out.to_string()
}
/// The `= <first 100 UTF-16 units>[...]` initializer signature used by
/// extractVariable (its `.length >= 100` check fires exactly when the slice
/// hit the cap).
pub fn init_signature(value_text: &str) -> String {
let (sliced, _) = slice_utf16(value_text, 100);
if utf16_len(&sliced) >= 100 {
format!("= {sliced}...")
} else {
format!("= {sliced}")
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn utf16_cols() {
let src = "aé😀b";
// 'a'=1, 'é'=1, '😀'=2 utf16 units; bytes: a=1, é=2, 😀=4
assert_eq!(utf16_len(src), 5);
let starts = line_starts(src);
assert_eq!(col16(src, &starts, 0, 1), 1); // after 'a'
assert_eq!(col16(src, &starts, 0, 3), 2); // after 'é'
assert_eq!(col16(src, &starts, 0, 7), 4); // after '😀'
}
#[test]
fn init_sig_short_and_long() {
assert_eq!(init_signature("[1, 2]"), "= [1, 2]");
let long = "x".repeat(150);
let sig = init_signature(&long);
assert!(sig.starts_with("= "));
assert!(sig.ends_with("..."));
assert_eq!(utf16_len(&sig[2..sig.len() - 3]), 100);
}
#[test]
fn generated_patterns() {
assert!(is_generated_file("src/api.generated.ts"));
assert!(is_generated_file("vendor/jquery.min.js"));
assert!(!is_generated_file("src/app.ts"));
}
}