feat(kernel): R4 — Java port with Lombok synthesis, gate passed, default-on

Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring
the wasm extractor's Java paths bug-for-bug: package namespaces,
imports, javadoc, annotations→decorates, type_list inheritance,
static-final constants, enum constants, anonymous classes (including
the TS side's 0-based-line quirk on the extends ref), method_invocation
calls with the this.field unwrap and the Foo.getInstance().bar() chain
encoding (#645/#608), static-member value reads, method-reference
fn-refs (#756), value-reference edges, and the full Lombok member
synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/
EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared
docstring/textutil modules moved to crate level. Grammar:
tree-sitter-java 0.23.5, with the wasm grammar vendored from the same
tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build.

Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit
341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test;
full-init dump-diffs byte-identical on gson (49,766 rows), retrofit
(62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux
container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1.

The gate caught a real cross-language bug: fn-ref dedupe and value-ref
self-target checks must compare node ID STRINGS, not node-table rows —
ids collide for same-(kind, name, line) nodes, which minified one-line
bundles hit routinely (retrofit's website JS exposed it; latent in the
TS/JS walker since R2, never released). Fixed in both walkers.

Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat
(parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that
wall is main-thread-bound (reads + store), not worker-CPU-bound — the
§6 expectation assumed otherwise. Where worker CPU binds the kernel
delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s
(~1.25×). The identified lever for the many-core headline is decoding
kernel buffers directly into store rows (skipping per-node JS object
materialization); the buffer contract already carries everything.

DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-07-16 23:25:29 -05:00
co-authored by Claude Fable 5
parent c8cca9a601
commit 03d54e47a1
19 changed files with 1828 additions and 30 deletions
+45 -2
View File
@@ -24,7 +24,12 @@ Work top to bottom; each step has a section below with the detail.
**passed + DEFAULT-ON 2026-07-16, see §4b.** One deferred leg: Windows-VM run
(VM stopped, `prlctl start` needs Parallels Pro — benign: no .node ⇒ wasm fallback;
the release matrix builds + gates win32 prebuilds).
- [ ] **R4. Port Java** → re-run the dubbo benchmark → the cbm-parity headline. (§4, §6)
- [x] **R4. Port Java** → re-run the dubbo benchmark → the cbm-parity headline. (§4, §6)
**ported + gate passed + DEFAULT-ON 2026-07-16, see §4c.** Benchmark reality
check: dubbo's parse-loop WALL on many-core machines is main-thread-bound (store/
dispatch), so the Mac headline barely moves (~11.3→11.1s; parse-loop 5.0→4.4s);
the win shows where worker CPU binds (dubbo on 2-CPU: 28→22.5s, ~1.25×). The
identified follow-up lever for the Mac number is decode-direct-to-store (§4c).
- [ ] **R5. Port Python, Go.** (§4)
- [ ] **R6. Kernel-scale re-validation** in the cg1212 container (expect parse 6m → ~2m). (§6)
- [ ] **R7. Long-tail languages opportunistically** per the tracker; T3 may stay TS forever. (§4)
@@ -221,6 +226,44 @@ Default routing: `DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}` in
`src/extraction/kernel/index.ts`. `CODEGRAPH_KERNEL_LANGS` REPLACES the set;
`CODEGRAPH_KERNEL=0` kills. Changelog entry added under [Unreleased].
### 4c. R4 — Java PORTED + gate PASSED + DEFAULT-ON (2026-07-16)
- **Walker:** `codegraph-kernel/src/java.rs` (self-contained, sharing the crate-level
docstring/textutil modules) — package namespaces, imports, javadoc, annotations →
decorates, type_list inheritance, fields/constants (static-final → constant),
enum_constant members, anonymous classes (`<T$anon@line>` incl. the TS side's
0-based-line quirk, mirrored bug-for-bug), method_invocation calls with the
`this.field` unwrap + the `Foo.getInstance().bar()` chain encoding, static-member
value reads, method_reference fn-refs (`this::x` / `Type::m`), value refs, and the
**full Lombok member synthesizer** (#912: @Getter/@Setter/@Data/@Value/@Builder/
@ToString/@EqualsAndHashCode/@Slf4j-family, taken-member dedup by exact
`classQN::name`). Grammar: tree-sitter-java crate 0.23.5; wasm vendored from the
SAME tag (94703d5, parser.c sha-matched), replacing tree-sitter-wasms' ^0.20.2 build.
- **Parity:** extraction sweeps — gson 262/262, retrofit 341/341, dubbo 4,048/4,048,
torture fixture (`__tests__/fixtures/kernel-parity/Torture.java`, in `npm test`).
Full-init dump-diffs byte-identical: gson (49,766 rows), retrofit (62,735),
**dubbo (441,266 rows)**. All R2/R3 repos re-verified after the fix below.
- **The gate caught a REAL cross-language bug:** retrofit's minified website JS
exposed that fn-ref dedupe and value-ref self-checks must compare node **ID
strings**, not table rows — IDs collide for same-(kind,name,line) nodes (routine in
minified one-liners: many `function e` on line 3) and the TS side keys on
`${fromNodeId}|${name}`. Fixed in BOTH walkers (`node_ids` per row); this affected
tsjs too (latent since R2, never released).
- **Benchmark honesty (the §6 expectation was wrong about WHERE the win lands):**
dubbo fresh-init on the 11-core M3 Pro is ~FLAT end-to-end (11.311.5 wasm →
11.011.6 kernel; parse-loop wall 5,020→4,394ms) because that phase's wall is
**main-thread-bound** (file reads + result store + SQLite), not worker-CPU-bound —
8 wasm workers already hide extraction CPU behind the main thread on big-core
machines. Where worker CPU binds, the kernel delivers: **dubbo on 2-CPU/6GB Linux
27.828.6s → 22.322.8s (~1.25×)**; excalidraw same envelope ~1.5×; vscode-on-Mac
1.28×. The **cbm-parity Mac headline therefore needs the next lever: decode the
kernel's buffers DIRECTLY into store rows** (skip per-node JS object
materialization on the main thread) — buffer contract already carries everything;
tracked as the top §7a-adjacent follow-up.
- **Platforms:** Linux container (arm64): all 23 kernel tests green EXPECT=1;
Windows VM still deferred (same fallback rationale as §4b).
- Default routing now includes `java`.
## 4. Per-language tracker
Tiers: **T1** = mostly `.scm` + mapping config. **T2** = needs bespoke pre/post passes kept
@@ -239,7 +282,7 @@ parity before porting the language.
| Language(s) | Today | Tier | Grammar source | Migration notes / known traps | Status |
|---|---|---|---|---|---|
| typescript, tsx, javascript, jsx | `languages/typescript.ts`, `javascript.ts` + shared branches | T1 | crates.io | First target. Value-reference edges (#895/#897) and component recognition (#841 forwardRef/memo/styled) must survive — they're extraction-side. Largest test surface; gate is strictest here. **PORTED + GATE PASSED + DEFAULT-ON (§4a/§4b); erroring files defer to wasm per-file.** | ✅ |
| java | `languages/java.ts` | T1 | crates.io | Second target; unlocks the dubbo-parity claim. Lombok member synthesis (#912) is a NODE synthesizer hook in extraction (`synthesizeMembers`) — port or keep as TS post-pass. | |
| java | `languages/java.ts` | T1 | crates.io | Second target; unlocks the dubbo-parity claim. Lombok member synthesis (#912) is a NODE synthesizer hook in extraction (`synthesizeMembers`) — port or keep as TS post-pass. **PORTED incl. Lombok + gate passed + DEFAULT-ON (§4c).** | |
| python | `languages/python.ts` | T1 | crates.io | Third. Decorator extraction feeds framework route detection — parity required. | ☐ |
| go | `languages/go.ts` | T1 | crates.io | Third (tie). Value-reference edges ship here too (#897). | ☐ |
| ruby, php | dedicated files | T1 | crates.io | Straightforward; PHP property-receiver shapes (#1220/#1251) are RESOLUTION-side, unaffected. | ☐ |