feat(kernel): R7b Scala walker — scala module, vendored-grammar-C master@0aca5d0a6f, scala default-routed (#1385)
R7b batch 4 #3 (docs/design/scala-kernel-port-checklist.md is the authoritative quirk list). The third vendored-grammar-C language and the biggest grammar in the tree (35MB parser.c): the vendored wasm is tree-sitter/tree-sitter-scala master@0aca5d0a6f — a post-v0.26.0 generation sync that is not a release (the 0.26.0 crate is 30 states BEHIND, so a crate pin would be a silent downgrade). NO wasm change: production has parsed with this exact revision since #91 — the kernel-grammar-parity row (ABI 15, 26,650 states, 32 fields, id-by-id tables) is the whole alignment proof. Preserved bug-for-bug (all probe-pinned): the leak-through asymmetries — extension methods mint NO nodes (first def's body calls leak to the enclosing scope, later defs invisible, and the braced form resolves its body field to the `{` TOKEN via first-match-wins field lookup → whole extension invisible); anonymous `new T { … }` template_body members leak to the enclosing scope (findAnonymousClassBody misses template_body); the bodied-vs-bodiless class asymmetry (bodiless headers walk class_parameters → default-value calls emit FROM the class; bodied ones never see them) — plus first-segment import names (`import com.example.C` → `com`), the val/var hook keyed on the enclosing-definition NODE TYPE (object vals → constants/value-ref targets, class/trait/enum/given vals → fields) with consumed initializers, every def routed through extractMethod with the top-level function fallback, nested defs in bodies minting NOTHING (the inverse of kotlin) while body-local classes extract fully, curried signatures keeping only the FIRST parameter list (type params win the `parameters` field), enum cases positioned at the CASE node with invisible params/extends tails, extends with-chains via scalaBaseTypeName, `@deprecated(args)` decorates, the #750 capitalized-chain re-encode (`WidgetS.create().render`), literal-receiver silence, static-member reads AND writes, infix invisibility, `derives` silence, scaladoc retention with the CRLF `\r` pin, full value-reference machinery (shadow prune, last-wins same-name targets, `$X`/`${X}` interpolation reads), and SCALA_SPEC fn-refs (bare ids + postfix eta unwrap + varinit, var-init non-capture). Gates: parity sweeps first-run 0-diff on os-lib/cats/scala3-compiler-src/ scala3-library-src — 1,935 clean files byte-parity, deferrals 0/15/57/116 matching the survey's predictions exactly (scala-3's PHANTOM hasError files — flag-true, zero ERROR nodes, capture-checking `^` — defer on the FLAG); full-init dumps byte-identical ×3 (os-lib, cats, scala3 whole-repo 950,889 dump lines); kernel-scala-parity suite (9 fixtures + 9 in-memory CRLF variants incl. Scala-3 indentation through the external scanner + phantom/real-error defer pins + first-segment/namespace/value-ref pins); full suite 2,669 green ×3 with CODEGRAPH_KERNEL_EXPECT=1 (kernel-scaffold's stays-wasm example moved scala → pascal). DEFAULT_ROUTED += scala (19 langs). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
e32135171e
commit
bdd687b49f
+1
-1
@@ -11,7 +11,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
### New Features
|
||||
|
||||
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, C++, Rust, C#, Ruby, PHP, Swift, Kotlin, R, Lua, and Luau projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, django-, git-, protobuf-, tokio-, rust-analyzer-, jellyfin-, rails-, symfony-, swift-nio-, kotlinx.coroutines-, ggplot2-, and Kong-scale codebases (Lombok-generated members, C function-pointer tables, and Unreal-Engine-style macro-heavy headers included; CUDA and Metal sources ride the C++ path). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
|
||||
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, C++, Rust, C#, Ruby, PHP, Swift, Kotlin, Scala, R, Lua, and Luau projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, django-, git-, protobuf-, tokio-, rust-analyzer-, jellyfin-, rails-, symfony-, swift-nio-, kotlinx.coroutines-, ggplot2-, Kong-, and Scala-3-compiler-scale codebases (Lombok-generated members, C function-pointer tables, and Unreal-Engine-style macro-heavy headers included; CUDA and Metal sources ride the C++ path). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
|
||||
- Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=<count>` to tune when it engages.
|
||||
- Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle.
|
||||
- The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical.
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
/** Scaladoc kept?
|
||||
* second line with star
|
||||
*/
|
||||
def scaladocOnly(): Int = 1
|
||||
|
||||
// line one
|
||||
// line two
|
||||
def lineRun(): Int = 2
|
||||
|
||||
/** block doc */
|
||||
// trailing line
|
||||
def mixedBlockThenLine(): Int = 3
|
||||
|
||||
// leading line
|
||||
/** block after line */
|
||||
def mixedLineThenBlock(): Int = 4
|
||||
|
||||
// detached
|
||||
|
||||
def blankLineBetween(): Int = 5
|
||||
|
||||
/** class doc */
|
||||
class DocClass {
|
||||
/** member doc */
|
||||
def documented(): Int = 6
|
||||
/** val doc */
|
||||
val docVal = 7
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
extension (s: String)
|
||||
def doubleUp: String = concat(s, s)
|
||||
def tripleUp: String = concat2(s, concat3(s, s))
|
||||
|
||||
extension (t: Int) {
|
||||
def bracedA: Int = callA(t)
|
||||
def bracedB: Int = callB(t)
|
||||
}
|
||||
|
||||
extension [A](xs: List[A]) def secondOpt: Option[A] = xs.drop(1).headOption
|
||||
extension (t: Int) {
|
||||
def bracedA: Int = callA(t)
|
||||
def bracedB: Int = callB(t)
|
||||
}
|
||||
extension (t: Int) {
|
||||
def bracedA: Int = callA(t)
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
object Handlers {
|
||||
def handler(): Unit = ()
|
||||
def other(): Unit = ()
|
||||
def wire(): Unit = {
|
||||
register(handler)
|
||||
registerEta(other _)
|
||||
val stored = handler
|
||||
obj.cb = other
|
||||
this.cb = cb
|
||||
register(missing)
|
||||
}
|
||||
val topStored = handler
|
||||
var cb: () => Unit = other
|
||||
}
|
||||
object H2 {
|
||||
def alpha(): Unit = ()
|
||||
def beta(): Unit = ()
|
||||
def gamma(): Unit = ()
|
||||
def delta(): Unit = ()
|
||||
def wire(): Unit = {
|
||||
val stored = alpha
|
||||
obj.cb = beta
|
||||
val eta = gamma _
|
||||
sink.handler = sink.handler
|
||||
}
|
||||
val listStored = List(delta)
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
object Registry {
|
||||
given regOrd: Ordering[Int] = new Ordering[Int] {
|
||||
def compare(a: Int, b: Int): Int = cmpHelper(a, b)
|
||||
val innerVal = 5
|
||||
}
|
||||
def member(): Int = 1
|
||||
}
|
||||
given topOrd: Ordering[Long] = new Ordering[Long] {
|
||||
def compare(a: Long, b: Long): Int = cmpTop(a, b)
|
||||
val topInnerVal = 6
|
||||
}
|
||||
class BodyStmt(size: Int) {
|
||||
require(size > 0)
|
||||
logInit(this)
|
||||
val fnField = (x: Int) => runLam(x)
|
||||
}
|
||||
object Boot extends App {
|
||||
bootUp()
|
||||
def local(): Unit = ()
|
||||
}
|
||||
object NamedArgs {
|
||||
def cbTarget(): Unit = ()
|
||||
def sink(): Unit = {
|
||||
wire(cb = cbTarget)
|
||||
forward(cb = cb)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
package com.ex
|
||||
|
||||
object Indented:
|
||||
val member = compute()
|
||||
def m(): Int =
|
||||
val local = helperI()
|
||||
local + 1
|
||||
|
||||
class IndentedC extends BaseI:
|
||||
def n(): Int = member
|
||||
|
||||
enum ColorI:
|
||||
case Red, Green
|
||||
def label(): String = "x"
|
||||
|
||||
trait TraitI:
|
||||
def tm(): Int
|
||||
|
||||
given ordI: Ordering[Int] = summonOrd()
|
||||
|
||||
extension (s: String)
|
||||
def extI: Int = s.length
|
||||
|
||||
def topI(): Int =
|
||||
if cond then one() else two()
|
||||
@@ -0,0 +1,29 @@
|
||||
package a.b {
|
||||
class Nested1 { def nm(): Int = 1 }
|
||||
}
|
||||
trait SelfTyped { self: Nested1 =>
|
||||
def stm(): Int = 2
|
||||
}
|
||||
class Outer2 {
|
||||
type Member = List[Int]
|
||||
object InnerObj { val IC = 1 }
|
||||
class InnerCls { def icm(): Int = 2 }
|
||||
}
|
||||
private[b] class QualPriv
|
||||
class `Weird Name` { def `strange def`(): Int = 3 }
|
||||
val multiA, multiB = 5
|
||||
class CurryCall {
|
||||
def run(): Unit = {
|
||||
curried(1)(2)
|
||||
Foo(1).bar()
|
||||
Foo(1)(2)
|
||||
new Widget(make())
|
||||
}
|
||||
}
|
||||
class A extends Base(1)(2) { def m(): Int = 1 }
|
||||
import single
|
||||
import a.b
|
||||
object Uni {
|
||||
val note = "ééé €€ 😀 end"
|
||||
def afteré(): Int = { helperé() }
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
val scriptVal = 10
|
||||
def scriptFn(): Int = scriptVal + 1
|
||||
println(scriptFn())
|
||||
runTop(scriptVal)
|
||||
@@ -0,0 +1,19 @@
|
||||
object Config {
|
||||
val RETRY_MAX = 3
|
||||
val TIMEOUT_MS = 500
|
||||
val SHARED_TABLE = Map("a" -> 1)
|
||||
val count = 9
|
||||
}
|
||||
val TOP_LIMIT = 99
|
||||
class Reader {
|
||||
def readTop(): Int = TOP_LIMIT + 1
|
||||
def readBoth(): Int = {
|
||||
val RETRY_MAX = 9
|
||||
RETRY_MAX + Config.TIMEOUT_MS
|
||||
}
|
||||
def readInterp(): String = s"limit $TIMEOUT_MS and ${SHARED_TABLE}"
|
||||
def readCount(): Int = count
|
||||
}
|
||||
object Config2 {
|
||||
val TIMEOUT_MS = 1
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
package com.example.torture
|
||||
|
||||
import com.example.other.OtherClass
|
||||
import com.example.multi.{Alpha, Beta}
|
||||
import com.example.wild._
|
||||
import com.example.star.*
|
||||
import com.example.alias.{LongName => Short}
|
||||
|
||||
/** Scaladoc for topLevel — kept? */
|
||||
def topLevel(a: Int): WidgetS = new WidgetS(a)
|
||||
|
||||
// line one
|
||||
// line two
|
||||
def lineDoc(): Int = 1
|
||||
|
||||
def curried(a: Int)(b: String)(implicit ord: Ordering[Int]): Int = a
|
||||
|
||||
def genericDef[A: Numeric, B <: BoundT](x: A): B = ???
|
||||
|
||||
def inferredRet(a: Int) = a + 1
|
||||
|
||||
def unitRet(): Unit = ()
|
||||
|
||||
def genericLeak[T](t: T): T = t
|
||||
|
||||
def qualRet(w: WidgetS): com.example.other.Remote = ???
|
||||
|
||||
private def privTop(): Int = 1
|
||||
|
||||
@main def entry(): Unit = topLevel(1)
|
||||
|
||||
@deprecated("gone", "1.0") def old(): Int = 0
|
||||
|
||||
val topVal: Int = 3
|
||||
var topVar = 4
|
||||
lazy val topLazy = compute()
|
||||
val (tupA, tupB) = (1, 2)
|
||||
val Some(extracted) = Option(9)
|
||||
val SHARED_TABLE: Map[String, Int] = Map.empty
|
||||
val topInit = WidgetS.create()
|
||||
|
||||
class WidgetS(val size: Int, label: String = defaultLabel()) extends BaseW(size) with Drawable with Ordered[WidgetS] {
|
||||
val cachedName: String = "w"
|
||||
var mutable = 0
|
||||
private val secret = 3
|
||||
@volatile var annotatedField: Int = 0
|
||||
|
||||
def this() = this(0, "d")
|
||||
|
||||
def render(): RenderResult = {
|
||||
val local = helperR(size)
|
||||
Registry.register(this)
|
||||
new RenderResult(local)
|
||||
}
|
||||
|
||||
def +(other: WidgetS): WidgetS = new WidgetS(size + other.size)
|
||||
|
||||
def compare(that: WidgetS): Int = size - that.size
|
||||
|
||||
@inline def fast(): Int = 1
|
||||
|
||||
protected def hook(): Unit = ()
|
||||
}
|
||||
|
||||
object WidgetS {
|
||||
val DEFAULT_SIZE = 10
|
||||
def create(): WidgetS = new WidgetS(DEFAULT_SIZE, "c")
|
||||
def readDefault(): Int = DEFAULT_SIZE + 1
|
||||
}
|
||||
|
||||
case class DataS(x: Int, y: String = mkY())
|
||||
|
||||
case object SingletonS extends MarkerT
|
||||
|
||||
abstract class AbsS {
|
||||
def abstractM(): Int
|
||||
def concrete(): Int = abstractM() + 1
|
||||
}
|
||||
|
||||
trait Drawable {
|
||||
def draw(): Unit
|
||||
def area(): Double = 0.0
|
||||
val traitVal: Int = 7
|
||||
}
|
||||
|
||||
sealed trait Shape
|
||||
object Circle extends Shape
|
||||
class Square extends Shape
|
||||
|
||||
trait MarkerT
|
||||
|
||||
class DelegatedImpl(d: Drawable) extends Drawable {
|
||||
def draw(): Unit = d.draw()
|
||||
}
|
||||
|
||||
enum Http {
|
||||
case Ok, NotFound
|
||||
case Custom(code: Int)
|
||||
def describe(): String = s"code"
|
||||
}
|
||||
|
||||
enum Planet(mass: Double) {
|
||||
case Earth extends Planet(5.9)
|
||||
case Mars extends Planet(0.6)
|
||||
}
|
||||
|
||||
given intOrd: Ordering[WidgetS] = new Ordering[WidgetS] {
|
||||
def compare(a: WidgetS, b: WidgetS): Int = a.size - b.size
|
||||
}
|
||||
|
||||
given String = "anon"
|
||||
|
||||
extension (s: String)
|
||||
def doubleUp: String = concat(s, s)
|
||||
def tripleUp: String = concat(s, concat(s, s))
|
||||
|
||||
extension [A](xs: List[A]) def secondOpt: Option[A] = xs.drop(1).headOption
|
||||
|
||||
implicit class RichIntS(val i: Int) {
|
||||
def twice: Int = i * 2
|
||||
}
|
||||
|
||||
type WidgetList = List[WidgetS]
|
||||
opaque type Meters = Double
|
||||
|
||||
object CallSites {
|
||||
def run(w: WidgetS): Unit = {
|
||||
helper(1)
|
||||
w.render()
|
||||
WidgetS(1)
|
||||
WidgetS.create().render()
|
||||
lowerFactory().chain()
|
||||
a.b.method3()
|
||||
this.mine()
|
||||
super.hashCode()
|
||||
"lit".toUpperCase()
|
||||
5.toString()
|
||||
genericCall[Int](1)
|
||||
val n2 = new Ordering[Int] { def compare(p: Int, q: Int): Int = p - q }
|
||||
val s1 = s"plain $topVal and ${w.render()} end"
|
||||
val e1 = registerCb(handler _)
|
||||
val e2 = handler _
|
||||
registerCb(handler)
|
||||
val r1 = list map transform
|
||||
val r2 = 1 :: rest
|
||||
w match {
|
||||
case ws: WidgetS => use(ws)
|
||||
case _ => ()
|
||||
}
|
||||
for { x <- xs; y <- ys if y > 0 } yield combineXY(x, y)
|
||||
}
|
||||
def mine(): Int = 1
|
||||
def handler(): Unit = ()
|
||||
def localHost(): Int = {
|
||||
def innerFn(k: Int): Int = k * 2
|
||||
val lam = (q: Int) => q + 1
|
||||
class LocalClass { def lm(): Int = 3 }
|
||||
object LocalObj { def om(): Int = 4 }
|
||||
innerFn(lam(1)) + helperCall()
|
||||
}
|
||||
}
|
||||
|
||||
object StaticReads {
|
||||
def reads(): Unit = {
|
||||
val c1 = Registry.count
|
||||
val c2 = Http.Ok
|
||||
val c3 = com.example.Fq.CONST_READ
|
||||
Registry.count = 5
|
||||
process(Registry.count)
|
||||
Registry.register(null)
|
||||
}
|
||||
}
|
||||
|
||||
package object utilpkg {
|
||||
def pkgHelper(): Int = 1
|
||||
val pkgShared = 2
|
||||
}
|
||||
@@ -36,7 +36,7 @@ const kernelBuilt = fs.existsSync(KERNEL_PATH);
|
||||
|
||||
// Every kernel-capable language. `jsx` shares the javascript grammar on BOTH
|
||||
// paths (langs.rs mirrors WASM_GRAMMAR_FILES), so the distinct grammars are:
|
||||
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau'];
|
||||
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau', 'scala'];
|
||||
|
||||
describe.skipIf(!kernelBuilt)('kernel↔wasm grammar parity', () => {
|
||||
beforeAll(async () => {
|
||||
|
||||
@@ -73,11 +73,11 @@ describe.skipIf(!kernelBuilt)('kernel scaffold', () => {
|
||||
});
|
||||
|
||||
it('TS/JS family + Java + Python + Go route to the kernel by default; others stay wasm', () => {
|
||||
for (const lang of ['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'ruby', 'php', 'swift', 'kotlin'] as const) {
|
||||
for (const lang of ['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'ruby', 'php', 'swift', 'kotlin', 'scala'] as const) {
|
||||
expect(kernelRoutes(lang), lang).toBe(true);
|
||||
}
|
||||
expect(kernelRoutes('scala')).toBe(false);
|
||||
expect(tryKernelExtract('src/a.scala', 'object A { def f(): Int = 1 }\n', 'scala')).toBeNull();
|
||||
expect(kernelRoutes('pascal')).toBe(false);
|
||||
expect(tryKernelExtract('src/a.pas', 'program A;\nbegin\nend.\n', 'pascal')).toBeNull();
|
||||
// CODEGRAPH_KERNEL_LANGS REPLACES the default set when present.
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'tsx';
|
||||
expect(kernelRoutes('typescript')).toBe(false);
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
/**
|
||||
* Kernel↔wasm Scala extraction parity (R7b batch 4 of the kernel migration).
|
||||
*
|
||||
* Asserts the native walker (codegraph-kernel/src/scala.rs) produces the SAME
|
||||
* ExtractionResult as the wasm TreeSitterExtractor — nodes, edges, and
|
||||
* unresolved refs compared as canonicalized multisets — over the checked-in
|
||||
* fixtures (torture.scala: first-segment imports, defs-as-methods with the
|
||||
* top-level function fallback, curried/type-params-first signatures, the
|
||||
* val/var hook with object-vs-class kinds and initializer invisibility,
|
||||
* companion pairs sharing a QN namespace, the bodiless-header asymmetry,
|
||||
* enum cases at case-node positions with invisible tails, extends
|
||||
* with-chains, `@deprecated(args)` decorates, the #750 capitalized-chain
|
||||
* re-encode, literal-receiver silence, static reads incl. the write-LHS
|
||||
* emission, nested-def invisibility with body-local classes extracting
|
||||
* fully; TortureDocs: scaladoc retention + the CRLF `\r` pin; TortureVref:
|
||||
* value-ref targets, shadow prune, interpolation reads, the last-wins
|
||||
* mis-target; TortureFnref: all five capture channels + var-init
|
||||
* non-capture + eta expansion; TortureGiven/TortureExt: the anon-body and
|
||||
* extension leak asymmetries — the port's likeliest regression sites;
|
||||
* TortureIndent: Scala-3 indentation syntax through the external scanner;
|
||||
* TortureMisc: package objects/braced packages/self-types/super-ctor args/
|
||||
* unicode columns; TortureScript.sc: top-level statements from the FILE)
|
||||
* and their CRLF variants (derived in-memory — #1329), plus phantom and
|
||||
* real-error defer pins.
|
||||
*
|
||||
* The full-repo sweeps live in scripts/kernel-parity.mjs (os-lib/cats +
|
||||
* scala3 compiler/src + library/src with --max-deferral 0.3); this suite
|
||||
* keeps the invariant alive in `npm test`. Skips when no kernel binary is
|
||||
* staged; CODEGRAPH_KERNEL_EXPECT=1 turns that into a failure.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { extractFromSource } from '../src/extraction';
|
||||
import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars';
|
||||
import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel';
|
||||
import type { ExtractionResult } from '../src/types';
|
||||
|
||||
const KERNEL_PATH = path.join(
|
||||
__dirname,
|
||||
'..',
|
||||
'codegraph-kernel',
|
||||
'prebuilds',
|
||||
`${process.platform}-${process.arch}`,
|
||||
'codegraph-kernel.node'
|
||||
);
|
||||
const kernelBuilt = fs.existsSync(KERNEL_PATH);
|
||||
|
||||
const FIXTURE_DIR = path.join(__dirname, 'fixtures', 'kernel-parity');
|
||||
|
||||
function canon(result: ExtractionResult): { nodes: string[]; edges: string[]; refs: string[] } {
|
||||
return {
|
||||
nodes: result.nodes
|
||||
.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort()))
|
||||
.sort(),
|
||||
edges: result.edges.map((e) => JSON.stringify(e, Object.keys(e).sort())).sort(),
|
||||
refs: result.unresolvedReferences
|
||||
.map((r) => JSON.stringify(r, Object.keys(r).sort()))
|
||||
.sort(),
|
||||
};
|
||||
}
|
||||
|
||||
const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const;
|
||||
let savedEnv: Record<string, string | undefined>;
|
||||
|
||||
describe.skipIf(!kernelBuilt)('kernel Scala extraction parity', () => {
|
||||
beforeAll(async () => {
|
||||
await initGrammars();
|
||||
await loadGrammarsForLanguages(['scala']);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]]));
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
for (const k of ENV_KEYS) {
|
||||
if (savedEnv[k] === undefined) delete process.env[k];
|
||||
else process.env[k] = savedEnv[k];
|
||||
}
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
function assertParity(filePath: string, source: string, minNodes = 2): ExtractionResult {
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
const viaKernel = tryKernelExtract(filePath, source, 'scala');
|
||||
expect(viaKernel, `kernel extraction failed for ${filePath}`).not.toBeNull();
|
||||
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource(filePath, source, 'scala');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
|
||||
const k = canon(viaKernel!);
|
||||
const w = canon(viaWasm);
|
||||
expect(k.nodes, `${filePath}: nodes`).toEqual(w.nodes);
|
||||
expect(k.edges, `${filePath}: edges`).toEqual(w.edges);
|
||||
expect(k.refs, `${filePath}: refs`).toEqual(w.refs);
|
||||
expect(viaWasm.nodes.length).toBeGreaterThanOrEqual(minNodes);
|
||||
return viaKernel!;
|
||||
}
|
||||
|
||||
const FIXTURES = [
|
||||
['torture.scala', 40],
|
||||
['TortureDocs.scala', 3],
|
||||
['TortureVref.scala', 4],
|
||||
['TortureFnref.scala', 4],
|
||||
['TortureGiven.scala', 3],
|
||||
['TortureExt.scala', 1],
|
||||
['TortureIndent.scala', 3],
|
||||
['TortureMisc.scala', 4],
|
||||
['TortureScript.sc', 1],
|
||||
] as const;
|
||||
|
||||
for (const [file, minNodes] of FIXTURES) {
|
||||
it(`${file}: parity`, () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, file), 'utf8');
|
||||
assertParity(`fixtures/${file}`, src, minNodes);
|
||||
});
|
||||
|
||||
it(`${file}: CRLF parity`, () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, file), 'utf8');
|
||||
const crlf = src.replace(/(?<!\r)\n/g, '\r\n');
|
||||
assertParity(`fixtures/${file} (crlf)`, crlf, minNodes);
|
||||
});
|
||||
}
|
||||
|
||||
it('torture pins: import first-segment names, companion pairs, value-ref edges', () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, 'torture.scala'), 'utf8');
|
||||
const result = assertParity('fixtures/torture.scala', src, 40);
|
||||
// Imports are named the FIRST path segment.
|
||||
const imports = result.nodes.filter((n) => n.kind === 'import');
|
||||
expect(imports.length).toBeGreaterThan(0);
|
||||
expect(imports.every((n) => !n.name.includes('.'))).toBe(true);
|
||||
// No namespace node, ever (package headers ignored).
|
||||
expect(result.nodes.some((n) => n.kind === 'namespace')).toBe(false);
|
||||
// Value-ref edges exist and are metadata-tagged.
|
||||
expect(result.edges.some((e) => e.kind === 'references' && e.metadata?.valueRef === true)).toBe(
|
||||
true
|
||||
);
|
||||
});
|
||||
|
||||
it('scala-3 PHANTOM hasError defers (flag-true, zero ERROR nodes)', () => {
|
||||
// Capture-checking postfix `^` — a complete, correct CST whose hasError
|
||||
// flag is still true. The kernel must defer on the FLAG.
|
||||
const phantom = 'def f(x: List[Int]^): Int = 1\n';
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(tryKernelExtract('src/phantom.scala', phantom, 'scala')).toBeNull();
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource('src/phantom.scala', phantom, 'scala');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(viaWasm.nodes.some((n) => n.kind === 'file')).toBe(true);
|
||||
});
|
||||
|
||||
it('real parse errors defer (given-with syntax)', () => {
|
||||
const broken = 'trait C\ngiven x: C with { def y = 1 }\n';
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(tryKernelExtract('src/gw.scala', broken, 'scala')).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -39,4 +39,25 @@ fn main() {
|
||||
lua.flag_if_supported("-utf-8"); // msvc
|
||||
lua.compile("tree-sitter-lua");
|
||||
println!("cargo:rerun-if-changed=grammars/lua");
|
||||
|
||||
// Scala grammar — vendored C (third vendored-grammar-C language): the
|
||||
// vendored wasm is tree-sitter/tree-sitter-scala master@0aca5d0a6f (the
|
||||
// 2026-04-22 generation sync — 30 states PAST the v0.26.0 tag/crate, so a
|
||||
// crate pin would be a silent downgrade). Sources are that commit's
|
||||
// checked-in generated artifacts, sha-recorded in the scala checklist
|
||||
// §Grammar prep:
|
||||
// parser.c bc3c3c794f19461d99d04de6c31d57fa3e41243509b9ab023a9b88ed3273d102
|
||||
// scanner.c e4ba242568ee3493015598997bf60f613802616eade62717c21109287ef64752
|
||||
// parser.c is 35 MB — the biggest grammar in the tree; expect a slow cc
|
||||
// step on clean builds.
|
||||
let mut scala = cc::Build::new();
|
||||
scala.include("grammars/scala");
|
||||
scala.file("grammars/scala/parser.c");
|
||||
scala.file("grammars/scala/scanner.c");
|
||||
scala.flag_if_supported("-Wno-unused-parameter");
|
||||
scala.flag_if_supported("-Wno-unused-but-set-variable");
|
||||
scala.flag_if_supported("-Wno-trigraphs");
|
||||
scala.flag_if_supported("-utf-8"); // msvc
|
||||
scala.compile("tree-sitter-scala");
|
||||
println!("cargo:rerun-if-changed=grammars/scala");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
The MIT License (MIT)
|
||||
|
||||
Copyright (c) 2018 Max Brunsfeld and GitHub
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,614 @@
|
||||
#include "tree_sitter/alloc.h"
|
||||
#include "tree_sitter/array.h"
|
||||
#include "tree_sitter/parser.h"
|
||||
|
||||
#include <wctype.h>
|
||||
|
||||
// #define DEBUG
|
||||
|
||||
#ifdef DEBUG
|
||||
#define LOG(...) fprintf(stderr, __VA_ARGS__)
|
||||
#else
|
||||
#define LOG(...)
|
||||
#endif
|
||||
|
||||
enum TokenType {
|
||||
AUTOMATIC_SEMICOLON,
|
||||
INDENT,
|
||||
OUTDENT,
|
||||
COMMA_OUTDENT,
|
||||
SIMPLE_STRING_START,
|
||||
SIMPLE_STRING_MIDDLE,
|
||||
SIMPLE_MULTILINE_STRING_START,
|
||||
INTERPOLATED_STRING_MIDDLE,
|
||||
INTERPOLATED_MULTILINE_STRING_MIDDLE,
|
||||
RAW_STRING_START,
|
||||
RAW_STRING_MIDDLE,
|
||||
RAW_STRING_MULTILINE_MIDDLE,
|
||||
SINGLE_LINE_STRING_END,
|
||||
MULTILINE_STRING_END,
|
||||
ELSE,
|
||||
CATCH,
|
||||
FINALLY,
|
||||
EXTENDS,
|
||||
DERIVES,
|
||||
WITH,
|
||||
ERROR_SENTINEL
|
||||
};
|
||||
|
||||
const char* token_name[] = {
|
||||
"AUTOMATIC_SEMICOLON",
|
||||
"INDENT",
|
||||
"OUTDENT",
|
||||
"COMMA_OUTDENT",
|
||||
"SIMPLE_STRING_START",
|
||||
"SIMPLE_STRING_MIDDLE",
|
||||
"SIMPLE_MULTILINE_STRING_START",
|
||||
"INTERPOLATED_STRING_MIDDLE",
|
||||
"INTERPOLATED_MULTILINE_STRING_MIDDLE",
|
||||
"RAW_STRING_MIDDLE",
|
||||
"RAW_STRING_MULTILINE_MIDDLE",
|
||||
"SINGLE_LINE_STRING_END",
|
||||
"MULTILINE_STRING_END",
|
||||
"ELSE",
|
||||
"CATCH",
|
||||
"FINALLY",
|
||||
"EXTENDS",
|
||||
"DERIVES",
|
||||
"WITH",
|
||||
"ERROR_SENTINEL"
|
||||
};
|
||||
|
||||
typedef struct {
|
||||
Array(int16_t) indents;
|
||||
int16_t last_indentation_size;
|
||||
int16_t last_newline_count;
|
||||
int16_t last_column;
|
||||
} Scanner;
|
||||
|
||||
void *tree_sitter_scala_external_scanner_create() {
|
||||
Scanner *scanner = ts_calloc(1, sizeof(Scanner));
|
||||
array_init(&scanner->indents);
|
||||
scanner->last_indentation_size = -1;
|
||||
scanner->last_column = -1;
|
||||
return scanner;
|
||||
}
|
||||
|
||||
void tree_sitter_scala_external_scanner_destroy(void *payload) {
|
||||
Scanner *scanner = payload;
|
||||
array_delete(&scanner->indents);
|
||||
ts_free(scanner);
|
||||
}
|
||||
|
||||
unsigned tree_sitter_scala_external_scanner_serialize(void *payload, char *buffer) {
|
||||
Scanner *scanner = (Scanner*)payload;
|
||||
|
||||
if ((scanner->indents.size + 3) * sizeof(int16_t) > TREE_SITTER_SERIALIZATION_BUFFER_SIZE) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t size = 0;
|
||||
memcpy(buffer + size, &scanner->last_indentation_size, sizeof(int16_t));
|
||||
size += sizeof(int16_t);
|
||||
memcpy(buffer + size, &scanner->last_newline_count, sizeof(int16_t));
|
||||
size += sizeof(int16_t);
|
||||
memcpy(buffer + size, &scanner->last_column, sizeof(int16_t));
|
||||
size += sizeof(int16_t);
|
||||
|
||||
for (unsigned i = 0; i < scanner->indents.size; i++) {
|
||||
memcpy(buffer + size, &scanner->indents.contents[i], sizeof(int16_t));
|
||||
size += sizeof(int16_t);
|
||||
}
|
||||
|
||||
return size;
|
||||
}
|
||||
|
||||
void tree_sitter_scala_external_scanner_deserialize(void *payload, const char *buffer,
|
||||
unsigned length) {
|
||||
Scanner *scanner = (Scanner*)payload;
|
||||
array_clear(&scanner->indents);
|
||||
scanner->last_indentation_size = -1;
|
||||
scanner->last_column = -1;
|
||||
scanner->last_newline_count = 0;
|
||||
|
||||
if (length == 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
size_t size = 0;
|
||||
|
||||
scanner->last_indentation_size = *(int16_t *)&buffer[size];
|
||||
size += sizeof(int16_t);
|
||||
scanner->last_newline_count = *(int16_t *)&buffer[size];
|
||||
size += sizeof(int16_t);
|
||||
scanner->last_column = *(int16_t *)&buffer[size];
|
||||
size += sizeof(int16_t);
|
||||
|
||||
while (size < length) {
|
||||
array_push(&scanner->indents, *(int16_t *)&buffer[size]);
|
||||
size += sizeof(int16_t);
|
||||
}
|
||||
|
||||
assert(size == length);
|
||||
}
|
||||
|
||||
static inline void advance(TSLexer *lexer) { lexer->advance(lexer, false); }
|
||||
|
||||
static inline void skip(TSLexer *lexer) { lexer->advance(lexer, true); }
|
||||
|
||||
// Used to detect leading infix operators on continuation lines.
|
||||
// See: https://www.scala-lang.org/api/3.x/docs/changed-features/operators.html
|
||||
static bool is_op_char(int32_t c) {
|
||||
switch (c) {
|
||||
case '!': case '#': case '%': case '&':
|
||||
case '*': case '+': case '-': case '<':
|
||||
case '=': case '>': case '?': case '@':
|
||||
case '\\': case '^': case '|': case '~':
|
||||
case ':':
|
||||
return true;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// We enumerate 3 types of strings that we need to handle differently:
|
||||
// 1. Simple strings, `"..."` or `"""..."""`
|
||||
// 2. Interpolated strings, `s"..."` or `f"..."` or `foo"..."` or foo"""...""".
|
||||
// 3. Raw strings, `raw"..."`
|
||||
typedef enum {
|
||||
STRING_MODE_SIMPLE,
|
||||
STRING_MODE_INTERPOLATED,
|
||||
STRING_MODE_RAW
|
||||
} StringMode;
|
||||
|
||||
static bool scan_string_content(TSLexer *lexer, bool is_multiline, StringMode string_mode) {
|
||||
LOG("scan_string_content(%d, %d, %c)\n", is_multiline, string_mode, lexer->lookahead);
|
||||
unsigned closing_quote_count = 0;
|
||||
for (;;) {
|
||||
if (lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
closing_quote_count++;
|
||||
if (!is_multiline) {
|
||||
lexer->result_symbol = SINGLE_LINE_STRING_END;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
if (closing_quote_count >= 3 && lexer->lookahead != '"') {
|
||||
lexer->result_symbol = MULTILINE_STRING_END;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
} else if (lexer->lookahead == '$' && string_mode != STRING_MODE_SIMPLE) {
|
||||
switch (string_mode) {
|
||||
case STRING_MODE_INTERPOLATED:
|
||||
lexer->result_symbol = is_multiline ? INTERPOLATED_MULTILINE_STRING_MIDDLE : INTERPOLATED_STRING_MIDDLE;
|
||||
break;
|
||||
case STRING_MODE_RAW:
|
||||
lexer->result_symbol = is_multiline ? RAW_STRING_MULTILINE_MIDDLE : RAW_STRING_MIDDLE;
|
||||
break;
|
||||
default:
|
||||
assert(false);
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
} else {
|
||||
closing_quote_count = 0;
|
||||
if (lexer->lookahead == '\\') {
|
||||
// Multiline strings ignore escape sequences
|
||||
if (is_multiline || string_mode == STRING_MODE_RAW) {
|
||||
// FIXME: In raw string mode, we have to jump over escaped quotes.
|
||||
advance(lexer);
|
||||
// In single-line raw strings, `\"` is not translated to `"`, but it also does
|
||||
// not close the string. Likewise, `\\` is not translated to `\`, but it does
|
||||
// stop the second `\` from stopping a double-quote from closing the string.
|
||||
if (!is_multiline && string_mode == STRING_MODE_RAW &&
|
||||
(lexer->lookahead == '"' || lexer->lookahead == '\\')) {
|
||||
advance(lexer);
|
||||
}
|
||||
} else {
|
||||
lexer->result_symbol = string_mode == STRING_MODE_SIMPLE ? SIMPLE_STRING_MIDDLE : INTERPOLATED_STRING_MIDDLE;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
// During error recovery and dynamic precedence resolution, the external
|
||||
// scanner will be invoked with all valid_symbols set to true, which means
|
||||
// we will be asked to scan a string token when we are not actually in a
|
||||
// string context. Here we detect these cases and return false.
|
||||
} else if (lexer->lookahead == '\n' && !is_multiline) {
|
||||
return false;
|
||||
} else if (lexer->eof(lexer)) {
|
||||
return false;
|
||||
} else {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static bool detect_comment_start(TSLexer *lexer) {
|
||||
lexer->mark_end(lexer);
|
||||
// Comments should not affect indentation
|
||||
if (lexer->lookahead == '/') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '/' || lexer -> lookahead == '*') {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
static bool scan_word(TSLexer *lexer, const char* const word) {
|
||||
for (uint8_t i = 0; word[i] != '\0'; i++) {
|
||||
if (lexer->lookahead != word[i]) {
|
||||
return false;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
return !iswalnum(lexer->lookahead);
|
||||
}
|
||||
|
||||
// Returns true if the lookahead starts a leading infix operator — a symbolic
|
||||
// operator or back-ticked identifier followed by whitespace and then a
|
||||
// non-whitespace operand on the same line. Such a line is a continuation of
|
||||
// the previous expression, so neither AUTOMATIC_SEMICOLON nor OUTDENT should
|
||||
// fire ahead of it. Advances the lexer; the caller must not rely on position.
|
||||
static bool is_leading_infix_continuation(TSLexer *lexer) {
|
||||
if (is_op_char(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
while (is_op_char(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
bool found_space = false;
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') {
|
||||
advance(lexer);
|
||||
found_space = true;
|
||||
}
|
||||
return found_space && !iswspace(lexer->lookahead) && !lexer->eof(lexer);
|
||||
}
|
||||
if (lexer->lookahead == '`') {
|
||||
advance(lexer);
|
||||
while (lexer->lookahead != '`' && !lexer->eof(lexer)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != '`') {
|
||||
return false;
|
||||
}
|
||||
advance(lexer);
|
||||
bool found_space = false;
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') {
|
||||
advance(lexer);
|
||||
found_space = true;
|
||||
}
|
||||
return found_space && !iswspace(lexer->lookahead) && !lexer->eof(lexer);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
static inline void debug_indents(Scanner *scanner) {
|
||||
LOG(" indents(%d): ", scanner->indents.size);
|
||||
for (unsigned i = 0; i < scanner->indents.size; i++) {
|
||||
LOG("%d ", scanner->indents.contents[i]);
|
||||
}
|
||||
LOG("\n");
|
||||
}
|
||||
|
||||
bool tree_sitter_scala_external_scanner_scan(void *payload, TSLexer *lexer,
|
||||
const bool *valid_symbols) {
|
||||
#ifdef DEBUG
|
||||
{
|
||||
if (valid_symbols[ERROR_SENTINEL]) {
|
||||
LOG("entering tree_sitter_scala_external_scanner_scan. ERROR_SENTINEL is valid\n");
|
||||
} else {
|
||||
char debug_str[1024] = "entering tree_sitter_scala_external_scanner_scan valid symbols: ";
|
||||
for (unsigned i = 0; i < ERROR_SENTINEL; i++) {
|
||||
if (valid_symbols[i]) {
|
||||
strcat(debug_str, token_name[i]);
|
||||
strcat(debug_str, ", ");
|
||||
}
|
||||
}
|
||||
strcat(debug_str, "\n");
|
||||
LOG("%s", debug_str);
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
Scanner *scanner = (Scanner *)payload;
|
||||
int16_t prev = scanner->indents.size > 0 ? *array_back(&scanner->indents) : -1;
|
||||
int16_t newline_count = 0;
|
||||
int16_t indentation_size = 0;
|
||||
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
if (lexer->lookahead == '\n') {
|
||||
newline_count++;
|
||||
indentation_size = 0;
|
||||
}
|
||||
else {
|
||||
indentation_size++;
|
||||
}
|
||||
skip(lexer);
|
||||
}
|
||||
|
||||
// Separate from OUTDENT because the scanner cannot distinguish a comma that
|
||||
// terminates an indented block (e.g. `map: x => f(x),`) from one that is
|
||||
// internal to it (e.g. `case EnumCase1, EnumCase2`). By using a distinct
|
||||
// token, tree-sitter only makes it valid in grammar contexts where comma
|
||||
// termination is expected (colon_argument, _indentable_expression).
|
||||
if (valid_symbols[COMMA_OUTDENT] && lexer->lookahead == ',' && prev != -1) {
|
||||
if (scanner->indents.size > 0) {
|
||||
array_pop(&scanner->indents);
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = COMMA_OUTDENT;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Before advancing the lexer, check if we can double outdent
|
||||
if (
|
||||
valid_symbols[OUTDENT] &&
|
||||
(
|
||||
lexer->lookahead == 0 ||
|
||||
(
|
||||
prev != -1 &&
|
||||
(
|
||||
lexer->lookahead == ')' ||
|
||||
lexer->lookahead == ']' ||
|
||||
lexer->lookahead == '}'
|
||||
)
|
||||
) ||
|
||||
(
|
||||
scanner->last_indentation_size != -1 &&
|
||||
prev != -1 &&
|
||||
scanner->last_indentation_size < prev
|
||||
)
|
||||
)
|
||||
) {
|
||||
if (scanner->indents.size > 0) {
|
||||
array_pop(&scanner->indents);
|
||||
}
|
||||
LOG(" pop\n");
|
||||
LOG(" OUTDENT\n");
|
||||
lexer->result_symbol = OUTDENT;
|
||||
return true;
|
||||
}
|
||||
scanner->last_indentation_size = -1;
|
||||
|
||||
if (
|
||||
valid_symbols[INDENT] &&
|
||||
newline_count > 0 &&
|
||||
(
|
||||
scanner->indents.size == 0 ||
|
||||
indentation_size > *array_back(&scanner->indents)
|
||||
)
|
||||
) {
|
||||
if (detect_comment_start(lexer)) {
|
||||
return false;
|
||||
}
|
||||
array_push(&scanner->indents, indentation_size);
|
||||
lexer->result_symbol = INDENT;
|
||||
LOG(" INDENT\n");
|
||||
return true;
|
||||
}
|
||||
|
||||
// This saves the indentation_size and newline_count so it can be used
|
||||
// in subsequent calls for multiple outdent or auto-semicolon.
|
||||
if (valid_symbols[OUTDENT] &&
|
||||
(lexer->lookahead == 0 ||
|
||||
(
|
||||
newline_count > 0 &&
|
||||
prev != -1 &&
|
||||
indentation_size < prev
|
||||
)
|
||||
)
|
||||
) {
|
||||
lexer->mark_end(lexer);
|
||||
if (detect_comment_start(lexer)) {
|
||||
return false;
|
||||
}
|
||||
scanner->last_indentation_size = indentation_size;
|
||||
scanner->last_newline_count = newline_count;
|
||||
if (lexer->eof(lexer)) {
|
||||
scanner->last_column = -1;
|
||||
} else {
|
||||
scanner->last_column = (int16_t)lexer->get_column(lexer);
|
||||
}
|
||||
// Don't close the indented block when the next line starts with a leading
|
||||
// infix operator: that operator continues the previous expression.
|
||||
if (lexer->lookahead != 0 && is_leading_infix_continuation(lexer)) {
|
||||
return false;
|
||||
}
|
||||
if (scanner->indents.size > 0) {
|
||||
array_pop(&scanner->indents);
|
||||
}
|
||||
LOG(" pop\n");
|
||||
LOG(" OUTDENT\n");
|
||||
lexer->result_symbol = OUTDENT;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Recover newline_count from the outdent reset
|
||||
bool is_eof = lexer->eof(lexer);
|
||||
if (
|
||||
(
|
||||
scanner->last_newline_count > 0 &&
|
||||
(is_eof && scanner->last_column == -1)
|
||||
) ||
|
||||
(!is_eof && lexer->get_column(lexer) == (uint32_t)scanner->last_column)
|
||||
) {
|
||||
newline_count += scanner->last_newline_count;
|
||||
}
|
||||
scanner->last_newline_count = 0;
|
||||
|
||||
if (valid_symbols[AUTOMATIC_SEMICOLON] && newline_count > 0) {
|
||||
// AUTOMATIC_SEMICOLON should not be issued in the middle of expressions
|
||||
// Thus, we exit this branch when encountering comments, else/catch clauses, etc.
|
||||
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = AUTOMATIC_SEMICOLON;
|
||||
|
||||
// Probably, a multi-line field expression, e.g.
|
||||
// a
|
||||
// .b
|
||||
// .c
|
||||
if (lexer->lookahead == '.') {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Single-line and multi-line comments
|
||||
if (lexer->lookahead == '/') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '/') {
|
||||
return false;
|
||||
}
|
||||
if (lexer->lookahead == '*') {
|
||||
advance(lexer);
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '*') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '/') {
|
||||
advance(lexer);
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
if (lexer->lookahead == '\n' || lexer->lookahead == '\r') {
|
||||
return false;
|
||||
}
|
||||
skip(lexer);
|
||||
}
|
||||
// If some code is present at the same line after comment end,
|
||||
// we should still produce AUTOMATIC_SEMICOLON, e.g. in
|
||||
// val a = 1
|
||||
// /* comment */ val b = 2
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[ELSE]) {
|
||||
return !scan_word(lexer, "else");
|
||||
}
|
||||
|
||||
if (valid_symbols[CATCH]) {
|
||||
if (scan_word(lexer, "catch")) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[FINALLY]) {
|
||||
if (scan_word(lexer, "finally")) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[EXTENDS]) {
|
||||
if (scan_word(lexer, "extends")) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[WITH]) {
|
||||
if (scan_word(lexer, "with")) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[DERIVES]) {
|
||||
if (scan_word(lexer, "derives")) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (newline_count > 1) {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Don't insert automatic semicolon before leading infix operators:
|
||||
// - symbolic, e.g. || or &&
|
||||
// - back-ticked, e.g. `in`
|
||||
// Only suppress if the operator is followed by horizontal whitespace
|
||||
// and then non-newline content on the same line, meaning it has an operand.
|
||||
if (is_leading_infix_continuation(lexer)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
if (lexer->lookahead == '\n') {
|
||||
newline_count++;
|
||||
}
|
||||
skip(lexer);
|
||||
}
|
||||
|
||||
if (valid_symbols[SIMPLE_STRING_START] && lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
if (lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
lexer->result_symbol = SIMPLE_MULTILINE_STRING_START;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
lexer->result_symbol = SIMPLE_STRING_START;
|
||||
return true;
|
||||
}
|
||||
|
||||
// We need two tokens of lookahead to determine if we are parsing a raw string,
|
||||
// the `raw` and the `"`, which is why we need to do it in the external scanner.
|
||||
if (valid_symbols[RAW_STRING_START] && lexer->lookahead == 'r') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == 'a') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == 'w') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '"') {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = RAW_STRING_START;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[SIMPLE_STRING_MIDDLE]) {
|
||||
return scan_string_content(lexer, false, STRING_MODE_SIMPLE);
|
||||
}
|
||||
|
||||
if (valid_symbols[INTERPOLATED_STRING_MIDDLE]) {
|
||||
return scan_string_content(lexer, false, STRING_MODE_INTERPOLATED);
|
||||
}
|
||||
|
||||
if (valid_symbols[RAW_STRING_MIDDLE]) {
|
||||
return scan_string_content(lexer, false, STRING_MODE_RAW);
|
||||
}
|
||||
|
||||
if (valid_symbols[RAW_STRING_MULTILINE_MIDDLE]) {
|
||||
return scan_string_content(lexer, true, STRING_MODE_RAW);
|
||||
}
|
||||
|
||||
if (valid_symbols[INTERPOLATED_MULTILINE_STRING_MIDDLE]) {
|
||||
return scan_string_content(lexer, true, STRING_MODE_INTERPOLATED);
|
||||
}
|
||||
|
||||
// We still need to handle the simple multiline string case, but there is
|
||||
// no `MULTILINE_STRING_MIDDLE` token, and `MULTILINE_STRING_END` is used
|
||||
// by all three of simple raw, and interpolated multiline strings. So this
|
||||
// check needs to come after the `INTERPOLATED_MULTILINE_STRING_MIDDLE` and
|
||||
// `RAW_STRING_MULTILINE_MIDDLE` check, so that we can be sure we are in a
|
||||
// simple multiline string context.
|
||||
if (valid_symbols[MULTILINE_STRING_END]) {
|
||||
return scan_string_content(lexer, true, STRING_MODE_SIMPLE);
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
//
|
||||
@@ -0,0 +1,54 @@
|
||||
#ifndef TREE_SITTER_ALLOC_H_
|
||||
#define TREE_SITTER_ALLOC_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
// Allow clients to override allocation functions
|
||||
#ifdef TREE_SITTER_REUSE_ALLOCATOR
|
||||
|
||||
extern void *(*ts_current_malloc)(size_t size);
|
||||
extern void *(*ts_current_calloc)(size_t count, size_t size);
|
||||
extern void *(*ts_current_realloc)(void *ptr, size_t size);
|
||||
extern void (*ts_current_free)(void *ptr);
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc ts_current_malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc ts_current_calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc ts_current_realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free ts_current_free
|
||||
#endif
|
||||
|
||||
#else
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free free
|
||||
#endif
|
||||
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ALLOC_H_
|
||||
@@ -0,0 +1,330 @@
|
||||
#ifndef TREE_SITTER_ARRAY_H_
|
||||
#define TREE_SITTER_ARRAY_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include "./alloc.h"
|
||||
|
||||
#include <assert.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(push)
|
||||
#pragma warning(disable : 4101)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-variable"
|
||||
#endif
|
||||
|
||||
#define Array(T) \
|
||||
struct { \
|
||||
T *contents; \
|
||||
uint32_t size; \
|
||||
uint32_t capacity; \
|
||||
}
|
||||
|
||||
/// Initialize an array.
|
||||
#define array_init(self) \
|
||||
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
|
||||
|
||||
/// Create an empty array.
|
||||
#define array_new() \
|
||||
{ NULL, 0, 0 }
|
||||
|
||||
/// Get a pointer to the element at a given `index` in the array.
|
||||
#define array_get(self, _index) \
|
||||
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
|
||||
|
||||
/// Get a pointer to the first element in the array.
|
||||
#define array_front(self) array_get(self, 0)
|
||||
|
||||
/// Get a pointer to the last element in the array.
|
||||
#define array_back(self) array_get(self, (self)->size - 1)
|
||||
|
||||
/// Clear the array, setting its size to zero. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_clear(self) ((self)->size = 0)
|
||||
|
||||
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
|
||||
/// less than the array's current capacity, this function has no effect.
|
||||
#define array_reserve(self, new_capacity) \
|
||||
((self)->contents = _array__reserve( \
|
||||
(void *)(self)->contents, &(self)->capacity, \
|
||||
array_elem_size(self), new_capacity) \
|
||||
)
|
||||
|
||||
/// Free any memory allocated for this array. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_delete(self) \
|
||||
do { \
|
||||
if ((self)->contents) ts_free((self)->contents); \
|
||||
(self)->contents = NULL; \
|
||||
(self)->size = 0; \
|
||||
(self)->capacity = 0; \
|
||||
} while (0)
|
||||
|
||||
/// Push a new `element` onto the end of the array.
|
||||
#define array_push(self, element) \
|
||||
do { \
|
||||
(self)->contents = _array__grow( \
|
||||
(void *)(self)->contents, (self)->size, &(self)->capacity, \
|
||||
1, array_elem_size(self) \
|
||||
); \
|
||||
(self)->contents[(self)->size++] = (element); \
|
||||
} while(0)
|
||||
|
||||
/// Increase the array's size by `count` elements.
|
||||
/// New elements are zero-initialized.
|
||||
#define array_grow_by(self, count) \
|
||||
do { \
|
||||
if ((count) == 0) break; \
|
||||
(self)->contents = _array__grow( \
|
||||
(self)->contents, (self)->size, &(self)->capacity, \
|
||||
count, array_elem_size(self) \
|
||||
); \
|
||||
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
|
||||
(self)->size += (count); \
|
||||
} while (0)
|
||||
|
||||
/// Append all elements from one array to the end of another.
|
||||
#define array_push_all(self, other) \
|
||||
array_extend((self), (other)->size, (other)->contents)
|
||||
|
||||
/// Append `count` elements to the end of the array, reading their values from the
|
||||
/// `contents` pointer.
|
||||
#define array_extend(self, count, other_contents) \
|
||||
(self)->contents = _array__splice( \
|
||||
(void*)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), (self)->size, 0, count, other_contents \
|
||||
)
|
||||
|
||||
/// Remove `old_count` elements from the array starting at the given `index`. At
|
||||
/// the same index, insert `new_count` new elements, reading their values from the
|
||||
/// `new_contents` pointer.
|
||||
#define array_splice(self, _index, old_count, new_count, new_contents) \
|
||||
(self)->contents = _array__splice( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), _index, old_count, new_count, new_contents \
|
||||
)
|
||||
|
||||
/// Insert one `element` into the array at the given `index`.
|
||||
#define array_insert(self, _index, element) \
|
||||
(self)->contents = _array__splice( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), _index, 0, 1, &(element) \
|
||||
)
|
||||
|
||||
/// Remove one element from the array at the given `index`.
|
||||
#define array_erase(self, _index) \
|
||||
_array__erase((void *)(self)->contents, &(self)->size, array_elem_size(self), _index)
|
||||
|
||||
/// Pop the last element off the array, returning the element by value.
|
||||
#define array_pop(self) ((self)->contents[--(self)->size])
|
||||
|
||||
/// Assign the contents of one array to another, reallocating if necessary.
|
||||
#define array_assign(self, other) \
|
||||
(self)->contents = _array__assign( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
(const void *)(other)->contents, (other)->size, array_elem_size(self) \
|
||||
)
|
||||
|
||||
/// Swap one array with another
|
||||
#define array_swap(self, other) \
|
||||
do { \
|
||||
void *_array_swap_tmp = (void *)(self)->contents; \
|
||||
(self)->contents = (other)->contents; \
|
||||
(other)->contents = _array_swap_tmp; \
|
||||
_array__swap(&(self)->size, &(self)->capacity, \
|
||||
&(other)->size, &(other)->capacity); \
|
||||
} while (0)
|
||||
|
||||
/// Get the size of the array contents
|
||||
#define array_elem_size(self) (sizeof *(self)->contents)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
///
|
||||
/// If an existing element is found to be equal to `needle`, then the `index`
|
||||
/// out-parameter is set to the existing value's index, and the `exists`
|
||||
/// out-parameter is set to true. Otherwise, `index` is set to an index where
|
||||
/// `needle` should be inserted in order to preserve the sorting, and `exists`
|
||||
/// is set to false.
|
||||
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using integer comparisons
|
||||
/// of a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_with`.
|
||||
#define array_search_sorted_by(self, field, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
#define array_insert_sorted_with(self, compare, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using integer comparisons of
|
||||
/// a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_by`.
|
||||
#define array_insert_sorted_by(self, field, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
// Private
|
||||
|
||||
// Pointers to individual `Array` fields (rather than the entire `Array` itself)
|
||||
// are passed to the various `_array__*` functions below to address strict aliasing
|
||||
// violations that arises when the _entire_ `Array` struct is passed as `Array(void)*`.
|
||||
//
|
||||
// The `Array` type itself was not altered as a solution in order to avoid breakage
|
||||
// with existing consumers (in particular, parsers with external scanners).
|
||||
|
||||
/// This is not what you're looking for, see `array_erase`.
|
||||
static inline void _array__erase(void* self_contents, uint32_t *size,
|
||||
size_t element_size, uint32_t index) {
|
||||
assert(index < *size);
|
||||
char *contents = (char *)self_contents;
|
||||
memmove(contents + index * element_size, contents + (index + 1) * element_size,
|
||||
(*size - index - 1) * element_size);
|
||||
(*size)--;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_reserve`.
|
||||
static inline void *_array__reserve(void *contents, uint32_t *capacity,
|
||||
size_t element_size, uint32_t new_capacity) {
|
||||
void *new_contents = contents;
|
||||
if (new_capacity > *capacity) {
|
||||
if (contents) {
|
||||
new_contents = ts_realloc(contents, new_capacity * element_size);
|
||||
} else {
|
||||
new_contents = ts_malloc(new_capacity * element_size);
|
||||
}
|
||||
*capacity = new_capacity;
|
||||
}
|
||||
return new_contents;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_assign`.
|
||||
static inline void *_array__assign(void* self_contents, uint32_t *self_size, uint32_t *self_capacity,
|
||||
const void *other_contents, uint32_t other_size, size_t element_size) {
|
||||
void *new_contents = _array__reserve(self_contents, self_capacity, element_size, other_size);
|
||||
*self_size = other_size;
|
||||
memcpy(new_contents, other_contents, *self_size * element_size);
|
||||
return new_contents;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_swap`.
|
||||
static inline void _array__swap(uint32_t *self_size, uint32_t *self_capacity,
|
||||
uint32_t *other_size, uint32_t *other_capacity) {
|
||||
uint32_t tmp_size = *self_size;
|
||||
uint32_t tmp_capacity = *self_capacity;
|
||||
*self_size = *other_size;
|
||||
*self_capacity = *other_capacity;
|
||||
*other_size = tmp_size;
|
||||
*other_capacity = tmp_capacity;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
|
||||
static inline void *_array__grow(void *contents, uint32_t size, uint32_t *capacity,
|
||||
uint32_t count, size_t element_size) {
|
||||
void *new_contents = contents;
|
||||
uint32_t new_size = size + count;
|
||||
if (new_size > *capacity) {
|
||||
uint32_t new_capacity = *capacity * 2;
|
||||
if (new_capacity < 8) new_capacity = 8;
|
||||
if (new_capacity < new_size) new_capacity = new_size;
|
||||
new_contents = _array__reserve(contents, capacity, element_size, new_capacity);
|
||||
}
|
||||
return new_contents;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_splice`.
|
||||
static inline void *_array__splice(void *self_contents, uint32_t *size, uint32_t *capacity,
|
||||
size_t element_size,
|
||||
uint32_t index, uint32_t old_count,
|
||||
uint32_t new_count, const void *elements) {
|
||||
uint32_t new_size = *size + new_count - old_count;
|
||||
uint32_t old_end = index + old_count;
|
||||
uint32_t new_end = index + new_count;
|
||||
assert(old_end <= *size);
|
||||
|
||||
void *new_contents = _array__reserve(self_contents, capacity, element_size, new_size);
|
||||
|
||||
char *contents = (char *)new_contents;
|
||||
if (*size > old_end) {
|
||||
memmove(
|
||||
contents + new_end * element_size,
|
||||
contents + old_end * element_size,
|
||||
(*size - old_end) * element_size
|
||||
);
|
||||
}
|
||||
if (new_count > 0) {
|
||||
if (elements) {
|
||||
memcpy(
|
||||
(contents + index * element_size),
|
||||
elements,
|
||||
new_count * element_size
|
||||
);
|
||||
} else {
|
||||
memset(
|
||||
(contents + index * element_size),
|
||||
0,
|
||||
new_count * element_size
|
||||
);
|
||||
}
|
||||
}
|
||||
*size += new_count - old_count;
|
||||
|
||||
return new_contents;
|
||||
}
|
||||
|
||||
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
|
||||
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
|
||||
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
|
||||
do { \
|
||||
*(_index) = start; \
|
||||
*(_exists) = false; \
|
||||
uint32_t size = (self)->size - *(_index); \
|
||||
if (size == 0) break; \
|
||||
int comparison; \
|
||||
while (size > 1) { \
|
||||
uint32_t half_size = size / 2; \
|
||||
uint32_t mid_index = *(_index) + half_size; \
|
||||
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
|
||||
if (comparison <= 0) *(_index) = mid_index; \
|
||||
size -= half_size; \
|
||||
} \
|
||||
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
|
||||
if (comparison == 0) *(_exists) = true; \
|
||||
else if (comparison < 0) *(_index) += 1; \
|
||||
} while (0)
|
||||
|
||||
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
|
||||
/// parameter by reference in order to work with the generic sorting function above.
|
||||
#define _compare_int(a, b) ((int)*(a) - (int)(b))
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(pop)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ARRAY_H_
|
||||
@@ -0,0 +1,286 @@
|
||||
#ifndef TREE_SITTER_PARSER_H_
|
||||
#define TREE_SITTER_PARSER_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
#define ts_builtin_sym_error ((TSSymbol)-1)
|
||||
#define ts_builtin_sym_end 0
|
||||
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
|
||||
|
||||
#ifndef TREE_SITTER_API_H_
|
||||
typedef uint16_t TSStateId;
|
||||
typedef uint16_t TSSymbol;
|
||||
typedef uint16_t TSFieldId;
|
||||
typedef struct TSLanguage TSLanguage;
|
||||
typedef struct TSLanguageMetadata {
|
||||
uint8_t major_version;
|
||||
uint8_t minor_version;
|
||||
uint8_t patch_version;
|
||||
} TSLanguageMetadata;
|
||||
#endif
|
||||
|
||||
typedef struct {
|
||||
TSFieldId field_id;
|
||||
uint8_t child_index;
|
||||
bool inherited;
|
||||
} TSFieldMapEntry;
|
||||
|
||||
// Used to index the field and supertype maps.
|
||||
typedef struct {
|
||||
uint16_t index;
|
||||
uint16_t length;
|
||||
} TSMapSlice;
|
||||
|
||||
typedef struct {
|
||||
bool visible;
|
||||
bool named;
|
||||
bool supertype;
|
||||
} TSSymbolMetadata;
|
||||
|
||||
typedef struct TSLexer TSLexer;
|
||||
|
||||
struct TSLexer {
|
||||
int32_t lookahead;
|
||||
TSSymbol result_symbol;
|
||||
void (*advance)(TSLexer *, bool);
|
||||
void (*mark_end)(TSLexer *);
|
||||
uint32_t (*get_column)(TSLexer *);
|
||||
bool (*is_at_included_range_start)(const TSLexer *);
|
||||
bool (*eof)(const TSLexer *);
|
||||
void (*log)(const TSLexer *, const char *, ...);
|
||||
};
|
||||
|
||||
typedef enum {
|
||||
TSParseActionTypeShift,
|
||||
TSParseActionTypeReduce,
|
||||
TSParseActionTypeAccept,
|
||||
TSParseActionTypeRecover,
|
||||
} TSParseActionType;
|
||||
|
||||
typedef union {
|
||||
struct {
|
||||
uint8_t type;
|
||||
TSStateId state;
|
||||
bool extra;
|
||||
bool repetition;
|
||||
} shift;
|
||||
struct {
|
||||
uint8_t type;
|
||||
uint8_t child_count;
|
||||
TSSymbol symbol;
|
||||
int16_t dynamic_precedence;
|
||||
uint16_t production_id;
|
||||
} reduce;
|
||||
uint8_t type;
|
||||
} TSParseAction;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
} TSLexMode;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
uint16_t reserved_word_set_id;
|
||||
} TSLexerMode;
|
||||
|
||||
typedef union {
|
||||
TSParseAction action;
|
||||
struct {
|
||||
uint8_t count;
|
||||
bool reusable;
|
||||
} entry;
|
||||
} TSParseActionEntry;
|
||||
|
||||
typedef struct {
|
||||
int32_t start;
|
||||
int32_t end;
|
||||
} TSCharacterRange;
|
||||
|
||||
struct TSLanguage {
|
||||
uint32_t abi_version;
|
||||
uint32_t symbol_count;
|
||||
uint32_t alias_count;
|
||||
uint32_t token_count;
|
||||
uint32_t external_token_count;
|
||||
uint32_t state_count;
|
||||
uint32_t large_state_count;
|
||||
uint32_t production_id_count;
|
||||
uint32_t field_count;
|
||||
uint16_t max_alias_sequence_length;
|
||||
const uint16_t *parse_table;
|
||||
const uint16_t *small_parse_table;
|
||||
const uint32_t *small_parse_table_map;
|
||||
const TSParseActionEntry *parse_actions;
|
||||
const char * const *symbol_names;
|
||||
const char * const *field_names;
|
||||
const TSMapSlice *field_map_slices;
|
||||
const TSFieldMapEntry *field_map_entries;
|
||||
const TSSymbolMetadata *symbol_metadata;
|
||||
const TSSymbol *public_symbol_map;
|
||||
const uint16_t *alias_map;
|
||||
const TSSymbol *alias_sequences;
|
||||
const TSLexerMode *lex_modes;
|
||||
bool (*lex_fn)(TSLexer *, TSStateId);
|
||||
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
|
||||
TSSymbol keyword_capture_token;
|
||||
struct {
|
||||
const bool *states;
|
||||
const TSSymbol *symbol_map;
|
||||
void *(*create)(void);
|
||||
void (*destroy)(void *);
|
||||
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
|
||||
unsigned (*serialize)(void *, char *);
|
||||
void (*deserialize)(void *, const char *, unsigned);
|
||||
} external_scanner;
|
||||
const TSStateId *primary_state_ids;
|
||||
const char *name;
|
||||
const TSSymbol *reserved_words;
|
||||
uint16_t max_reserved_word_set_size;
|
||||
uint32_t supertype_count;
|
||||
const TSSymbol *supertype_symbols;
|
||||
const TSMapSlice *supertype_map_slices;
|
||||
const TSSymbol *supertype_map_entries;
|
||||
TSLanguageMetadata metadata;
|
||||
};
|
||||
|
||||
static inline bool set_contains(const TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
|
||||
uint32_t index = 0;
|
||||
uint32_t size = len - index;
|
||||
while (size > 1) {
|
||||
uint32_t half_size = size / 2;
|
||||
uint32_t mid_index = index + half_size;
|
||||
const TSCharacterRange *range = &ranges[mid_index];
|
||||
if (lookahead >= range->start && lookahead <= range->end) {
|
||||
return true;
|
||||
} else if (lookahead > range->end) {
|
||||
index = mid_index;
|
||||
}
|
||||
size -= half_size;
|
||||
}
|
||||
const TSCharacterRange *range = &ranges[index];
|
||||
return (lookahead >= range->start && lookahead <= range->end);
|
||||
}
|
||||
|
||||
/*
|
||||
* Lexer Macros
|
||||
*/
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#define UNUSED __pragma(warning(suppress : 4101))
|
||||
#else
|
||||
#define UNUSED __attribute__((unused))
|
||||
#endif
|
||||
|
||||
#define START_LEXER() \
|
||||
bool result = false; \
|
||||
bool skip = false; \
|
||||
UNUSED \
|
||||
bool eof = false; \
|
||||
int32_t lookahead; \
|
||||
goto start; \
|
||||
next_state: \
|
||||
lexer->advance(lexer, skip); \
|
||||
start: \
|
||||
skip = false; \
|
||||
lookahead = lexer->lookahead;
|
||||
|
||||
#define ADVANCE(state_value) \
|
||||
{ \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ADVANCE_MAP(...) \
|
||||
{ \
|
||||
static const uint16_t map[] = { __VA_ARGS__ }; \
|
||||
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
|
||||
if (map[i] == lookahead) { \
|
||||
state = map[i + 1]; \
|
||||
goto next_state; \
|
||||
} \
|
||||
} \
|
||||
}
|
||||
|
||||
#define SKIP(state_value) \
|
||||
{ \
|
||||
skip = true; \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ACCEPT_TOKEN(symbol_value) \
|
||||
result = true; \
|
||||
lexer->result_symbol = symbol_value; \
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
#define END_STATE() return result;
|
||||
|
||||
/*
|
||||
* Parse Table Macros
|
||||
*/
|
||||
|
||||
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
|
||||
|
||||
#define STATE(id) id
|
||||
|
||||
#define ACTIONS(id) id
|
||||
|
||||
#define SHIFT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value) \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_REPEAT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value), \
|
||||
.repetition = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_EXTRA() \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.extra = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define REDUCE(symbol_name, children, precedence, prod_id) \
|
||||
{{ \
|
||||
.reduce = { \
|
||||
.type = TSParseActionTypeReduce, \
|
||||
.symbol = symbol_name, \
|
||||
.child_count = children, \
|
||||
.dynamic_precedence = precedence, \
|
||||
.production_id = prod_id \
|
||||
}, \
|
||||
}}
|
||||
|
||||
#define RECOVER() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeRecover \
|
||||
}}
|
||||
|
||||
#define ACCEPT_INPUT() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeAccept \
|
||||
}}
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_PARSER_H_
|
||||
@@ -23,12 +23,17 @@ extern "C" {
|
||||
extern "C" {
|
||||
fn tree_sitter_lua() -> *const ();
|
||||
}
|
||||
// Vendored scala grammar (build.rs-compiled C — the wasm is master@0aca5d0a6f,
|
||||
// 30 states past the v0.26.0 crate; see grammars/scala and the scala checklist).
|
||||
extern "C" {
|
||||
fn tree_sitter_scala() -> *const ();
|
||||
}
|
||||
|
||||
/// Languages this kernel binary can extract (reported by contractInfo;
|
||||
/// TS-side routing policy decides what actually routes).
|
||||
pub const LANGUAGES: [&str; 18] = [
|
||||
pub const LANGUAGES: [&str; 19] = [
|
||||
"typescript", "tsx", "javascript", "jsx", "java", "python", "go", "c", "cpp", "rust",
|
||||
"csharp", "ruby", "php", "swift", "kotlin", "r", "lua", "luau",
|
||||
"csharp", "ruby", "php", "swift", "kotlin", "r", "lua", "luau", "scala",
|
||||
];
|
||||
|
||||
pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
@@ -73,6 +78,11 @@ pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
// R7b batch 4: crate =1.2.0, sha-identical to the v1.2.0 tag the
|
||||
// vendored wasm was built from (lua-luau checklist §Grammar prep).
|
||||
"luau" => Some(tree_sitter_luau::LANGUAGE.into()),
|
||||
// R7b batch 4: master@0aca5d0a6f vendored C compiled in build.rs (the
|
||||
// revision is not a release — crate 0.26.0 is 30 states behind).
|
||||
"scala" => {
|
||||
Some(unsafe { tree_sitter_language::LanguageFn::from_raw(tree_sitter_scala) }.into())
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -31,6 +31,7 @@ mod php;
|
||||
mod rlang;
|
||||
mod ruby;
|
||||
mod rustlang;
|
||||
mod scala;
|
||||
mod swift;
|
||||
mod textutil;
|
||||
mod python;
|
||||
@@ -227,6 +228,7 @@ pub fn extract_file(file_path: String, content: String, language: String) -> Res
|
||||
"kotlin" => kotlin::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"r" => rlang::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"lua" | "luau" => lua::extract(&file_path, &content, &language).map_err(Error::from_reason)?,
|
||||
"scala" => scala::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
_ => tsjs::extract(&file_path, &content, &language).map_err(Error::from_reason)?,
|
||||
};
|
||||
Ok(ExtractBuffers {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -610,7 +610,8 @@ parity before porting the language.
|
||||
| rust | `languages/rust.ts` | T1 | crates.io | **DONE (R7b #1, 2026-07-20)** — `rustlang.rs` walker; grammar bumped to v0.24.2 (crate + vendored wasm together). Parity 0-diff on ripgrep/tokio/rust-analyzer + dump byte-identical ×3; rust-analyzer's parser crates defer 18% (token-macro tables, both-arm parse errors — grammar-inherent). Quirk list: docs/design/rust-lang-kernel-port-checklist.md. | ☑ |
|
||||
| r | `languages/r.ts` | T1 | crates.io | **DONE (R7b batch 4 #1, 2026-07-20)** — `rlang.rs` walker; NO grammar change (crate pin `=1.2.0`: the crates.io tarball ships parser.c AND scanner.c sha-identical to the r-lib v1.2.0 tag the vendored wasm was built from — first true no-op grammar prep). The lightest-shared-surface, heaviest-hook port: r.ts works entirely through visitNode (every type list empty but `callTypes`), four shared machineries dead by language gates, walker = file node + hook transcription + generic extractCall. Parity 0-diff on AnomalyDetection/dplyr/ggplot2/shiny (838 files, deferrals 0/0/0/1 — the 1 = a moustache-template pseudo-R file, both-arm) + dump byte-identical ×3 (dplyr/ggplot2/shiny). kernel-parity.mjs gained lowercased-extension matching (`.R` is the dominant casing). Quirk list: docs/design/r-kernel-port-checklist.md. | ☑ |
|
||||
| lua, luau | `languages/lua.ts` + `luau.ts` (36-line extension) | T1 | lua: **vendored C** (v0.4.1 not on crates.io); luau: crates.io `=1.2.0` (tag≡crate sha-verified) | **DONE (R7b batch 4 #2, 2026-07-20)** — ONE walker (`lua.rs`, ccpp-style dialect flag); lua = the second vendored-grammar-C language (v0.4.1 tag artifacts, shas in the checklist); luau = plain crate pin. NO wasm change for either — grammar-parity rows replace the bump gate. Ports the require/visitNode-hook asymmetries (top-level imports vs body `calls "require"`), receiver-QN methods, the top-level initializer-visibility inversion, raw-text callee world (colon/bracket/glue chains, paren-conversion), LUA_SPEC fn-ref capture, LuaDoc `- `-keeping docstrings, and the lua↔luau isExported wire divergence. Parity 0-diff kong/lazy.nvim/lua-resty-core/lune/Fusion (1,734 clean files; deferrals 1/0/0/3/8 — every one matching the survey's both-arm predictions) + dump byte-identical ×4 (kong 157,650 dump lines). Quirk list: docs/design/lua-luau-kernel-port-checklist.md. | ☑ |
|
||||
| dart, scala | dedicated files | T1 | **vendored C** (revs not on crates.io — batch-4 probe) | Long-tail T1, surveys COMPLETE 2026-07-20 (checklists in docs/design/). dart = tree-sitter-wasms artifact byte-copy re-vendor + UserNobody14 d4d8f3e vendored-C (sibling-body double-walk is the port hazard); scala = master 0aca5d0a6f vendored-C (35MB parser.c; phantom-hasError dominant on scala-3). | ☐ |
|
||||
| scala | `languages/scala.ts` | T1 | **vendored C** (master@0aca5d0a6f — not a release; crate 0.26.0 is 30 states behind) | **DONE (R7b batch 4 #3, 2026-07-20)** — `scala.rs` walker; the third vendored-grammar-C language (35MB parser.c — the biggest grammar in the tree). NO wasm change (production has parsed with this exact revision since #91) — the grammar-parity row is the whole alignment proof. Ports the leak-through asymmetries (extension first-def call leak + braced-form invisibility via the `{`-token body field, anon `new T {…}` template_body member leaks, bodied-vs-bodiless class_parameters), first-segment import names, the val/var hook's enclosing-NODE-TYPE kinds, defs-as-methods with top-level function fallback, nested-def invisibility, curried/type-params-first signatures, the #750 capitalized re-encode, static-member WRITES, scaladoc retention, full value-refs (last-wins targets) + SCALA_SPEC fn-refs (varinit + postfix eta). Parity 0-diff os-lib/cats/scala3-compiler-src/scala3-library-src (1,935 clean files; deferrals 0/15/57/116 — every count matching the survey exactly; scala-3 PHANTOM hasError deferred on the flag) + dump byte-identical ×3 (scala3 whole-repo 950,889 dump lines). Quirk list: docs/design/scala-kernel-port-checklist.md. | ☑ |
|
||||
| dart | `languages/dart.ts` | T1 | **vendored C** (UserNobody14 d4d8f3e — reached via tree-sitter-wasms' unpinned github dep) | Long-tail T1, survey COMPLETE 2026-07-20 (docs/design/dart-kernel-port-checklist.md). Port = vendored-C + byte-copy the tree-sitter-wasms wasm artifact into src/extraction/wasm/ (kills the unpinned-dep hazard). The sibling-body DOUBLE-WALK (duplicate ids, ×3 refs) is the port hazard; sweeps --max-deferral 0.3. | ☐ |
|
||||
| kotlin | `languages/kotlin.ts` | T1½ | **vendored C** (crate unusable) | **DONE (R7b #6, 2026-07-20)** — `kotlin.rs` walker; the arc's FIRST vendored-grammar-C language: fwcd 0.3.8's sha-matched parser.c/scanner.c compile inside codegraph-kernel via build.rs + cc (the crates.io crate pins tree-sitter <0.23; tree-sitter-kotlin-ng is a different grammar). Behavior-neutral wasm re-vendor (dumps byte-identical old-vs-new ×3). Two walker firsts: extension-fn receiver QNs + owner-contains, and extractModifiers→decorators (KMP expect/actual — 412 synthesized edges identical both arms on kotlinx.coroutines). Parity 0-diff okio/okhttp/kotlinx.coroutines (1,861 clean files; deferral 4.7–8.5% both-arm incl. PHANTOM hasError files). Quirk list: docs/design/kotlin-kernel-port-checklist.md. | ☑ |
|
||||
| swift | shared + dedicated branch | T1½ | crates.io | **DONE (R7b #5, 2026-07-20)** — `swift.rs` walker incl. the #1020 dedicated property branch (Alamofire's 348 property nodes reproduced exactly on the kernel arm); grammar bumped to crate 0.7.3 (wasm built from the CRATE TARBALL's src — the tag ships an older ABI-14 generation; delta = error-set membership + 2 gate-found categories, all classified). Parity 0-diff Alamofire/vapor/swift-nio (720 clean files; deferral 9–27% both-arm structural — sweeps use --max-deferral 0.3). One walker fix found by the sweep: the shared `assignment` shadow-prune case is swift-live (declared-then-assigned `let X: T`). Quirk list: docs/design/swift-kernel-port-checklist.md. | ☑ |
|
||||
| c, cpp | `languages/c-cpp.ts` | **T2** | crates.io | **DONE (R7a, 2026-07-17)** — `ccpp/` walker; ALL pre-passes stayed TS-side via the route-point preParse hoist (+6 new blanks added during gating — see the checklist doc); content-based `.h` C-vs-C++ detection stays upstream at detectLanguage. Parity 0-diff + dump byte-identical on redis/git/fmt/protobuf/ALS. | ☑ |
|
||||
|
||||
@@ -0,0 +1,958 @@
|
||||
# Scala kernel port (R7b) — the bug-for-bug checklist
|
||||
|
||||
**Status: SURVEY COMPLETE (2026-07-20), port not started.** Survey basis: every
|
||||
TS-side branch a `.scala`/`.sc` file exercises, with file:line anchors as of
|
||||
**`45a53eb`** (HEAD at survey time, clean main; a6c62d7..45a53eb touched only
|
||||
grammars.ts/kernel/index.ts/wasm inside src/extraction — tree-sitter.ts,
|
||||
languages/scala.ts, function-ref.ts, and tree-sitter-helpers.ts are
|
||||
byte-unchanged, so kotlin/swift-checklist anchors into shared code remain
|
||||
valid). Every grammar-shape claim below was **probed against the
|
||||
production vendored wasm** (`dist/extraction/wasm/tree-sitter-scala.wasm`,
|
||||
sha256 `7945b13e…`, identical to `src/…`), and every extraction-behavior claim
|
||||
was **pinned against the real `dist/` extractor** (`extract-*.txt` ground-truth
|
||||
dumps) — not derived from code reading. childForFieldName truth tables were
|
||||
probed directly (`brace-field.out`, the FIELDS sections of `cst-snippets.txt`)
|
||||
because **this grammar attaches field names to ANONYMOUS tokens** — CST dump
|
||||
labels alone mislead (the swift lesson, §Extension). Probe scripts + dumps in
|
||||
the session scratchpad `svy-scala/` (§Probe artifacts). Read WITH
|
||||
`docs/design/rust-kernel-migration-plan.md` (§0a recipe, §2 boundary, §4
|
||||
tracker row "scala", §5 gates) and the format precedents
|
||||
(`kotlin-kernel-port-checklist.md` — the closest sibling: JVM family, hook
|
||||
property branch, re-encode gate — and `swift-kernel-port-checklist.md`).
|
||||
|
||||
**Blocking findings: none — but three eyes-open items.** (1) The wasm is
|
||||
**already vendored and needs NO bump**: `VENDORED_WASM_LANGS` has contained
|
||||
`scala` since 2026-05-07 (#91), and the vendored wasm is table-identical to
|
||||
tree-sitter/tree-sitter-scala **master@`0aca5d0a6f`** (verified twice: the
|
||||
batch-4 positional table compare, and this survey's clone-sha match). The port
|
||||
is vendored-grammar-C only (kotlin mechanism) — there is no behavior-delta
|
||||
gate to run. (2) **Error incidence is bimodal**: mainstream Scala-2-style
|
||||
repos are clean (os-lib 0.00%, cats 1.80%) but bleeding-edge Scala-3 code is
|
||||
not (scala3 compiler/src 9.88%, library/src **17.79%** — capture-checking `^`
|
||||
types), and **~40–60% of erroring files are PHANTOMS** (hasError=true, zero
|
||||
ERROR/missing nodes — trust the flag). Sweep guidance in §Architecture #6.
|
||||
(3) **scala has 32 real fields but three load-bearing places where
|
||||
`childForFieldName` returns the FIRST of SEVERAL same-named fields — or an
|
||||
anonymous `{` token** (import `path`, curried `parameters`, extension `body`).
|
||||
The walker must reproduce first-match-wins over the full (named+anonymous)
|
||||
child list, not "the" field (§Extractor config, §Extension).
|
||||
|
||||
## Grammar prep (NO wasm bump — vendored-C kernel build only)
|
||||
|
||||
- **Production wasm**: `src/extraction/wasm/tree-sitter-scala.wasm`, sha256
|
||||
`7945b13e6f9b15b578c5e5e4e60253c049fec07c531518163f3415a76c0621aa`
|
||||
(src == dist byte-identical), ABI **15**, STATE_COUNT 26650, 357+5 symbols,
|
||||
**FIELD_COUNT 32**. In `VENDORED_WASM_LANGS` (grammars.ts:292) since #91
|
||||
(2026-05-07). Mapping `scala: 'tree-sitter-scala.wasm'` (grammars.ts:38).
|
||||
- **Provenance (verified)**: tree-sitter/tree-sitter-scala
|
||||
**master@`0aca5d0a6fe115b16d55cb100e1bb05e7fb11385`** (2026-04-22 "chore:
|
||||
generate and sync from ec71cd9d51" — the post-v0.26.0 fix batch:
|
||||
scala2-compiler-100%, lambda-body restrict, wildcard self-type). The clone's
|
||||
`src/parser.c` sha256 equals the batch-4 probe's positional-table-verified
|
||||
copy (`bc3c3c79…`), and that probe found the wasm's kind/field tables
|
||||
positionally identical to it. The v0.26.0 tag == crate 0.26.0 is 26620
|
||||
states — **30 states BEHIND our wasm** — so a crate pin would be a silent
|
||||
downgrade; later master (`fc99b1bd`, Apr-27) is a 23959-state restructure —
|
||||
a future-bump candidate, NOT this port. Full record:
|
||||
`../scratchpad/batch4-grammar-probe.md`.
|
||||
- **Vendored-C route (kotlin mechanism, second use)** — copy from the
|
||||
`0aca5d0a6f` clone into `codegraph-kernel/grammars/scala/` (shas recorded in
|
||||
a comment; survey record `grammar-shas.txt`):
|
||||
- `src/parser.c` `bc3c3c794f19461d99d04de6c31d57fa3e41243509b9ab023a9b88ed3273d102` (34,970,232 bytes — 35 MB, the biggest grammar in the tree; expect a slow `cc` step)
|
||||
- `src/scanner.c` `e4ba242568ee3493015598997bf60f613802616eade62717c21109287ef64752` (17,731 bytes — a REAL external scanner: significant-indentation + interpolation; it handles `\r` explicitly, scanner.c:476)
|
||||
- `src/tree_sitter/alloc.h` `b29c1c9f…`, `array.h` `31e60a1b…`, `parser.h` `180b893c…`
|
||||
- build.rs: crib the upstream `bindings/rust/build.rs` flag set — `cc::Build`
|
||||
`.std("c11").include(grammars/scala).flag_if_supported("-Wno-unused")`,
|
||||
msvc `-utf-8`; compile parser.c + scanner.c; symbol `tree_sitter_scala`
|
||||
(parser.c:1199034). Same shape as the kotlin block (build.rs:14-22).
|
||||
- Cargo: `tree-sitter-language` shim already present (kotlin). langs.rs:
|
||||
`extern "C" { fn tree_sitter_scala() -> *const (); }` +
|
||||
`"scala" => LanguageFn::from_raw(...)` + `LANGUAGES` 15 → **16**.
|
||||
- `__tests__/kernel-grammar-parity.test.ts:39` `GRAMMAR_LANGUAGES += 'scala'`
|
||||
— the id-by-id table compare against the vendored wasm proves the C build
|
||||
is the same revision (ABI 15 / 26650 states / 32 fields must match).
|
||||
- License: MIT (tree-sitter org). No metadata shim needed (repo has
|
||||
tree-sitter.json-era layout; we never run `tree-sitter generate`).
|
||||
- **NO grammar-bump gate.** Unlike every other R7b language there is no
|
||||
old-vs-new wasm diff to run — production already parses with this exact
|
||||
revision. The kernel-grammar-parity row IS the alignment proof.
|
||||
- **Scanner-state parity risks — probed clean:** the indentation scanner
|
||||
handles CRLF: LF-vs-CRLF parses are **s-expression-identical on every
|
||||
fixture including Scala-3 indentation syntax** (`crlf-cst.cjs`: indent/
|
||||
torture/docs/vref/misc/ext/script all `sexpEqual=true`, no error flips).
|
||||
Extraction under CRLF is byte-identical except the multi-line-block-comment
|
||||
docstring `\r` retention (§Docstrings — a shared `js_multiline_strip`
|
||||
concern, not a scanner one). The kernel parses UTF-8 while wasm parses
|
||||
UTF-16 — error-recovery differences are exactly what per-file
|
||||
`has_error() → defer:` guards; nothing scala-specific beyond the elevated
|
||||
incidence below.
|
||||
|
||||
### Error incidence (production wasm, all `.scala`/`.sc` ≤1 MiB, `error-sweep.cjs`)
|
||||
|
||||
| Repo | files | hasError | rate | of which PHANTOM |
|
||||
|---|---|---|---|---|
|
||||
| os-lib (small) | 59 | 0 | 0.00% | — |
|
||||
| cats (medium) | 835 | 15 | 1.80% | 4 (27%) |
|
||||
| scala3 compiler/src | 577 | 57 | 9.88% | — |
|
||||
| scala3 library/src | 652 | 116 | **17.79%** | **69 (59%)** |
|
||||
| scala3 whole repo | 18,411 | 1,991 | 10.81% | (incl. `tests/` = deliberately-invalid neg fixtures, 11.78%) |
|
||||
|
||||
`.sc` files: 14 in scala3 (1 error); os-lib/cats have none. Error classes
|
||||
(sampled `err-samples.out`, minimized `errvariants.out` / `phantom-min.out`):
|
||||
|
||||
- **(a) PHANTOM hasError — the dominant scala-3 class.** hasError=true with a
|
||||
COMPLETE, correct CST and ZERO ERROR/missing nodes. Minimal repro:
|
||||
capture-checking postfix `^` — `def f(x: List[Int]^): Int = 1` (the whole
|
||||
scala3 library uses `import language.experimental.captureChecking`).
|
||||
cats' scala-2 macro files (`FreeFoldStep.scala`) phantom too. **The kernel
|
||||
must defer on the FLAG, never on ERROR-node presence** (kotlin lesson,
|
||||
worse here).
|
||||
- **(b) `end` used as an identifier** — `end` is hard-reserved by the grammar
|
||||
(`end match`, `val end = 1` both ERROR). cats `AndThen.scala`.
|
||||
- **(c) generic + curried super-constructor args** —
|
||||
`extends Eq[A]()(ev)` ERRORS (plain `Base(1)(2)` parses clean —
|
||||
`extract-super2.txt`). cats kernel instances.
|
||||
- **(d) Unicode symbolic type names** — `type ⊥ = Nothing` (cats
|
||||
`package.scala`).
|
||||
- **(e) scala 3.0–3.3 `given … with { }` syntax** — `given x: C with { … }`
|
||||
ERRORS (the `= new C {}` and colon forms parse fine).
|
||||
- **(f) assorted dotty-frontier syntax** (union-of-singleton `.type` unions,
|
||||
compact `catch case`, braceless `match` at margin) — scala3 compiler files.
|
||||
|
||||
All classes are grammar-inherent and identical across arms by construction
|
||||
(same grammar revision compiled twice). Deferral guidance: §Architecture #6.
|
||||
|
||||
## Architecture decisions
|
||||
|
||||
1. **No preParse, no POST_PASSES.** `scalaExtractor` has no `preParse`
|
||||
(languages/scala.ts — whole file) → the kernel/index.ts preParse hoist is a
|
||||
no-op; no `POST_PASSES` entry → `tryKernelExtractRaw` stays eligible.
|
||||
2. **One framework resolver can force the DECODED path for scala:
|
||||
`playResolver`** (resolution/frameworks/play.ts:30, `languages: ['scala',
|
||||
'java', 'yaml']`, and it HAS `extract()`), via parse-worker.ts's
|
||||
frameworksNeedDecode check (parse-worker.ts:95-100). detect() (play.ts:32):
|
||||
`build.sbt` matching `/playframework|"play"|sbt-plugin|PlayScala|PlayJava/i`,
|
||||
OR `conf/routes` exists, OR `conf/application.conf` exists. **None of the
|
||||
three gate repos trips it** (no `conf/`, no playframework in build.sbt) —
|
||||
they exercise the raw buffers-to-store transport; a Play app (or any repo
|
||||
with a root `conf/application.conf` — akka-style apps can!) is the
|
||||
decoded-path smoke check. The Play extract() itself only produces output
|
||||
for `conf/routes`/`*.routes` files (isPlayRoutesFile, grammars.ts:222-228)
|
||||
which are NOT scala files (extensionless → no-grammar path) — the cost of
|
||||
detection is only the decode, not wrong output.
|
||||
3. **One walker module** (`codegraph-kernel/src/scala.rs`), registered in
|
||||
langs.rs; per-file `has_error()` → `defer:`. **kotlin.rs is the closest
|
||||
crib** (visitNode-hook property branch, classify-by-node-type, re-encode
|
||||
gate, JVM import shapes) but scala diverges in TEN places, each detailed
|
||||
below: (a) **no namespace node ever** (no packageTypes — package headers
|
||||
are ignored; QNs are bare); (b) functionTypes EMPTY → every def routes
|
||||
through the methodTypes branch (extractMethod → top-level fallback to
|
||||
extractFunction); (c) the val/var hook keys on the **enclosing-definition
|
||||
NODE TYPE walk**, not the stack; (d) getSignature is LIVE (fields exist)
|
||||
with the curried/type-params first-field quirk; (e) extension/given/
|
||||
package_object have NO ladder branch — their leak-through behaviors are
|
||||
the port's hardest part; (f) `instance_expression` ∈ INSTANTIATION_KINDS +
|
||||
scalaBaseTypeName; (g) the scala extends branch iterates ALL supertypes;
|
||||
(h) scala-only type-annotation walks (every `parameters` + type_parameters
|
||||
context bounds) plus the hook's own emitScalaTypeRefs; (i) fn-ref spec
|
||||
with bare-identifier idTypes + postfix eta unwrap; (j) imports named by
|
||||
the FIRST path segment. No c/cpp-style dialect, no content sniffing:
|
||||
`.scala` and `.sc` both → `scala` (grammars.ts:120-121; `.sbt` is NOT
|
||||
mapped — build.sbt files are never indexed).
|
||||
4. **`.sc` files are ordinary scala files** whose top-level statements
|
||||
attribute calls to the FILE node (pinned `extract-script.txt`: `calls
|
||||
println/runTop from=file`, top-level `val` → constant with file parent).
|
||||
Same for `.scala` files with top-level statements (grammar accepts them).
|
||||
5. **REF_FLAG_FILE_PATH (wire v2 slot) is NOT needed.** No scala path emits
|
||||
refs carrying `filePath` (hook refs via emitScalaTypeRefs carry
|
||||
fromNodeId/name/kind/line/column only; extractImport sets no handledRefs;
|
||||
verified across every ground-truth dump — zero refs printed a filePath).
|
||||
Node `decorators` are likewise never set (no extractModifiers hook) — the
|
||||
decorator channel is `decorates` REFS only.
|
||||
6. **Deferral expectations:** os-lib 0, cats 15/835 = 1.8%, but Scala-3-heavy
|
||||
repos run 10–18% (§Grammar prep table) with phantom-dominated error sets.
|
||||
Default `--max-deferral 0.1` HOLDS on os-lib/cats and on mainstream
|
||||
Scala-2 style; **sweeps over scala3-style repos need `--max-deferral 0.3`
|
||||
(swift precedent)**. A deferral-rate JUMP on cats/os-lib is the bug
|
||||
signal; a big number on dotty-frontier code is grammar reality.
|
||||
|
||||
## Extractor config (languages/scala.ts — 212 lines, read it whole)
|
||||
|
||||
Types: functionTypes=**[]** (comment: "top-level function_definition is
|
||||
handled via methodTypes"); classTypes=[`class_definition`, `object_definition`,
|
||||
`trait_definition`]; methodTypes=[`function_definition`,
|
||||
`function_declaration`]; interfaceTypes=[]; structTypes=[];
|
||||
enumTypes=[`enum_definition`]; enumMemberTypes=**[]** (hook-handled);
|
||||
typeAliasTypes=[`type_definition`]; importTypes=[`import_declaration`];
|
||||
callTypes=[`call_expression`]; variableTypes=[] and fieldTypes=[] (hook);
|
||||
extraClassNodeTypes=[]. nameField=`name`, bodyField=`body`,
|
||||
paramsField=`parameters`, returnField=`return_type`. interfaceKind=`trait`
|
||||
(unused in practice — extractInterface is unreachable, see §dispatch).
|
||||
|
||||
**Field semantics (32 real fields — but first-match-wins bites 3×):**
|
||||
`childForFieldName(f)` returns the FIRST child carrying field `f`, and in this
|
||||
grammar (i) several parents attach the same field to MULTIPLE children, and
|
||||
(ii) **anonymous tokens can carry fields** (probed, `brace-field.out`):
|
||||
braced `extension (t: Int) { … }` puts field `body` on **`{`**, the
|
||||
function_definition, and `}` — first match is the `{` token with
|
||||
namedChildCount 0; a `for` header puts `enumerators` on `{`/enumerators/`}`.
|
||||
The walker's field lookup must scan the FULL child list (named + anonymous)
|
||||
in order, exactly like tree-sitter's `ts_node_child_by_field_name`.
|
||||
|
||||
Hooks PRESENT (port each exactly):
|
||||
|
||||
- **visitNode (scala.ts:131-198)** — runs for EVERY node the main walker
|
||||
visits (tree-sitter.ts:943-953; NOT in visitFunctionBody). Three branches:
|
||||
1. **`val_definition` / `var_definition` (:135-170) — the LIVE branch.**
|
||||
Name via getValVarName (:5-11): `pattern` field; `identifier` → its
|
||||
text; else the pattern's first DIRECT namedChild of type `identifier`
|
||||
(**`val (ta, tb)` → ONE node named `ta` spanning the whole val; `val
|
||||
Some(v)` → `v`; `val multiA, multiB = 5` → `multiA` only** — pinned,
|
||||
unlike kotlin's mint-nothing destructuring); no identifier → return
|
||||
false (falls through to… nothing — no other branch matches, children
|
||||
recursed). Then the **enclosing-definition walk** up node.parent
|
||||
(:146-156): first of `class_definition | trait_definition |
|
||||
enum_definition | given_definition | object_definition` wins.
|
||||
isInstanceField = class/trait/enum/**given** → kind **`field`**;
|
||||
object_definition or NOTHING (top level, package_object, braced package)
|
||||
→ `val`→**`constant`** / `var`→**`variable`**. NOTE vs kotlin: there is
|
||||
NO 'local' arm — the hook never runs inside bodies (visitFunctionBody
|
||||
doesn't call it), so body-local vals are handled by plain recursion
|
||||
(§Body walker). `lazy val` → constant (modifiers don't matter).
|
||||
Extra: `signature` = `` `val|var ${name}: ${typeText}` `` ONLY when a
|
||||
`type` field exists (else undefined — `val x = 1` has NO signature;
|
||||
pinned), `visibility` via extractVisibility (:69-80). Then
|
||||
**emitScalaTypeRefs(typeNode, created.id)** (:27-45): every
|
||||
`type_identifier` in the type subtree EXCEPT SCALA_BUILTIN_TYPES
|
||||
(:14-17 — Int/Long/Short/Byte/Float/Double/Boolean/Char/Unit/String/
|
||||
Any/AnyRef/AnyVal/Nothing/Null) → `references` ref FROM THE VAL NODE at
|
||||
the type_identifier's position (pinned: `val SHARED_TABLE:
|
||||
Map[String, Int]` → references `Map` only — String/Int builtin-skipped;
|
||||
`var cb: () => Unit` → nothing). Return true → dispatcher runs
|
||||
`scanFnRefSubtree(node, 0)` and NEVER descends → **top-level/class-scope
|
||||
property initializers emit NO calls/instantiates refs** (`val topInit =
|
||||
WidgetS.create()`, `val topLazy = compute()`, `val n = new Foo {…}` →
|
||||
nothing — pinned) — but the SCAN still captures fn-ref candidates
|
||||
(§Function-refs; note the scan's nested-def halt checks functionTypes,
|
||||
which is EMPTY for scala, so it descends into nested
|
||||
function_definitions inside a hook-consumed val — it halts only at
|
||||
`lambda_expression` (tree-sitter.ts:611): `val fnField = (x) =>
|
||||
runLam(x)` captures nothing, pinned `extract-edge2.txt`).
|
||||
2. **`enum_case_definitions` (:173-183)** — for each direct
|
||||
`simple_enum_case` | `full_enum_case` child: `enum_member` node named by
|
||||
the case's `name` field, **positioned at the CASE node** (so `case
|
||||
Custom(rgb: Int)` spans the params and `case Earth extends Planet(5.9)`
|
||||
spans the extends — pinned cols in `extract-torture.txt`). One wrapper
|
||||
per `case` line; `case Red, Green` = one wrapper, two cases. Return
|
||||
true → scanFnRefSubtree; consequences: **case parameters are invisible,
|
||||
a case's extends_clause emits NO extends ref, and calls inside case
|
||||
ctor-args emit nothing**.
|
||||
3. **`extension_definition` (:186-195)** — `body = childForFieldName('body')`
|
||||
then visit the body's namedChildren. Because field `body` is attached to
|
||||
EACH def (and to `{`/`}` in braced form), this is a triple quirk, all
|
||||
pinned (`extract-ext.txt`, `brace-field.out`):
|
||||
- paren/indent form: body = the FIRST `function_definition` → its
|
||||
CHILDREN are visited → **no node is ever minted for any extension
|
||||
method**; the first def's body expressions reach the ladder → its
|
||||
calls emit FROM THE ENCLOSING SCOPE (file/class) at their own
|
||||
positions (`calls concat from=file`); non-call bodies (`s.length`)
|
||||
emit nothing.
|
||||
- **every def after the first is COMPLETELY invisible** (never visited).
|
||||
- **braced form (`extension (t) { … }`): body resolves to the `{` TOKEN
|
||||
→ namedChildCount 0 → the whole extension is invisible** (zero nodes,
|
||||
zero refs — `extract-ext.txt` ext2).
|
||||
Return true always (even when body lookup finds nothing).
|
||||
|
||||
- **getSignature (scala.ts:110-117) — LIVE.** `params =
|
||||
childForFieldName('parameters')`, `ret = childForFieldName('return_type')`;
|
||||
none → undefined; sig = paramsText + (ret ? `: ${retText}` : ''). QUIRKS,
|
||||
PRESERVE (pinned in `extract-torture.txt`):
|
||||
- **Curried defs: FIRST parameter list only** — `def curried(a: Int)(b:
|
||||
String)(implicit ord: Ordering[Int]): Int` → sig `(a: Int): Int`.
|
||||
- **A def with type parameters: the TYPE param list wins** — on
|
||||
`function_definition` the type_parameters node carries field name
|
||||
`parameters` and precedes the value list → `def genericDef[A: Numeric,
|
||||
B <: BoundT](x: A): B` → sig `[A: Numeric, B <: BoundT]: B`;
|
||||
`def genericLeak[T](t: T): T` → `[T]: T`.
|
||||
- No params, ret only → `: Int` (RichIntS::twice). Empty parens → `()`.
|
||||
Secondary ctor `def this()` → `()`.
|
||||
- **getReturnType = extractScalaReturnType (scala.ts:56-67)** —
|
||||
`return_type` field text, trimmed: `this.`-prefixed (fluent `this.type`) →
|
||||
undefined; strip `\[[^\]]*\]` generic args (**non-greedy single pass:
|
||||
`List[Bar]`→`List`**), strip all `\s`, take last `.`-segment; must match
|
||||
`/^[A-Za-z_]\w*$/`. Pinned: `: WidgetS`→WidgetS; `: com.example.other.
|
||||
Remote`→`Remote`; `: T`→`T` (generic leak, preserve); inferred → undefined;
|
||||
`Unit`/`Nothing` are NOT filtered (unlike kotlin — `unitRet` has
|
||||
ret="Unit", pinned). Feeds matchDottedCallChain (§Frameworks).
|
||||
- **getVisibility → extractVisibility (scala.ts:69-80)** — scan direct
|
||||
namedChildren of type `modifiers` OR `access_modifier`; TEXT
|
||||
`.includes('private')` → 'private', `.includes('protected')` →
|
||||
'protected'; default **'public'**. Kotlin-style includes-on-raw-text:
|
||||
`private[b]`/`private[this]` → private (pinned QualPriv). Applied to
|
||||
functions/methods/classes(+objects)/enums AND (via the hook) vals/vars.
|
||||
NOTE: in the real CST `access_modifier` sits INSIDE `modifiers` — the
|
||||
modifiers arm is what fires; keep both arms anyway.
|
||||
- **isAsync (scala.ts:121)** — literally `() => false`: every function/method
|
||||
carries `isAsync: false`.
|
||||
- **isStatic (scala.ts:123-129)** — modifiers text `.includes('static')` →
|
||||
scala has no `static` keyword → **always false in practice** (annotations
|
||||
are NOT inside modifiers in this grammar, so no kotlin-style text false
|
||||
positive channel — but port the text scan, not a constant).
|
||||
- **classifyClassNode (scala.ts:105-108)** — `trait_definition` → 'trait',
|
||||
else 'class'. So **object_definition → kind `class`** (companions/case
|
||||
objects included) and trait → kind **`trait`** via
|
||||
`extractClass(node, 'trait')` (ladder :1014-1015 — extractInterface/
|
||||
interfaceKind is DEAD code for scala).
|
||||
- **extractImport (scala.ts:200-211)** — signature = trimmed full node text;
|
||||
moduleName = `childForFieldName('path')` text. **Each dotted segment is a
|
||||
separate `path`-fielded identifier → first-match-wins → the import node/ref
|
||||
is named the FIRST SEGMENT**: `import com.example.other.OtherClass` →
|
||||
name/ref `com` (pinned ×5 in `extract-torture.txt`; `import single` →
|
||||
`single`; `import a.b` → `a`). The identifier/stable_identifier fallback
|
||||
(:204-209) is dead (path always present). Consequence for the fn-ref gate:
|
||||
importedNames = {`com`, `single`, …} — **imported class simple names NEVER
|
||||
enter the gate** (unlike kotlin's last-segment rule — flushFnRefCandidates'
|
||||
QUALIFIED_IMPORT never sees the full path because the ref name is only the
|
||||
first segment).
|
||||
|
||||
Hooks ABSENT (the walker must NOT invent them): `preParse`, `resolveName`,
|
||||
`recoverMangledName`, `isMisparsedFunction`, `isConst`, `isExported`
|
||||
(**undefined everywhere except the file node's literal `false`**),
|
||||
`classifyMethodNode`, `extractPropertyName`, `propertyTypes`, `packageTypes`/
|
||||
`extractPackage` (**→ extractFilePackage returns null → NO namespace node,
|
||||
EVER — package headers are ignored and every top-level QN is bare**; pinned),
|
||||
`getReceiverType` (no receiver-QN surface, no owner-contains fallback),
|
||||
`resolveBody` (body via the `body` FIELD everywhere), `extractModifiers` (no
|
||||
node.decorators), `extractBareCall`, `synthesizeMembers`, `skipBodilessClass`
|
||||
(bodiless `class Foo` mints a node — the :1685 comment names Scala),
|
||||
`methodsAreTopLevel`, `resolveTypeAliasKind`, `interfaceTypes` machinery.
|
||||
|
||||
## tree-sitter.ts branches (anchors as of `45a53eb`)
|
||||
|
||||
### visitNode dispatch — what each scala node hits (ladder at 936-1303)
|
||||
|
||||
| Node | Branch | Behavior |
|
||||
|---|---|---|
|
||||
| every node | visitNode hook first (:943) | val/var, enum_case_definitions, extension consumed; handled → scanFnRefSubtree + STOP |
|
||||
| every node | maybeCaptureFnRefs (:990) | fires for `arguments`/`assignment_expression`/`val_definition` (SCALA_SPEC keys) in visitNode context too |
|
||||
| `function_definition`/`function_declaration` | methodTypes:1027 (functionTypes EMPTY — :994 never fires) | extractMethod:1737 → gate :1747: inside class-like → **method**; top level (no receiver hook, no methodsAreTopLevel, parent never `object`/`object_expression`) → falls to extractFunction:1517 → **function**. `function_declaration` = bodiless def (`def m(): Int` in traits/abstract classes) — same routing, no body walk |
|
||||
| `class_definition` | classTypes:1005 → classify | 'class' → extractClass:1679. Includes `case class`, `implicit class`, `abstract class` |
|
||||
| `object_definition` | classTypes:1005 | extractClass → kind **`class`** (companion objects, `case object`, `object X extends App`) |
|
||||
| `trait_definition` | classTypes:1005 | classify 'trait' → extractClass(node, 'trait') → kind **`trait`** (extractInterface:1834 is UNREACHABLE) |
|
||||
| `enum_definition` | enumTypes:1064 → extractEnum:1914 | §Enums |
|
||||
| `type_definition` | typeAliasTypes:1071 → extractTypeAlias:2890 | plain `type_alias` node (top level AND as a class member — `Outer2::Member` pinned). QUIRK: the alias-value ref walk reads field `'value'` → scala's field is `'type'` → **NO reference to the aliased type** (returns false → children re-visited, nothing matches). `opaque type` identical |
|
||||
| `import_declaration` | importTypes:1209 → extractImport:3170 | §Imports |
|
||||
| `package_clause` | **no branch** | recursed. Header form: nothing extracted, nothing pushed — contents stay file-parented with bare QNs. **Braced form `package a.b { class X }`: recursion reaches the members** (class X → file-parented, QN `X` — pinned `extract-misc.txt`). Multiple/chained package clauses likewise ignored |
|
||||
| `package_object` | **no branch** | recursed → template_body members visited with FILE on stack: defs → extractMethod → not-class-like → **functions**, vals → hook (enclosingDef walk finds NOTHING — package_object is not in the list) → **constants/variables**; all bare-QN file children (pinned: `pkgHelper` function, `pkgShared` constant) |
|
||||
| `given_definition` | **no branch** | recursed. `given x: T = new T {…}` → instance_expression child → :1255 → **instantiates** from the enclosing scope + (anon-body via recursion) §instance_expression row. `given T = expr` → expr recursed (calls emit from enclosing scope — `summonOrd()` pinned). **No given node is ever minted; a given's name binds nothing** |
|
||||
| `extension_definition` | hook | §Extractor config — first-def leak / braced invisibility |
|
||||
| `call_expression` (top level / template_body statements) | callTypes:1248 → extractCall:3684 | class-body statements (`require(size > 0)` after the primary ctor, `object Boot extends App { bootUp() }`) attribute to the CLASS node — pinned `extract-edge2.txt` |
|
||||
| `instance_expression` | INSTANTIATION_KINDS:1255 (:360 names scala) → extractInstantiation:4610 | §Instantiation. findAnonymousClassBody:4815 looks for `class_body`/`declaration_list` — scala's anon body is `template_body` → **null → extractAnonymousClass NEVER runs** → skipChildren stays false → **children recursed**: the `template_body`'s defs hit methodTypes → extractMethod → (not class-like at top level) → **anon-class methods leak out as functions/methods of the ENCLOSING scope** — pinned: top-level given's `compare` → `function compare` (bare QN); given-inside-object's `compare`/`innerVal` → **method/field OF the object** (`Registry::compare`, `extract-given2.txt`) |
|
||||
| `infix_expression` | **no branch** | recursed; `left`/`operator`/`right` fields. **Infix calls are INVISIBLE**: `list map transform`, `a foo b`, `1 :: rest`, `x + y`, `counter += 1` emit NOTHING (no calls ref, no fn-ref — pinned) |
|
||||
| `assignment_expression` | no ladder branch; SCALA_SPEC dispatch | fn-ref rhs capture (§Function-refs); children recursed (LHS field_expression reaches extractStaticMemberRef in bodies — §Static-member) |
|
||||
| `annotation` | no branch (child of its definition) | consumed by extractDecoratorsFor from the decorated node (§Decorators) |
|
||||
| `lambda_expression` / `case_block` / `match_expression` / `for_expression` / `indented_block` / `block` | no branch | recursed transparently (in bodies via visitForCallsAndStructure) |
|
||||
| INSTANTIATION_KINDS others / `impl_item` / swift property / property_signature / export_statement (TS) | never | not scala node kinds; the swift property branch :1121 is language-gated |
|
||||
|
||||
### Node creation, IDs, qualified names
|
||||
|
||||
- createNode (:1308): id = `generateNodeId(filePath, kind, name,
|
||||
startRow+1)` = `` `${kind}:${sha256(`${filePath}:${kind}:${name}:${line}`).hex.slice(0,32)}` ``.
|
||||
FILE node id = `file:${filePath}` (:509), name = basename, qualifiedName =
|
||||
filePath, endLine = `source.split('\n').length`, isExported false.
|
||||
- endLine extension via resolveBody (:1329-1334) calls the hook — ABSENT for
|
||||
scala → `getChildByField(node, 'body')`… note :1330 only calls
|
||||
`this.extractor.resolveBody?.()` — hook absent → **no extension ever**
|
||||
(scala bodies are child nodes, in-range anyway). Kernel: skip.
|
||||
- contains edge from stack top for every node (:1363); extractModifiers merge
|
||||
(:1355) inert; captureValueRefScope (:1374) live (§Value-refs).
|
||||
- **buildQualifiedName (:1447-1460)**: stack names joined `::`, file node
|
||||
excluded, namespacePrefix always empty. **No namespace node** → top-level
|
||||
QN = bare name even in a packaged file (pinned everywhere).
|
||||
**Companion objects: class + object are TWO `class`-kind nodes with the
|
||||
SAME name** (different start lines → different ids) and **their members
|
||||
share one QN namespace** — `WidgetS::render` (class) and `WidgetS::create`
|
||||
(companion) are indistinguishable by QN (pinned; no disambiguation —
|
||||
preserve).
|
||||
- isInsideClassLikeNode (:1486): stack-top kind ∈ {class, struct, interface,
|
||||
trait, enum, module} — objects count (kind class), traits count, enums
|
||||
count.
|
||||
- isClassScopeConstantAssignment (:1508): needs node.type `assignment` —
|
||||
scala's is `assignment_expression` → always false (Ruby-only, dead).
|
||||
|
||||
### extractFunction / extractMethod (:1517 / :1737) — every def
|
||||
|
||||
Route recap: ALL defs → extractMethod first. In class-like → method node.
|
||||
Top level → :1747 gate fails → extractFunction (the `object`/
|
||||
`object_expression` parent check :1751 never matches scala) → function node.
|
||||
extras (both): docstring (§Docstrings), signature (LIVE hook — §config),
|
||||
visibility, isAsync false, isStatic false, returnType (hook); isExported only
|
||||
on extractFunction path → undefined (hook absent). extractTypeAnnotations
|
||||
(§Type-annotation refs) then extractDecoratorsFor (§Decorators), push, body =
|
||||
`getChildByField(node, 'body')` (block / indented_block / EXPRESSION — a
|
||||
single-expression body like `= new WidgetS(a)` or `= a + 1` is walked as the
|
||||
body; pinned instantiates from `topLevel`), visitFunctionBody, pop.
|
||||
|
||||
- Names: `name` field. **operator defs keep the operator text as the name**
|
||||
(`+`, `::` — `operator_identifier` node; `unary_-` is a plain identifier;
|
||||
pinned method `WidgetS::+`). Backtick names keep backticks
|
||||
(`` class `Weird Name` ``/`` method `strange def` `` pinned).
|
||||
**Secondary constructors are methods named `this`** (`def this() =
|
||||
this(0)` → method `WidgetS::this` + calls ref `this` — pinned; the
|
||||
NAME_STOPLIST only applies to fn-refs, not calls).
|
||||
- `function_declaration` (bodiless def): method node minted, sig/ret intact,
|
||||
no body walk (`AbsS::abstractM` pinned).
|
||||
|
||||
### extractClass (:1679) — classes, objects, traits + the bodiless-header asymmetry
|
||||
|
||||
resolvedBody = hook(absent) ?? `getChildByField(node, 'body')` → `template_body`
|
||||
(braces or scala-3 colon form — same node type) / `enum_body` for enums. NO
|
||||
skipBodilessClass → bodiless mints. extras: docstring, visibility, isExported
|
||||
undefined. Then extractInheritance (§below) — extends refs precede member
|
||||
emissions. extractCsharpPrimaryCtorParamRefs — csharp-gated no-op.
|
||||
extractDecoratorsFor (annotations on the class — `@deprecated class Old` →
|
||||
decorates). Push, walk body children via visitNode, pop.
|
||||
|
||||
- **Bodied class: only template_body children visited** → the
|
||||
`class_parameters` (primary ctor) child is NEVER walked: **ctor params mint
|
||||
no nodes, their types emit NO references, their default-value calls emit
|
||||
NOTHING** (pinned: WidgetS's `label: String = defaultLabel()` → silent;
|
||||
case-class fields invisible — DataS has zero members).
|
||||
- **Bodiless class/object: body = the node itself (:1714) → HEADER children
|
||||
visited**: class_parameters recursion reaches default-value
|
||||
`call_expression`s → **calls from the CLASS node** (pinned: `case class
|
||||
DataS(x: Int, y: String = mkY())` → `calls mkY from=class:DataS`), and
|
||||
extends_clause `arguments` recursion reaches super-ctor arg calls the same
|
||||
way. Reproduce the asymmetry exactly. (The re-visited extends_clause emits
|
||||
nothing extra — extends refs come only from extractInheritance; type
|
||||
nodes/class_parameter children match no ladder branch.)
|
||||
- Class-body members: val/var → hook (fields — or constants/variables inside
|
||||
OBJECTS via the enclosingDef walk, making them value-ref targets); defs →
|
||||
extractMethod; nested class/object/trait/enum/type_definition → their
|
||||
branches (QN chains pinned: `Outer2::InnerObj::IC`); template_body
|
||||
STATEMENTS (calls) → extractCall from the class (§dispatch); secondary
|
||||
ctor → method `this`.
|
||||
- Traits: extractClass with kind trait — **visibility IS emitted for traits**
|
||||
(extractClass path, not extractInterface — vis="public" pinned) unlike
|
||||
kotlin's interface path. Trait vals → hook 'instance' → **field**
|
||||
(`Drawable::traitVal` pinned); bodiless trait defs → methods.
|
||||
- Self-types (`trait X { self: Y => … }`): the self-type is invisible (no
|
||||
refs, no node); members extract normally (pinned `extract-misc.txt`).
|
||||
|
||||
### Enums (:1914 extractEnum + hook branch 2)
|
||||
|
||||
body = `enum_body` (REQUIRED — a bodiless enum would mint nothing; doesn't
|
||||
occur). extras: docstring, visibility, isExported undefined.
|
||||
extractInheritance runs (an enum's own extends_clause). Body loop
|
||||
(:1941-1950): enumMemberTypes is EMPTY → every child goes through visitNode:
|
||||
`enum_case_definitions` → hook → **enum_member nodes positioned at the case
|
||||
nodes** (simple case = just the name extent; param/extends cases span their
|
||||
tails — cols pinned); `function_definition` → extractMethod (enum is
|
||||
class-like → `Http::describe`); vals → hook → fields. extractEnumMembers
|
||||
(:1958) is DEAD for scala. QUIRKS, PRESERVE: **case parameters
|
||||
(`Custom(rgb: Int)`) and per-case extends (`case Earth extends Planet(5.9)`)
|
||||
are completely invisible** — no field nodes, no extends refs, no calls from
|
||||
ctor args (hook consumption). Enum class_parameters (`enum Planet(mass:
|
||||
Double)`) are invisible like any bodied class's.
|
||||
|
||||
### Imports (:3170-3236)
|
||||
|
||||
Hook returns {moduleName: FIRST path segment, signature: trimmed full text} →
|
||||
import node (name/QN = first segment) + the generic `imports` ref
|
||||
(:3183-3194): {fromNodeId: **always the file node** (no namespace),
|
||||
referenceName: first segment, line/column of the import_declaration}. No
|
||||
scala-specific emit pass (:3197-3234 all gated to other languages). Shapes
|
||||
(CST pinned in `cst-snippets.txt` §imports):
|
||||
|
||||
- `import a.b.C` → three `path` identifiers → name `a`.
|
||||
- selectors `{C, D}` (namespace_selectors), wildcard `_`/`*`/`given`
|
||||
(namespace_wildcard), renames `{X => Y}` (arrow_renamed_identifier) /
|
||||
`{X as Y}` (as_renamed_identifier, name/alias fields) — ALL invisible: the
|
||||
name is still the first `path` segment; selectors/aliases bind nothing.
|
||||
- `import single` (one segment) → `single`.
|
||||
- No comment-gluing (kotlin's quirk does NOT reproduce — scala comments stay
|
||||
siblings; docs pinned separately).
|
||||
|
||||
### extractCall (:3684) — the scala paths
|
||||
|
||||
Entry: not vbnet/erlang/ruby/arkts. `func = getChildByField(node,
|
||||
'function') ?? namedChild(0)` (:4313) — scala call_expression HAS a real
|
||||
`function` field. cpp operator recovery :4324 gated off.
|
||||
|
||||
**Member branch (:4364)** — func.type === `field_expression` (in the :4364
|
||||
list):
|
||||
|
||||
1. property = getChildByField('property') → null; **getChildByField('field')
|
||||
→ the member identifier** (scala field_expression fields: `value` +
|
||||
`field`). The kotlin navigation_suffix fallback is dead.
|
||||
2. receiver = object/operand/argument fields → all null → `func.namedChild(0)`
|
||||
= the `value` child.
|
||||
3. LITERAL_RECEIVER_TYPES (:4397, set :373-388): scala hits `string`
|
||||
(`"lit".toUpperCase()`) and `integer_literal` (`5.toString()`) → **emit
|
||||
NOTHING** (pinned). Port the whole set.
|
||||
4. receiver `identifier` (:4401) not in SKIP_RECEIVERS {self,this,cls,super}
|
||||
→ `` `${recv}.${method}` `` (`w.render`, `Registry.register`,
|
||||
`obj.method`). **`this`/`super` receivers are plain `identifier` nodes
|
||||
with those TEXTS** → SKIP → bare methodName (`this.mine()` → `mine`,
|
||||
`super.hashCode()` → `hashCode` — pinned; unlike kotlin's
|
||||
this_expression path, same net effect).
|
||||
5. **receiver `call_expression` + scala in the gate (:4408-4418) → the #750
|
||||
re-encode, scala arm (:4443-4464):** innerFn = `getChildByField(receiver,
|
||||
'function')` (a REAL field here — NOT kotlin's namedChild(0)) → text with
|
||||
`->`→`.` then `\s+` stripped; **re-encode ONLY when `/^[A-Z]/`**
|
||||
(:4461) → `` `${innerCallee}().${methodName}` ``. Pinned:
|
||||
`WidgetS.create().render()` → `WidgetS.create().render` + inner
|
||||
`WidgetS.create` (recursion); `Foo(1).bar()` → `Foo().bar` + `Foo`
|
||||
(companion-apply chain); `lowerFactory().chain()` → bare `chain` +
|
||||
`lowerFactory`.
|
||||
6. receiver `field_expression` (2-hop `a.b.method3()`) or anything else →
|
||||
bare methodName (pinned).
|
||||
|
||||
**Else branch (:4518-4520)** — calleeName = RAW func text: bare `helper`;
|
||||
**apply-sugar `WidgetS(1)` → calls ref `WidgetS`** (capitalized bare —
|
||||
resolution's CONSTRUCTS_VIA_BARE_CALL handles it, §Frameworks);
|
||||
`this(0)` in a secondary ctor → calls `this`. QUIRKS, PRESERVE (pinned):
|
||||
|
||||
- **`generic_function` callee keeps its type args**: `genericCall[Int](1)` →
|
||||
calls **`genericCall[Int]`** (the c/cpp `<`-strip :4542 is gated off and
|
||||
targets `<` anyway; deterministic garbage — reproduce byte-for-byte).
|
||||
- **Curried calls emit raw-text inners**: `curried(1)(2)` → outer callee =
|
||||
func (a call_expression) raw text **`curried(1)`** + inner `curried`
|
||||
(recursion); `Foo(1)(2)` → `Foo(1)` + `Foo` (pinned `extract-misc.txt`).
|
||||
- The parenthesized-conversion regex (:4530) applies (single-name parens
|
||||
rewrite) — port it.
|
||||
- Final ref: {callerId = stack top, name, line = call startRow+1, column =
|
||||
call startColumn (UTF-16)}. Chains/args re-visited by recursion.
|
||||
- Calls inside string-interpolations EMIT (in bodies): `s"… ${w.render()} …"`
|
||||
→ calls `w.render` at the inner call's position (interpolation > block >
|
||||
call recursion); `$id` (interpolation > identifier) emits no call (pinned).
|
||||
- Lambda args (`xs.map(el => …)` / `{ el => … }` / partial-fn `{ case q =>
|
||||
… }`) — arguments node is `arguments`/`block`/`case_block` respectively;
|
||||
all recursed; inner calls attribute to the ENCLOSING function (lambdas mint
|
||||
nothing).
|
||||
|
||||
### Instantiation — instance_expression (:4610, scala arm :4647-4662)
|
||||
|
||||
ctor = constructor/type/name fields → **all null on instance_expression**
|
||||
(only `arguments` is a field on it) → `namedChild(0)` = the type node →
|
||||
**scalaBaseTypeName (:201-224)**: type_identifier/identifier → text;
|
||||
generic_type → recurse namedChild(0) (`new Monoid[Int]` → `Monoid`);
|
||||
stable_type_identifier/stable_identifier → LAST identifier segment
|
||||
(`new a.b.C()` → `C`); default → first direct type_identifier child ?? null.
|
||||
→ **`instantiates` ref at the instance_expression's position** (the `new`).
|
||||
Emitted from: expression bodies (`def f = new W(a)` — from the function),
|
||||
body statements/initializers in bodies, **given RHS at any scope** (from
|
||||
file/class), ladder-visited statement positions. **NOT emitted from
|
||||
hook-consumed val initializers** (`val topInit = new W(…)` at top/class
|
||||
scope → nothing — pinned). Anonymous-class bodies: §dispatch row — never an
|
||||
anon class node; members leak to the enclosing scope; **no extends ref to
|
||||
the instantiated type** (extractAnonymousClass never runs).
|
||||
|
||||
### Static-member / value-read refs (:4750-4808) — scala IS in STATIC_MEMBER_LANGS (:346)
|
||||
|
||||
Called ONLY from the body walker (:5218). `field_expression` ∈
|
||||
MEMBER_ACCESS_TYPES (:327). Mechanics: callee-of-call skip (:4772-4778 —
|
||||
`Registry.register(w)`'s callee nav emits nothing); recv = object/expression/
|
||||
scope fields (null) ?? namedChild(0); accepted type `identifier` (:4792);
|
||||
text `/^[A-Z][A-Za-z0-9_]*$/` → **`references` ref at the RECEIVER's
|
||||
position**. Pinned (`extract-torture.txt` StaticReads):
|
||||
|
||||
- `val c1 = Registry.count` → references `Registry`; `Http.Ok` → references
|
||||
`Http`; `process(Registry.count)` (argument position) → references
|
||||
`Registry` (plus the `process` call).
|
||||
- `com.example.Fq.CONST_READ` → NOTHING (outer recv is a field_expression;
|
||||
the innermost `com` is lowercase).
|
||||
- **Assignment WRITES DO emit** (unlike kotlin): `Registry.count = 5` — the
|
||||
LHS field_expression is a plain child of `assignment_expression`, visited
|
||||
by body recursion → references `Registry` (pinned). Preserve.
|
||||
- Top-level/class-scope reads emit nothing (body walker only); hook-consumed
|
||||
initializers doubly so.
|
||||
|
||||
### Decorators — scala annotations DO emit `decorates`, args included
|
||||
|
||||
extractDecoratorsFor (:4897) runs for functions/methods/classes/objects/
|
||||
traits/enums (NOT hook-created vals/enum members — pinned: `@volatile var`
|
||||
→ nothing). Scala annotations are DIRECT CHILDREN of the definition node
|
||||
(`annotation` node with fields name: type_identifier, arguments) — scan #1
|
||||
(:4976-4978) hits them; consider() accepts type `annotation` (:4928); the
|
||||
target loop finds the **`type_identifier` name child** (:4951) → name text →
|
||||
`<`-strip + last-`.`/`::`-segment normalization (:4959-4962) →
|
||||
**`decorates` ref {from: decorated node, name, line/col of the ANNOTATION
|
||||
node}**. Pinned: `@main def entry` → decorates `main`; `@inline def fast` →
|
||||
`inline`; **`@deprecated("gone", "1.0") def old` → decorates `deprecated`**
|
||||
(WITH-args annotations still emit — the name field precedes the arguments;
|
||||
kotlin's constructor_invocation dropout does NOT reproduce). Annotation
|
||||
ARGUMENT expressions are never visited (no calls refs from them). The
|
||||
modifiers-descent (:4983) and backward-sibling scan (:5013) are inert for
|
||||
scala (annotations are neither inside `modifiers` nor preceding siblings).
|
||||
|
||||
### Inheritance — extends_clause, the scala branch (:5339-5360)
|
||||
|
||||
extractInheritance loops the DEFINITION node's direct namedChildren for
|
||||
`extends_clause` (field `extend`; a direct child on class/object/trait/enum
|
||||
definitions). **The scala branch iterates ALL namedChildren of the
|
||||
extends_clause** and maps each through scalaBaseTypeName → one **`extends`**
|
||||
ref per supertype {name, line/col of the supertype node}. Pinned:
|
||||
|
||||
- `extends BaseW(size) with Drawable with Ordered[WidgetS]` → extends
|
||||
`BaseW` + `Drawable` + `Ordered` (generic unwrapped; the `arguments` child
|
||||
yields null via the default case → skipped).
|
||||
- scala-3 comma form `extends B, C` → both. Single `extends Shape` on
|
||||
objects (`object Circle extends Shape`) ✓ (pinned).
|
||||
- Curried super-ctor args `extends Base(1)(2)` (plain, parse-clean) → extends
|
||||
`Base` only (the second `arguments` child → null → skipped;
|
||||
`extract-super2.txt`).
|
||||
- **`derives Show` (derives_clause, field `derive`) emits NOTHING** — not an
|
||||
extends_clause; preserve the silence.
|
||||
- Scala NEVER emits `implements` — traits ride extends.
|
||||
- Enum CASE extends (`case Earth extends Planet(5.9)`) — hook-consumed,
|
||||
nothing (§Enums). Anonymous `new T {…}` — no class node, no extends
|
||||
(§Instantiation).
|
||||
|
||||
### Type-annotation references — scala ∈ TYPE_ANNOTATION_LANGUAGES (:5753), THREE live walks + the hook's
|
||||
|
||||
extractTypeAnnotations (:5788) for functions/methods:
|
||||
|
||||
1. **scala-only params walk (:5839-5842): EVERY direct `parameters` child**
|
||||
(all curried lists — the trailing implicit list included) →
|
||||
extractTypeRefsFromSubtree (:6090): every `type_identifier` leaf except
|
||||
BUILTIN_TYPES (:5768-5782 — includes the scala block Int/Long/…/Null
|
||||
PLUS the cross-language entries; note lowercase `error` etc. can't occur)
|
||||
→ `references` ref at the leaf. **The type_parameters node also carries
|
||||
field name `parameters` but is NOT type `parameters`** → not matched by
|
||||
this walk (it's matched by walk 3).
|
||||
2. return walk (:5851): `getChildByField('return_type')` subtree → refs
|
||||
(generic return `Option[A]` → `Option` + `A`).
|
||||
3. **scala-only type-parameters walk (:5863-5870)**: the first
|
||||
`type_parameters` child → subtree refs — **context/upper bounds emit**
|
||||
(`[A: Numeric, B <: BoundT]` → `Numeric` + `BoundT`; the declared names
|
||||
A/B are `identifier` nodes → silent). Pinned order for `genericDef`:
|
||||
params-walk refs (`A`) → return (`B`) → bounds (`Numeric`, `BoundT`).
|
||||
4. the `type_annotation` direct-child search (:5873) — no such node kind in
|
||||
scala → dead.
|
||||
|
||||
Plus **the hook's emitScalaTypeRefs on val/var type ascriptions**
|
||||
(§Extractor config — its OWN builtin set SCALA_BUILTIN_TYPES, which lacks
|
||||
the cross-language entries; `val m: Monoid[Int]` → references `Monoid`).
|
||||
extractVariableTypeAnnotation (:6074) needs a `type_annotation` child → dead;
|
||||
the body-walker variable_declarator branch (:5230) — no such kind → dead.
|
||||
**Net: param/return/bound types on defs + ascribed val/var types emit;
|
||||
class_parameters types (bodied classes), local val types in bodies (no hook
|
||||
there, no variable_declarator), and pattern types (`case ws: WidgetS`) emit
|
||||
NOTHING.** Class primary-ctor types are the notable coverage hole — preserve
|
||||
it.
|
||||
|
||||
### Docstrings (tree-sitter-helpers.ts:95-127) — Scaladoc is KEPT
|
||||
|
||||
Scala comment node kinds: `comment` (`//`) and `block_comment` (`/* */` and
|
||||
Scaladoc `/** */`). **BOTH are in getPrecedingDocstring's accepted set**
|
||||
(:110-115) — unlike kotlin, Scaladoc survives, and block/line runs CHAIN in
|
||||
either order. cleanCommentMarkers (:77-90): `/*`-open strip + per-line `*`
|
||||
continuation strip + `//` strip. Pinned (`extract-docs.txt`): Scaladoc →
|
||||
`"Scaladoc kept?\nsecond line with star"`; `//` runs joined with `\n`;
|
||||
`/** block */` + `// trailing` → `"block doc\ntrailing line"`; **a blank
|
||||
line does NOT break the chain** (`// detached` two lines above still
|
||||
attaches — previousNamedSibling skips whitespace); hook-created vals get NO
|
||||
docstring (docVal pinned) but class/method/function/trait/enum/type_alias
|
||||
do. DOCSTRING_WRAPPER_TYPES contains no scala kinds → no climbing.
|
||||
**CRLF: multi-line block comments keep a stray `\r`** at each internal line
|
||||
break (`"Scaladoc kept?\rsecond line with star"` — pinned
|
||||
`extract-docs-crlf.txt` diff) — the JS `^/m`-matches-after-`\r` semantics
|
||||
(#1329): **call the shared `js_multiline_strip` in docstring.rs, port
|
||||
nothing**. Line-comment runs are CRLF-clean (per-comment trim eats the
|
||||
`\r`).
|
||||
|
||||
### Value-reference edges (:398-931) — scala IS in VALUE_REF_LANGS (:401)
|
||||
|
||||
Port the full machinery (crib kotlin.rs): `CODEGRAPH_VALUE_REFS=0` kill;
|
||||
MAX_VALUE_REF_NODES 20,000; isGeneratedFile skip.
|
||||
|
||||
- **Targets** (captureValueRefScope:735): kind constant|variable, name len ≥3
|
||||
AND `/[A-Z_]/`, parent id prefix ∈ {file:, class:, module:, struct:,
|
||||
enum:}. **Because scala mints NO namespace node, top-level constants ARE
|
||||
targets** (kotlin's namespace-drop quirk does NOT apply — `TOP_LIMIT`
|
||||
pinned with a class-method reader). Object/class/trait members (class:
|
||||
parent) are targets; `count` (no uppercase/underscore) is not (pinned).
|
||||
**Same-name targets: LAST registration wins the map slot** — with
|
||||
`Config.TIMEOUT_MS` and `Config2.TIMEOUT_MS` in one file, readers of
|
||||
EITHER get an edge to **Config2's node** (fileScopeValues.set overwrite —
|
||||
pinned `vref2.out`; wrong-target quirk, PRESERVE).
|
||||
- **Reader scopes**: every function/method/constant/variable node (:764) —
|
||||
fields are NOT readers.
|
||||
- **Shadow prune** (:803-878): the scala declarator case is
|
||||
**`val_definition`/`var_definition` (:838-843)** — `pattern` field, bump
|
||||
ONLY when the pattern IS an `identifier` (tuple/case-class patterns bump
|
||||
nothing → a destructured local shadow never prunes). bump() counts
|
||||
`identifier` nodes (:807). Every val/var ANYWHERE (the target's own
|
||||
declarator + body locals — the hook never sees body locals but the prune
|
||||
DFS does) — pinned: companion-ish `Config.RETRY_MAX` + method-local
|
||||
`val RETRY_MAX = 9` → RETRY_MAX pruned (readBoth emits only TIMEOUT_MS).
|
||||
The kotlin/swift `property_declaration` case and Dart/Pascal cases are
|
||||
null paths here.
|
||||
- **Emission** (:880-930): per reader scope, stack-based DFS
|
||||
(reverse-source-order pop — ruby precedent; edge order follows), reader
|
||||
node type `identifier` (:907 — `simple_identifier`/`constant`/`name`
|
||||
never occur in scala trees). Reads through `Config.TIMEOUT_MS` member
|
||||
positions count (the member half is an `identifier`). **String
|
||||
interpolations: BOTH `$CONST` and `${CONST}` count** (interpolation >
|
||||
identifier / > block > identifier — pinned `readInterp` ×2; kotlin's
|
||||
`$X`-inert quirk does NOT apply). Skip self/same-name, dedupe per
|
||||
(scope,target) → EDGE {kind:'references', metadata:{valueRef:true}},
|
||||
**appended after the walk** (§Emission order). The Dart/Pascal
|
||||
next-sibling pull (:891) is inert (scala bodies nest).
|
||||
|
||||
### Function-as-value capture (#756) — SCALA_SPEC (function-ref.ts:300-308)
|
||||
|
||||
idTypes = **{`identifier`}** (bare identifiers ARE candidates — unlike
|
||||
kotlin). dispatch: `arguments` → args; `assignment_expression` → rhs field
|
||||
`right`; **`val_definition` → varinit field `value`** (var_definition is
|
||||
ABSENT — **`var cb = other` initializers are NEVER captured**, pinned
|
||||
`extract-fnref.txt`). unwrap: `postfix_expression` → null field → first
|
||||
namedChild (**eta-expansion `handler _`** → `handler`, explicitRef=true).
|
||||
No layers/special/ungatedModes/addressOfOnly. NAME_STOPLIST applies
|
||||
(`this`, `null`, …).
|
||||
|
||||
- Capture points: ladder :990 (top-level/template statements),
|
||||
visitFunctionBody :5137 (body nodes), scanFnRefSubtree (hook-consumed
|
||||
val/enum-case/extension subtrees — the scan visits the val_definition
|
||||
itself at depth 0, so **class/object/top-level `val x = fn` IS captured**;
|
||||
the nested-def halt checks functionTypes=[] so only `lambda_expression`
|
||||
halts it — `val fnField = (x) => runLam(x)` captures nothing, pinned).
|
||||
- Pinned channels (`extract-fnref.txt`, `extract-fnref2.txt`,
|
||||
`extract-edge2.txt`): args bare id `register(handler)` ✓; args eta
|
||||
`registerEta(other _)` ✓; body varinit `val stored = alpha` ✓; class-scope
|
||||
varinit `val topStored = handler` ✓ (from the CLASS node); assignment rhs
|
||||
`obj.cb = beta` ✓; **named-argument `wire(cb = cbTarget)`** — parses as an
|
||||
assignment_expression inside arguments → rhs capture ✓ (at the rhs id's
|
||||
position) with the param-forward skip when `cb = cb` (lhs tail == rhs
|
||||
text, :430-443); list `List(delta)` inside a hook-consumed val ✓ (args
|
||||
mode via `arguments`).
|
||||
- NOT captured: `fn.member` values (field_expression — no special), infix
|
||||
positions (`list map transform` — no dispatch key), var initializers,
|
||||
lambda bodies under hook-consumed vals.
|
||||
- Flush gate (:639-728): definedHere (same-file function/method names — a
|
||||
top-level def IS kind function, so `register(missing)` drops but
|
||||
same-file names pass) ∪ importedNames — which for scala is **first path
|
||||
segments only** (§Imports; effectively: cross-file callables never pass
|
||||
the gate). No `::`/`this.` always-flush forms are ever produced. Dedupe
|
||||
`${fromNodeId}|${name}` (pinned: three same-scope `handler` captures → one
|
||||
ref). referenceKind `function_ref` (wire code 200).
|
||||
|
||||
### visitFunctionBody (:5129-5286) — scala rows
|
||||
|
||||
- maybeCaptureFnRefs (:5137) per node (assignment/varinit/args captures in
|
||||
bodies).
|
||||
- `call_expression` → extractCall (:5143), children recursed after.
|
||||
- **`instance_expression` → :5145 extractInstantiation; findAnonymousClassBody
|
||||
→ null (template_body) → NO return → children recursed** → anon-body defs
|
||||
are NOT dispatched here (functionTypes empty, methodTypes not checked in
|
||||
this walker) → **`new Ordering[Int] { def compare … }` in a body: the
|
||||
instantiates ref only; `compare` mints NOTHING and its body's calls
|
||||
attribute to the enclosing method** (pinned).
|
||||
- extractBareCall — absent.
|
||||
- **Nested named defs mint NOTHING** (:5245 checks functionTypes — EMPTY):
|
||||
`def innerFn(k) = …` inside a body is recursed-through; its calls
|
||||
attribute to the ENCLOSING method (pinned: `innerFn`/`lam`/`helperCall`
|
||||
all from `CallSites::localHost`). **This is the inverse of kotlin** (which
|
||||
minted local fns) — preserve.
|
||||
- classTypes (:5255): body-local `class LocalClass`/`object LocalObj`/trait →
|
||||
FULL extractClass (kind class/trait) contained by the enclosing method,
|
||||
members extract normally (pinned QNs `CallSites::localHost::LocalClass::lm`).
|
||||
enumTypes (:5268) likewise. typeAliasTypes NOT checked → body-local
|
||||
`type X = …` is invisible.
|
||||
- extractStaticMemberRef (:5218) every node (§Static-member).
|
||||
- variable_declarator branch (:5230) dead. Body-local val/var: no hook here →
|
||||
plain recursion → **initializer calls/instantiations EMIT from the
|
||||
enclosing function** (`val local = compute()` → calls compute — pinned)
|
||||
and no local nodes are minted. Match/for/try/if recursed transparently
|
||||
(case_clause pattern `Some(n)` emits nothing; `for … yield combine(x,y)`
|
||||
emits the call — pinned).
|
||||
|
||||
### Misc shared paths
|
||||
|
||||
- Positions: line = startRow+1, column = startColumn — **UTF-16 code units**
|
||||
(pinned `extract-uni.txt`: `😀` counts 2, `é` counts 1), as are
|
||||
startIndex/endIndex substrings (getNodeText) and the import-signature
|
||||
trim.
|
||||
- Refs carry NO filePath/language (§arch-5).
|
||||
- `extract()` wrap (:454-577): file node → (no namespace) → walk →
|
||||
`flushFnRefCandidates` → `flushValueRefs` → pop. Table order: nodes in
|
||||
creation order; contains edges interleaved; **value-ref EDGES appended
|
||||
LAST** (pinned — after all contains); walk-order refs, then function_ref
|
||||
refs at flush (pinned last in every dump). Store/harness are
|
||||
rowid-order-sensitive — reproduce exactly.
|
||||
- **CRLF hazards inventory for the scala path**: scala.ts has ONE regex over
|
||||
node text (extractScalaReturnType's strips — `\s+` handles `\r`, JS/Rust
|
||||
parity holds) and the visibility/isStatic `.includes` (single-token). The
|
||||
shared regexes that fire: extractCall's parenthesized-conversion (:4530),
|
||||
decorator name normalization (:4959-4962), and cleanCommentMarkers' gm
|
||||
strips → **`js_multiline_strip` in docstring.rs (#1329), call it** (the
|
||||
block-comment `\r` retention is the observable — §Docstrings). Grammar
|
||||
CRLF probed clean (§Grammar prep). `extract-torture.txt` vs
|
||||
`extract-torture-crlf.txt` diff is EMPTY (torture's only multi-line
|
||||
comment is `//`-run based… its scaladoc is single-line); the docs fixture
|
||||
carries the multi-line pin.
|
||||
- Defer policy: per-file `has_error()` → `defer:` — **including phantoms**
|
||||
(§Grammar prep (a): flag-true with zero ERROR nodes is COMMON in scala-3
|
||||
code; trust the flag). wasm recovery is canonical.
|
||||
- MAX_FILE_SIZE (1 MiB) / generated-file skips: shared, nothing
|
||||
scala-specific.
|
||||
|
||||
## Frameworks & synthesis consumers (stay TS-side — pin the walker's output contract)
|
||||
|
||||
- **playResolver (resolution/frameworks/play.ts)** — §Architecture #2 for
|
||||
detection/decoded-path. Its extract() only fires on `conf/routes` /
|
||||
`*.routes` files (NOT .scala — they ride the no-grammar path with
|
||||
language 'yaml'); route nodes carry literal ids
|
||||
(`route:${filePath}:${line}:${method}:${routePath}`) and `language:
|
||||
'scala'`; resolve() maps `Controller.method` handler refs via
|
||||
`getNodesByName(className)` (kind class) + `getNodesInFile(cls.filePath)`
|
||||
(kind method|function, exact name). **Walker obligations: class/method
|
||||
node names, kinds, filePath** — nothing else.
|
||||
- **matchDottedCallChain** (name-matcher.ts:2145-2166, scala gated) —
|
||||
consumes the `Foo.create().bar` re-encode + the inner method's
|
||||
**returnType** (extractScalaReturnType's product — why `Unit` is left
|
||||
unfiltered doesn't matter here, but the last-segment/generic-strip rules
|
||||
do). CHAIN_LANGUAGES (resolution/index.ts:42) includes scala (the `().`
|
||||
split).
|
||||
- **CONSTRUCTS_VIA_BARE_CALL** (name-matcher.ts:918) — `Foo(args).method()`
|
||||
receivers resolve as class Foo (companion-apply convention).
|
||||
- **Local-variable receiver inference** (name-matcher.ts:1197-1201) —
|
||||
re-reads SOURCE inside the enclosing node's span with
|
||||
`` `\bval lg = (new )?Logger` `` / `` `\blg: Logger` `` regexes. **Walker
|
||||
obligations: accurate node startLine/endLine spans + language tag.**
|
||||
- **JVM name-family** (name-matcher.ts:141: scala → 'jvm') — scala refs can
|
||||
bind to java/kotlin nodes and vice versa; nothing extra to emit.
|
||||
- **NO synthesis passes are scala-gated**: callback-synthesizer has zero
|
||||
scala mentions (CC_LANGUAGES = swift/kotlin; expect/actual kotlin-only;
|
||||
rnCrossPlatform NATIVE set has no scala). Generic passes (interface-impl
|
||||
Phase 5.5 over extends refs + method names, function_ref resolution)
|
||||
consume the standard tables.
|
||||
|
||||
## Parity mechanics (all have bitten before)
|
||||
|
||||
- **Emission order** per §Misc: file → source-order walk (per construct:
|
||||
node + contains → extends refs BEFORE member emissions → extractor-order
|
||||
refs) → function_ref refs → value-ref EDGES last.
|
||||
- **generateNodeId inputs**: (filePath, kind, name, startRow+1) — name keeps
|
||||
backticks (`` `Weird Name` ``) and operator glyphs (`+`, `::`); import
|
||||
nodes are named the FIRST path segment; enum_member line/column = the CASE
|
||||
node's (which for simple cases equals the identifier); companion pairs
|
||||
collide on (kind,name) but differ on line. **`node_ids` dedupe-vec pattern
|
||||
still required** — same-(kind,name,line) CAN collide (one-line
|
||||
`class X { … }; object X` constructions, secondary-ctor `this` twins).
|
||||
- **First-match-wins field lookups over named+anonymous children** — import
|
||||
`path`, def `parameters`, extension/for `body`/`enumerators` (§config).
|
||||
Implement `child_by_field` exactly once, tree-sitter-faithful.
|
||||
- **UTF-16 columns/slices** (textutil::col16/slice_utf16) — pinned via
|
||||
`uni.scala` (surrogate-pair emoji counts 2).
|
||||
- **CRLF**: variants of every fixture derived in-memory (kernel-tsjs-parity
|
||||
pattern); the docs fixture MUST include a multi-line `/** … */` (the `\r`
|
||||
retention pin) and a Scala-3 indented file (scanner CRLF).
|
||||
- **Defer policy**: `has_error()` → `defer:` including phantoms; sweep
|
||||
defaults `--max-deferral 0.1` for os-lib/cats-style repos, **0.3 for
|
||||
scala3-style** (§arch-6).
|
||||
|
||||
## Gates (per plan §5 — NOTE: no grammar-bump gate, §Grammar prep)
|
||||
|
||||
- **Kernel C vendor + build.rs + langs.rs + grammar-parity row** land first
|
||||
(inert until the walker exists): `GRAMMAR_LANGUAGES += 'scala'`
|
||||
(kernel-grammar-parity.test.ts:39) must show ABI 15 / 26650 states /
|
||||
32 fields / identical kind+field tables vs the production wasm.
|
||||
- **Torture fixtures** per `## Fixtures to build`, exercised by a new
|
||||
`__tests__/kernel-scala-parity.test.ts`.
|
||||
- **Parity sweeps** (`scripts/kernel-parity.mjs <dir>`, order-sensitive
|
||||
full-object):
|
||||
- `…/scratchpad/svy-scala/gate-repos/os-lib` (small, 59 files, expect 0
|
||||
deferrals)
|
||||
- `…/gate-repos/cats` (medium, 835 files, expect ~15 deferrals ≈1.8%)
|
||||
- `…/gate-repos/scala3` — sweep `compiler/src` (577 files, ~57 deferrals
|
||||
≈10%, `--max-deferral 0.3`) and `library/src` (652 files, ~116 ≈18%,
|
||||
the phantom-heavy set) rather than the whole repo (tests/ is
|
||||
deliberately-invalid input — fine for a robustness pass, useless as a
|
||||
parity denominator).
|
||||
(cloned fresh at survey; re-clone public OSS if gone — agent-eval policy.)
|
||||
Expect 0-diff on every NON-deferred file. Then **full-init dump-diffs
|
||||
byte-identical** (kernel arm vs `CODEGRAPH_KERNEL=0`, `dump-graph.mjs`,
|
||||
cmp) on all three.
|
||||
- **Suite**: kernel-scala-parity torture + CRLF variants + `.sc` fixture +
|
||||
defer fixtures (§Fixtures 5-6) ×2 green with `CODEGRAPH_KERNEL_EXPECT=1`.
|
||||
- **`DEFAULT_ROUTED += 'scala'`** (kernel/index.ts:37 list) only after ALL of
|
||||
the above; changelog rides the existing kernel entry.
|
||||
- Post-route perf sanity: gate repos ride the raw path; a Play-detected repo
|
||||
(e.g. playframework/playframework itself) is the decoded-path smoke check.
|
||||
|
||||
## Fixtures to build
|
||||
|
||||
1. `__tests__/fixtures/kernel-parity/torture.scala` — seed from
|
||||
`svy-scala/torture.scala` (its `extract-torture.txt` is the expected-output
|
||||
pin: 80 nodes / 82 edges / 75 refs). Inventory: package header (ignored, no
|
||||
namespace); imports ×5 (first-segment names; selectors/wildcards/renames);
|
||||
scaladoc + `//`-run docstrings; top-level defs (function kind) incl.
|
||||
curried (first-list sig), generic (type-params-first sig + bound refs),
|
||||
inferred/Unit/qualified/generic-leak returns, private, `@main`/
|
||||
`@deprecated(args)` decorates; top-level val/var/lazy/tuple/pattern
|
||||
destructuring (first-identifier names) + ascribed types (refs) +
|
||||
initializer-invisibility; bodied class with primary ctor (params
|
||||
invisible), fields (val/var/private/annotated), secondary ctor (`this`
|
||||
method + `this` call), methods incl. operator `+`, member calls, extends
|
||||
with-chain; companion object (second class:WidgetS + shared QN space +
|
||||
DEFAULT_SIZE value-ref target read by both create/readDefault); case
|
||||
class bodiless-header asymmetry (`mkY()` call from class) vs bodied twin;
|
||||
case object extends; abstract class + function_declaration; trait with
|
||||
default method + val field; sealed trait + object/class extends; enum
|
||||
(simple/param/extends cases at case-node positions, invisible tails,
|
||||
method); enum with class_parameters; given → instantiates + leaked
|
||||
function; anonymous given; extension (first-def call leak, second-def
|
||||
invisibility) + single-def extension; implicit class; type aliases (no
|
||||
value refs) + opaque; call shapes: bare, apply-sugar `WidgetS(1)`,
|
||||
member, chained cap re-encode + inner, lower chain, 2-hop, this/super
|
||||
bare, literal receivers (nothing), `genericCall[Int]` bracket callee,
|
||||
curried-call raw inner, `new` in expression body, anon `new … {}` in body
|
||||
(instantiates + leaked-nothing), interpolation call + `$id`, eta + bare +
|
||||
named-arg fn-refs, infix invisibility, match/for; static reads
|
||||
(`Registry.count`, `Http.Ok`, `com.example.Fq.X` nothing, write LHS
|
||||
EMITS, callee skip); local defs (innerFn invisible, LocalClass/LocalObj
|
||||
full); package object (flattened); UTF-16 line. Keep it parse-clean
|
||||
(avoid `given … with {}`, `end` identifiers, curried+generic super-args).
|
||||
2. **CRLF variants** in-memory (kernel-tsjs-parity pattern) — plus the docs
|
||||
fixture's multi-line scaladoc (the `\r` pin) and an indented Scala-3
|
||||
fixture (external-scanner CRLF — `indent.scala` seed, extraction
|
||||
byte-identical pinned).
|
||||
3. **`.sc` fixture** — top-level statements → calls from FILE; file-parent
|
||||
value-ref target (`script.sc` seed).
|
||||
4. **given/extension/anon-class fixture** — `given2.scala` +
|
||||
`ext-probe.scala` seeds (the leak asymmetries are the port's likeliest
|
||||
regression site).
|
||||
5. **Defer fixture #1: PHANTOM** — `def f(x: List[Int]^): Int = 1` — kernel
|
||||
defers on has_error() despite a complete, ERROR-free CST; wasm output
|
||||
byte-normal.
|
||||
6. **Defer fixture #2: real error** — a `given x: C with { … }` or
|
||||
`end`-identifier file — kernel defers; wasm-served output pinned.
|
||||
7. **vref fixture** — `vref.scala` seed: object + top-level targets, shadow
|
||||
prune, `$X`/`${X}` interpolation reads, lowercase non-target, **the
|
||||
same-name last-wins mis-target** (Config/Config2 TIMEOUT_MS →
|
||||
Config2's node).
|
||||
8. **fnref fixture** — `fnref.scala`/`fnref2.scala` seeds: all five capture
|
||||
channels + var-init non-capture + dedupe + `register(missing)` gate drop.
|
||||
|
||||
## Probe artifacts (session scratchpad `svy-scala/`)
|
||||
|
||||
`cst-probe.cjs` + `snippets.json` + `cst-snippets.txt` (20 snippet CST dumps
|
||||
WITH per-node childForFieldName truth tables), `brace-field.out` (the
|
||||
anonymous-token field proof: extension `{`-body, for `enumerators`),
|
||||
`extract-probe.cjs` (runs the REAL dist extractor) + ground-truth dumps
|
||||
`extract-{torture,torture-crlf,docs,docs-crlf,vref,fnref,fnref2,misc,indent,
|
||||
indent-crlf,script,given2,edge2,ext,super2,imp2,uni}.txt` (pinned outputs
|
||||
cited throughout; double as walker test expectations), `vref2.out`
|
||||
(value-ref target identity), `crlf-cst.cjs` (LF/CRLF s-exp equality ×7),
|
||||
`error-sweep.cjs` + `errors-{os-lib,cats,scala3,scala3-compiler,
|
||||
scala3-library,scala3-tests}.txt`, `err-sample.cjs` + `err-samples.out`
|
||||
(error-class contexts), `errvariants.out` (17 minimal shapes),
|
||||
`phantom-scan.cjs` + `phantom-scan-{lib,cats}.out` + `phantom-min.out` (the
|
||||
`^`-postfix one-liner), fixtures (`torture.scala`, `docs.scala`,
|
||||
`vref.scala`, `fnref.scala`, `fnref2.scala`, `misc.scala`, `given2.scala`,
|
||||
`edge2.scala`, `ext-probe.scala`, `indent.scala`, `uni.scala`,
|
||||
`script.sc`, …), `tsscala-clone/` (the `0aca5d0a6f` checkout) +
|
||||
`grammar-shas.txt`, `gate-repos/{os-lib,cats,scala3}`. Scratch is
|
||||
throwaway — re-derive from this doc if gone.
|
||||
@@ -48,7 +48,7 @@ if (paths.length === 0) {
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau']);
|
||||
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau', 'scala']);
|
||||
const EXTS = new Map([
|
||||
['.ts', 'typescript'], ['.mts', 'typescript'], ['.cts', 'typescript'],
|
||||
['.tsx', 'tsx'], ['.js', 'javascript'], ['.mjs', 'javascript'],
|
||||
@@ -66,6 +66,7 @@ const EXTS = new Map([
|
||||
['.kt', 'kotlin'], ['.kts', 'kotlin'], // R7b
|
||||
['.r', 'r'], // R7b batch 4 — real-world casing is `.R`; extname lowercased below
|
||||
['.lua', 'lua'], ['.luau', 'luau'], // R7b batch 4
|
||||
['.scala', 'scala'], ['.sc', 'scala'], // R7b batch 4
|
||||
]);
|
||||
|
||||
/** Collect candidate files. */
|
||||
|
||||
@@ -94,6 +94,15 @@ const DEFAULT_ROUTED: ReadonlySet<Language> = new Set<Language>([
|
||||
// packs, default type params) — a JUMP past ~10% is the bug signal.
|
||||
'lua',
|
||||
'luau',
|
||||
// R7b batch 4 (2026-07-20): parity swept 0-diff on os-lib/cats + scala3
|
||||
// compiler/src + library/src + full-init dump-diffs byte-identical.
|
||||
// Error incidence is bimodal: mainstream Scala-2-style repos ~0-2%
|
||||
// (os-lib 0%, cats 1.8%) but scala-3-frontier code runs 10-18% with
|
||||
// PHANTOM-dominated error sets (hasError=true, zero ERROR nodes —
|
||||
// capture-checking `^` types; defer on the FLAG). Sweeps over
|
||||
// scala3-style repos use --max-deferral 0.3 (swift precedent); a
|
||||
// deferral JUMP on cats/os-lib-style code is the bug signal.
|
||||
'scala',
|
||||
]);
|
||||
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user