feat(kernel): R7b Kotlin walker — kotlin module, vendored-grammar-C build, kotlin default-routed (#1382)
Sixth R7b port — the T1½ batch finale. Checklist-first recipe
(docs/design/kotlin-kernel-port-checklist.md, 1,121 lines, dist-extractor
ground truth); parity passed FIRST RUN on all three repos.
THE NOVEL MECHANISM — vendored-grammar-C (the §4 tracker's prescription,
first use): the crates.io tree-sitter-kotlin 0.3.8 pins `tree-sitter >= 0.21,
< 0.23` (the kernel links 0.25) and tree-sitter-kotlin-ng is a DIFFERENT
grammar (8 fields vs 0, renamed kinds — extractor-breaking), so no crate dep
is possible. The fwcd 0.3.8 tag's sha-matched parser.c + scanner.c are
vendored into codegraph-kernel/grammars/kotlin and compiled by build.rs (cc),
exposed via tree-sitter-language::LanguageFn. The wasm re-vendor is
behavior-NEUTRAL (0 CST/error disagreements across 1,984 gate-repo files;
old-vs-new full-init dumps byte-identical ×3) — a reproducibility re-vendor,
ABI stays 14.
Walker firsts: extension-function receivers (getReceiverType →
`WidgetK::extend` QN OVERRIDE with no package prefix, the qualified-receiver
`com::qext` first-segment bug, and the owner-contains fallback that excludes
`interface` kinds and is source-order dependent) and extractModifiers
(expect/actual platform modifiers → the node DECORATORS wire field on every
created node — the KMP synthesizer's feed, incl. `actual typealias`).
Preserved bug-for-bug: the FIELD_COUNT-0 dead cluster (no signatures, ZERO
type-annotation refs), hook-consumed property initializers emitting nothing
(incl. `by lazy {}`), the bodiless-vs-bodied class header asymmetry, enum-
entry bodies being invisible, KDoc never a docstring AND chain-breaking,
comment-gluing into import/package extents, `@Anno(args)` emitting nothing
while `@Marker` decorates, zero instantiates refs, the paren-then-lambda
`trailing()` garbage callee, text-includes visibility/suspend false
positives, and the packaged-file value-ref target drop. The fun-interface
misparse-recovery hook is DEFER-SHIELDED (every such file has_error) and
deliberately not ported. The swift-sweep lesson pre-applied: the shared
`assignment` shadow-prune case is implemented alongside the
property_declaration case.
Gates: sweeps 0-diff okio 299/322, okhttp 531/580, kotlinx.coroutines
1031/1082 (deferrals exactly the predicted 23/49/51 — both-arm grammar
reality incl. PHANTOM hasError files with complete CSTs; the kernel trusts
the flag); full-init dumps byte-identical ×3 (46.5k/108.9k/92.3k lines); KMP
expect/actual synthesis IDENTICAL across arms (412 edges on
kotlinx.coroutines — the tracker's KMP validation); kernel-kotlin-parity
suite (torture reflowed off the phantom shapes + .kts script + CRLF variants
+ fun-interface and phantom defer pins) + kotlin grammar-parity row (the
C-build ↔ wasm table identity proof); full suite 2,633 green ×2 under
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += kotlin (15 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Fable 5
Claude Fable 5
parent
09e301bbfa
commit
45a53eb5b5
+1
-1
@@ -11,7 +11,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
### New Features
|
||||
|
||||
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, C++, Rust, C#, Ruby, PHP, and Swift projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, django-, git-, protobuf-, tokio-, rust-analyzer-, jellyfin-, rails-, symfony-, and swift-nio-scale codebases (Lombok-generated members, C function-pointer tables, and Unreal-Engine-style macro-heavy headers included; CUDA and Metal sources ride the C++ path). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
|
||||
- Indexing TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, C++, Rust, C#, Ruby, PHP, Swift, and Kotlin projects is faster: parsing and symbol extraction now run in a native engine when a prebuilt binary is available for your platform (release bundles include one), producing exactly the same graph — verified byte-for-byte against the previous engine on real repositories, from small libraries up to vscode-, dubbo-, django-, git-, protobuf-, tokio-, rust-analyzer-, jellyfin-, rails-, symfony-, swift-nio-, and kotlinx.coroutines-scale codebases (Lombok-generated members, C function-pointer tables, and Unreal-Engine-style macro-heavy headers included; CUDA and Metal sources ride the C++ path). The speedup is largest on resource-constrained machines like CI runners. No setup needed: platforms without the native binary, and individual files with syntax errors, automatically use the previous engine, and `CODEGRAPH_KERNEL=0` turns the native path off entirely.
|
||||
- Reference resolution now runs in parallel on large projects. When a project has enough pending references to make it worthwhile (roughly 150k+, typical for big Java/Kotlin/Spring codebases), resolution fans out across worker threads while results are applied in the exact order the single-threaded path would have used — the graph comes out byte-for-byte identical, about twice as fast end-to-end on a 4,000-file Java project in our testing. Small projects keep the single-threaded path automatically (the fan-out costs more than it saves there). Set `CODEGRAPH_NO_PARALLEL_RESOLVE=1` to disable, or `CODEGRAPH_PARALLEL_RESOLVE_MIN=<count>` to tune when it engages.
|
||||
- Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle.
|
||||
- The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
val greeting = "hi"
|
||||
println(greeting)
|
||||
fun scripted() { work() }
|
||||
scripted()
|
||||
@@ -0,0 +1,267 @@
|
||||
/** KDoc for the file's package. */
|
||||
package com.example.torture
|
||||
|
||||
import com.example.other.OtherClass
|
||||
import com.example.util.helper
|
||||
import com.example.wild.*
|
||||
import com.example.alias.LongName as Short
|
||||
|
||||
// line comment run 1
|
||||
// line comment run 2
|
||||
fun topLevel(x: Int, s: String): WidgetK {
|
||||
val local = x + 1
|
||||
return WidgetK(local)
|
||||
}
|
||||
|
||||
/** KDoc on extension fn. */
|
||||
fun WidgetK.extend(n: Int): Int {
|
||||
render()
|
||||
return n
|
||||
}
|
||||
|
||||
fun <T> List<T>.genericExt(): T = first()
|
||||
|
||||
fun com.example.Qualified.qext() {}
|
||||
|
||||
suspend fun suspender(): Unit { helper() }
|
||||
|
||||
private internal fun visFn() {}
|
||||
|
||||
fun inferred() = helper()
|
||||
|
||||
fun nullableRet(): WidgetK? = null
|
||||
|
||||
fun lambdaRet(): (Int) -> Unit = { }
|
||||
|
||||
expect fun platformThing(): Int
|
||||
|
||||
actual fun actualThing(): Int = 1
|
||||
|
||||
tailrec fun tailer(n: Int): Int = if (n <= 0) 0 else tailer(n - 1)
|
||||
|
||||
infix fun Int.pow(e: Int): Int = this
|
||||
|
||||
operator fun WidgetK.plus(o: WidgetK): WidgetK = this
|
||||
|
||||
val topVal: Int = 3
|
||||
var topVar = "s"
|
||||
const val TOP_CONST = 99
|
||||
val topDelegated by lazy { WidgetK(1) }
|
||||
val (destA, destB) = makePair()
|
||||
val withGetter: Int
|
||||
get() = 42
|
||||
|
||||
class WidgetK(val size: Int, private var name: String = defaultName()) {
|
||||
val area: Int = size * size
|
||||
var label: String? = null
|
||||
val computed: Int
|
||||
get() = size * 2
|
||||
|
||||
init {
|
||||
val initLocal = 5
|
||||
register(initLocal)
|
||||
}
|
||||
|
||||
constructor(s: String) : this(s.length) {
|
||||
log(s)
|
||||
}
|
||||
|
||||
fun render(): Unit {
|
||||
draw(size)
|
||||
}
|
||||
|
||||
fun chainInner(): WidgetK = this
|
||||
|
||||
companion object {
|
||||
val SHARED = WidgetK(0)
|
||||
const val COMPANION_CONST = 7
|
||||
fun create(): WidgetK = WidgetK(1)
|
||||
}
|
||||
|
||||
companion object Named { }
|
||||
}
|
||||
|
||||
data class DataK(val a: Int, val b: String)
|
||||
|
||||
abstract class AbstractK {
|
||||
abstract fun impl(): Int
|
||||
}
|
||||
|
||||
open class OpenBase(n: Int) {
|
||||
open fun over() {}
|
||||
}
|
||||
|
||||
class SubK(n: Int) : OpenBase(n), Drawable, Comparable<SubK> {
|
||||
override fun over() {}
|
||||
override fun compareTo(other: SubK): Int = 0
|
||||
override fun draw() {}
|
||||
}
|
||||
|
||||
class QualifiedSuper : com.example.deep.RemoteBase() { }
|
||||
|
||||
class DelegatedImpl(d: Drawable) : Drawable by d
|
||||
|
||||
interface Drawable {
|
||||
fun draw()
|
||||
fun outline(): Int = 1
|
||||
val prop: Int get() = 2
|
||||
}
|
||||
|
||||
sealed class SealedOp {
|
||||
object Add : SealedOp()
|
||||
data class Mul(val f: Int) : SealedOp()
|
||||
}
|
||||
|
||||
sealed interface SealedIface
|
||||
|
||||
enum class Color {
|
||||
RED, GREEN, BLUE
|
||||
}
|
||||
|
||||
enum class Http(val code: Int) {
|
||||
OK(200) {
|
||||
override fun label(): String = "ok"
|
||||
},
|
||||
ERR(500) {
|
||||
override fun label(): String = "err"
|
||||
};
|
||||
|
||||
abstract fun label(): String
|
||||
fun common(): Int = code
|
||||
companion object {
|
||||
fun of(c: Int): Http = OK
|
||||
}
|
||||
}
|
||||
|
||||
object Registry {
|
||||
val instances = mutableListOf<WidgetK>()
|
||||
var count = 0
|
||||
const val REG_CONST = 1
|
||||
fun register(w: WidgetK) { instances.add(w) }
|
||||
}
|
||||
|
||||
annotation class MyMarker(val why: String = "")
|
||||
|
||||
@MyMarker
|
||||
class Annotated {
|
||||
@JvmStatic
|
||||
fun jvmStatic() {}
|
||||
|
||||
@Deprecated("gone", ReplaceWith("new"))
|
||||
fun old() {}
|
||||
|
||||
@field:JvmField
|
||||
val fielded: Int = 1
|
||||
|
||||
@get:MyMarker
|
||||
val got: Int = 2
|
||||
}
|
||||
|
||||
typealias Handler = (Int) -> Unit
|
||||
typealias WidgetList = List<WidgetK>
|
||||
|
||||
expect class PlatformFile {
|
||||
fun path(): String
|
||||
}
|
||||
|
||||
actual class ActualFile {
|
||||
actual fun path(): String = "/"
|
||||
}
|
||||
|
||||
actual typealias PlatformClock = java.time.Clock
|
||||
|
||||
fun caller() {
|
||||
val w = WidgetK(1)
|
||||
w.render()
|
||||
this.toString()
|
||||
super.hashCode()
|
||||
Registry.register(w)
|
||||
Registry.count
|
||||
Color.RED
|
||||
com.example.Fq.CONST_READ
|
||||
WidgetK.create().render()
|
||||
Foo.getInstance().bar()
|
||||
lowerFactory().chain()
|
||||
w.chainInner().render()
|
||||
"literal".uppercase()
|
||||
5.toString()
|
||||
listOf(1, 2).size
|
||||
w.label?.length
|
||||
w.label!!.length
|
||||
helper()
|
||||
Short.static()
|
||||
val fn: Handler = { i -> println(i) }
|
||||
fn(3)
|
||||
(fn)(4)
|
||||
run { helper() }
|
||||
listOf(1).forEach { it + 1 }
|
||||
w.let { it.render() }
|
||||
generic<Int>(1)
|
||||
register(::topLevel)
|
||||
register(OtherClass::handle)
|
||||
register(w::render)
|
||||
register(this::caller)
|
||||
obtain(String::class)
|
||||
val m = ::caller
|
||||
val bound = w::render
|
||||
val s = "interp $topVal and ${w.render()} end"
|
||||
val multi = """raw $topVal"""
|
||||
when (w.size) {
|
||||
1 -> helper()
|
||||
else -> draw(0)
|
||||
}
|
||||
if (topVal > 1) { helper() }
|
||||
for (i in 1..3) { draw(i) }
|
||||
fun localFn(): Int = 5
|
||||
localFn()
|
||||
class LocalClass {
|
||||
fun lm() {}
|
||||
}
|
||||
object LocalObj {
|
||||
fun om() {}
|
||||
}
|
||||
val anon = object : Drawable {
|
||||
override fun draw() { helper() }
|
||||
}
|
||||
anon.draw()
|
||||
label@ for (i in 1..2) { break@label }
|
||||
val backtick = `weird name`()
|
||||
}
|
||||
|
||||
fun `weird name`(): Int = 1
|
||||
|
||||
fun trailing(block: (Int) -> Int): Int = block(1)
|
||||
|
||||
fun useTrailing() {
|
||||
trailing { it * 2 }
|
||||
trailing() { it * 3 }
|
||||
}
|
||||
|
||||
fun defaults(a: Int = compute(), b: String = "x") {}
|
||||
|
||||
fun varargFn(vararg xs: Int) {}
|
||||
|
||||
fun destructuringBody(p: Pair<Int, Int>) {
|
||||
val (x, y) = p
|
||||
draw(x + y)
|
||||
}
|
||||
|
||||
fun assignRefs() {
|
||||
Registry.count = 5
|
||||
Registry.count += 1
|
||||
}
|
||||
|
||||
fun nullish(x: WidgetK?) {
|
||||
x?.render()
|
||||
val l = x ?: WidgetK(0)
|
||||
}
|
||||
|
||||
fun stringsEdge() {
|
||||
val a = "quote \" and dollar ${'$'} done"
|
||||
}
|
||||
|
||||
fun labeledLambda() {
|
||||
listOf(1).forEach loop@{ if (it == 0) return@loop }
|
||||
}
|
||||
|
||||
fun whereClause(): Int where Int : Comparable<Int> = 1
|
||||
@@ -36,7 +36,7 @@ const kernelBuilt = fs.existsSync(KERNEL_PATH);
|
||||
|
||||
// Every kernel-capable language. `jsx` shares the javascript grammar on BOTH
|
||||
// paths (langs.rs mirrors WASM_GRAMMAR_FILES), so the distinct grammars are:
|
||||
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift'];
|
||||
const GRAMMAR_LANGUAGES: Language[] = ['typescript', 'tsx', 'javascript', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin'];
|
||||
|
||||
describe.skipIf(!kernelBuilt)('kernel↔wasm grammar parity', () => {
|
||||
beforeAll(async () => {
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* Kernel↔wasm Kotlin extraction parity (R7b of the kernel migration).
|
||||
*
|
||||
* Asserts the native walker (codegraph-kernel/src/kotlin.rs — grammar
|
||||
* compiled from the vendored fwcd 0.3.8 C sources, the arc's first
|
||||
* vendored-grammar-C language) produces the SAME ExtractionResult as the
|
||||
* wasm TreeSitterExtractor over the checked-in torture fixture (torture.kt:
|
||||
* the property hook's scope classification, extension-function receiver QNs
|
||||
* (`WidgetK::extend`, the qualified `com::qext` bug) + the owner-contains
|
||||
* fallback, expect/actual → node DECORATORS (the KMP synthesizer feed),
|
||||
* the bodiless-vs-bodied class header asymmetry, comment-glued
|
||||
* import/package extents, KDoc dropped-and-chain-breaking docstrings,
|
||||
* `@Marker` decorates vs `@Anno(args)` nothing, zero type-annotation refs,
|
||||
* zero instantiates, the #750 capitalized-chain re-encode, paren-then-
|
||||
* lambda garbage callees, `${X}`-reads-vs-`$X`-non-reads value refs and the
|
||||
* packaged-file target drop) plus a `.kts` script fixture (file-attributed
|
||||
* top-level calls), with in-memory CRLF variants (#1329), and two defer
|
||||
* fixtures — a `fun interface` file and a PHANTOM error (a one-line class
|
||||
* body sets hasError with a complete, ERROR-node-free CST; the kernel
|
||||
* trusts the flag).
|
||||
*
|
||||
* The full-repo sweep lives in scripts/kernel-parity.mjs (okio/okhttp/
|
||||
* kotlinx.coroutines — expected deferrals 23/49/51, grammar-inherent).
|
||||
* Skips when no kernel binary is staged; CODEGRAPH_KERNEL_EXPECT=1 turns
|
||||
* that into a failure (kernel-scaffold.test.ts).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { extractFromSource } from '../src/extraction';
|
||||
import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars';
|
||||
import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel';
|
||||
import type { ExtractionResult } from '../src/types';
|
||||
|
||||
const KERNEL_PATH = path.join(
|
||||
__dirname,
|
||||
'..',
|
||||
'codegraph-kernel',
|
||||
'prebuilds',
|
||||
`${process.platform}-${process.arch}`,
|
||||
'codegraph-kernel.node'
|
||||
);
|
||||
const kernelBuilt = fs.existsSync(KERNEL_PATH);
|
||||
|
||||
const FIXTURE_DIR = path.join(__dirname, 'fixtures', 'kernel-parity');
|
||||
|
||||
function canon(result: ExtractionResult): { nodes: string[]; edges: string[]; refs: string[] } {
|
||||
return {
|
||||
nodes: result.nodes
|
||||
.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort()))
|
||||
.sort(),
|
||||
edges: result.edges.map((e) => JSON.stringify(e, Object.keys(e).sort())).sort(),
|
||||
refs: result.unresolvedReferences
|
||||
.map((r) => JSON.stringify(r, Object.keys(r).sort()))
|
||||
.sort(),
|
||||
};
|
||||
}
|
||||
|
||||
const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const;
|
||||
let savedEnv: Record<string, string | undefined>;
|
||||
|
||||
describe.skipIf(!kernelBuilt)('kernel Kotlin extraction parity', () => {
|
||||
beforeAll(async () => {
|
||||
await initGrammars();
|
||||
await loadGrammarsForLanguages(['kotlin']);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]]));
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
for (const k of ENV_KEYS) {
|
||||
if (savedEnv[k] === undefined) delete process.env[k];
|
||||
else process.env[k] = savedEnv[k];
|
||||
}
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
function assertParity(filePath: string, source: string, minNodes = 3): void {
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
const viaKernel = tryKernelExtract(filePath, source, 'kotlin');
|
||||
expect(viaKernel, `kernel extraction failed for ${filePath}`).not.toBeNull();
|
||||
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource(filePath, source, 'kotlin');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
|
||||
const k = canon(viaKernel!);
|
||||
const w = canon(viaWasm);
|
||||
expect(k.nodes, `${filePath}: nodes`).toEqual(w.nodes);
|
||||
expect(k.edges, `${filePath}: edges`).toEqual(w.edges);
|
||||
expect(k.refs, `${filePath}: refs`).toEqual(w.refs);
|
||||
expect(viaWasm.nodes.length).toBeGreaterThanOrEqual(minNodes);
|
||||
}
|
||||
|
||||
const FIXTURES: Array<{ file: string; minNodes: number }> = [
|
||||
{ file: 'torture.kt', minNodes: 40 },
|
||||
{ file: 'TortureScript.kts', minNodes: 2 },
|
||||
];
|
||||
|
||||
for (const { file, minNodes } of FIXTURES) {
|
||||
it(`${file}: hook properties, receivers, decorators, calls, value refs`, () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, file), 'utf8');
|
||||
assertParity(`fixtures/${file}`, src, minNodes);
|
||||
});
|
||||
|
||||
it(`${file} CRLF parity`, () => {
|
||||
const src = fs.readFileSync(path.join(FIXTURE_DIR, file), 'utf8');
|
||||
const crlf = src.replace(/(?<!\r)\n/g, '\r\n');
|
||||
assertParity(`fixtures/${file} (crlf)`, crlf, minNodes);
|
||||
});
|
||||
}
|
||||
|
||||
it('fun-interface files defer to the wasm extractor (grammar-inherent error)', () => {
|
||||
const src = 'package p\n\nfun interface Transformer {\n fun transform(x: Int): Int\n}\n\nfun after() { work() }\n';
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(tryKernelExtract('src/FunIface.kt', src, 'kotlin')).toBeNull();
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource('src/FunIface.kt', src, 'kotlin');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
// The wasm arm's misparse-recovery hook still mints the interface node.
|
||||
expect(viaWasm.nodes.some((n) => n.kind === 'interface' && n.name === 'Transformer')).toBe(true);
|
||||
});
|
||||
|
||||
it('PHANTOM errors defer too — hasError with a complete, ERROR-node-free CST', () => {
|
||||
const src = 'abstract class A { abstract fun i(): Int }\n';
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(tryKernelExtract('src/Phantom.kt', src, 'kotlin')).toBeNull();
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
const viaWasm = extractFromSource('src/Phantom.kt', src, 'kotlin');
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
expect(viaWasm.nodes.some((n) => n.kind === 'class' && n.name === 'A')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -73,11 +73,11 @@ describe.skipIf(!kernelBuilt)('kernel scaffold', () => {
|
||||
});
|
||||
|
||||
it('TS/JS family + Java + Python + Go route to the kernel by default; others stay wasm', () => {
|
||||
for (const lang of ['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'ruby', 'php'] as const) {
|
||||
for (const lang of ['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'ruby', 'php', 'swift', 'kotlin'] as const) {
|
||||
expect(kernelRoutes(lang), lang).toBe(true);
|
||||
}
|
||||
expect(kernelRoutes('kotlin')).toBe(false);
|
||||
expect(tryKernelExtract('src/a.kt', 'fun f() {}\n', 'kotlin')).toBeNull();
|
||||
expect(kernelRoutes('scala')).toBe(false);
|
||||
expect(tryKernelExtract('src/a.scala', 'object A { def f(): Int = 1 }\n', 'scala')).toBeNull();
|
||||
// CODEGRAPH_KERNEL_LANGS REPLACES the default set when present.
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'tsx';
|
||||
expect(kernelRoutes('typescript')).toBe(false);
|
||||
|
||||
Generated
+2
@@ -46,6 +46,7 @@ checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
|
||||
name = "codegraph-kernel"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"napi",
|
||||
"napi-build",
|
||||
"napi-derive",
|
||||
@@ -58,6 +59,7 @@ dependencies = [
|
||||
"tree-sitter-go",
|
||||
"tree-sitter-java",
|
||||
"tree-sitter-javascript",
|
||||
"tree-sitter-language",
|
||||
"tree-sitter-php",
|
||||
"tree-sitter-python",
|
||||
"tree-sitter-ruby",
|
||||
|
||||
@@ -47,8 +47,13 @@ tree-sitter-php = "=0.24.2"
|
||||
# ~20MB generated — expect slow compiles.
|
||||
tree-sitter-swift = "=0.7.3"
|
||||
|
||||
# tree-sitter-language: the version-agnostic LanguageFn shim for the vendored
|
||||
# kotlin grammar C (see build.rs — no kotlin crate dep is possible).
|
||||
tree-sitter-language = "0.1"
|
||||
|
||||
[build-dependencies]
|
||||
napi-build = "2"
|
||||
cc = "1"
|
||||
|
||||
[profile.release]
|
||||
lto = true
|
||||
|
||||
@@ -1,3 +1,23 @@
|
||||
fn main() {
|
||||
napi_build::setup();
|
||||
|
||||
// Kotlin grammar — vendored C, compiled here instead of a crate dep: the
|
||||
// crates.io tree-sitter-kotlin 0.3.8 pins `tree-sitter >= 0.21, < 0.23`
|
||||
// (the kernel links 0.25) and tree-sitter-kotlin-ng is a DIFFERENT
|
||||
// grammar (8 fields vs 0, renamed kinds — extractor-breaking). Sources
|
||||
// are the fwcd 0.3.8 tag's checked-in generated artifacts, sha-matched
|
||||
// against the crates.io tarball (kotlin checklist §Grammar prep):
|
||||
// parser.c 54104a7ef1555c265b746c790e0f8bb953cc17806e9df0c3af82f7f62c06a70a
|
||||
// scanner.c 27f73337ec357fc341fa57538f34c14277b0346980c3405dc30beab6202ec6d0
|
||||
// Flags crib the tarball's own bindings/rust/build.rs.
|
||||
let mut c = cc::Build::new();
|
||||
c.include("grammars/kotlin");
|
||||
c.file("grammars/kotlin/parser.c");
|
||||
c.file("grammars/kotlin/scanner.c");
|
||||
c.flag_if_supported("-Wno-unused-parameter");
|
||||
c.flag_if_supported("-Wno-unused-but-set-variable");
|
||||
c.flag_if_supported("-Wno-trigraphs");
|
||||
c.flag_if_supported("-utf-8"); // msvc
|
||||
c.compile("tree-sitter-kotlin");
|
||||
println!("cargo:rerun-if-changed=grammars/kotlin");
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,530 @@
|
||||
#include "tree_sitter/array.h"
|
||||
#include "tree_sitter/parser.h"
|
||||
|
||||
#include <string.h>
|
||||
#include <wctype.h>
|
||||
|
||||
// Mostly a copy paste of tree-sitter-javascript/src/scanner.c
|
||||
|
||||
enum TokenType {
|
||||
AUTOMATIC_SEMICOLON,
|
||||
IMPORT_LIST_DELIMITER,
|
||||
SAFE_NAV,
|
||||
MULTILINE_COMMENT,
|
||||
STRING_START,
|
||||
STRING_END,
|
||||
STRING_CONTENT,
|
||||
};
|
||||
|
||||
/* Pretty much all of this code is taken from the Julia tree-sitter
|
||||
parser.
|
||||
|
||||
Julia has similar problems with multiline comments that can be nested,
|
||||
line comments, as well as line and multiline strings.
|
||||
|
||||
The most heavily edited section is `scan_string_content`,
|
||||
particularly with respect to interpolation.
|
||||
*/
|
||||
|
||||
// Block comments are easy to parse, but strings require extra-attention.
|
||||
|
||||
// The main problems that arise when parsing strings are:
|
||||
// 1. Triple quoted strings allow single quotes inside. e.g. """ "foo" """.
|
||||
// 2. Non-standard string literals don't allow interpolations or escape
|
||||
// sequences, but you can always write \" and \`.
|
||||
|
||||
// To efficiently store a delimiter, we take advantage of the fact that:
|
||||
// (int)'"' == 34 && (34 & 1) == 0
|
||||
// i.e. " has an even numeric representation, so we can store a triple
|
||||
// quoted delimiter as (delimiter + 1).
|
||||
|
||||
#define DELIMITER_LENGTH 3
|
||||
|
||||
typedef char Delimiter;
|
||||
|
||||
// We use a stack to keep track of the string delimiters.
|
||||
typedef Array(Delimiter) Stack;
|
||||
|
||||
static inline void stack_push(Stack *stack, char chr, bool triple) {
|
||||
if (stack->size >= TREE_SITTER_SERIALIZATION_BUFFER_SIZE) abort();
|
||||
array_push(stack, (Delimiter)(triple ? (chr + 1) : chr));
|
||||
}
|
||||
|
||||
static inline Delimiter stack_pop(Stack *stack) {
|
||||
if (stack->size == 0) abort();
|
||||
return array_pop(stack);
|
||||
}
|
||||
|
||||
static inline void skip(TSLexer *lexer) { lexer->advance(lexer, true); }
|
||||
|
||||
static inline void advance(TSLexer *lexer) { lexer->advance(lexer, false); }
|
||||
|
||||
// Scanner functions
|
||||
|
||||
static bool scan_string_start(TSLexer *lexer, Stack *stack) {
|
||||
if (lexer->lookahead != '"') return false;
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
for (unsigned count = 1; count < DELIMITER_LENGTH; ++count) {
|
||||
if (lexer->lookahead != '"') {
|
||||
// It's not a triple quoted delimiter.
|
||||
stack_push(stack, '"', false);
|
||||
return true;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
stack_push(stack, '"', true);
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool scan_string_content(TSLexer *lexer, Stack *stack) {
|
||||
if (stack->size == 0) return false; // Stack is empty. We're not in a string.
|
||||
Delimiter end_char = stack->contents[stack->size - 1]; // peek
|
||||
bool is_triple = false;
|
||||
bool has_content = false;
|
||||
if (end_char & 1) {
|
||||
is_triple = true;
|
||||
end_char -= 1;
|
||||
}
|
||||
while (lexer->lookahead) {
|
||||
if (lexer->lookahead == '$') {
|
||||
// if we did not just start reading stuff, then we should stop
|
||||
// lexing right here, so we can offer the opportunity to lex a
|
||||
// interpolated identifier
|
||||
if (has_content) {
|
||||
lexer->result_symbol = STRING_CONTENT;
|
||||
return has_content;
|
||||
}
|
||||
// otherwise, if this is the start, determine if it is an
|
||||
// interpolated identifier.
|
||||
// otherwise, it's just string content, so continue
|
||||
advance(lexer);
|
||||
if (iswalpha(lexer->lookahead) || lexer->lookahead == '{') {
|
||||
// this must be a string interpolation, let's
|
||||
// fail so we parse it as such
|
||||
return false;
|
||||
}
|
||||
lexer->result_symbol = STRING_CONTENT;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
if (lexer->lookahead == '\\') {
|
||||
// if we see a \, then this might possibly escape a dollar sign
|
||||
// in which case, we should not defer to the interpolation
|
||||
advance(lexer);
|
||||
// this dollar sign is escaped, so it must be content.
|
||||
// we consume it here so we don't enter the dollar sign case above,
|
||||
// which leaves the possibility that it is an interpolation
|
||||
if (lexer->lookahead == '$') {
|
||||
advance(lexer);
|
||||
// however this leaves an edgecase where an escaped dollar sign could
|
||||
// appear at the end of a string (e.g "aa\$") which isn't handled
|
||||
// correctly; if we were at the end of the string, terminate properly
|
||||
if (lexer->lookahead == end_char) {
|
||||
stack_pop(stack);
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = STRING_END;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
} else if (lexer->lookahead == end_char) {
|
||||
if (is_triple) {
|
||||
lexer->mark_end(lexer);
|
||||
for (unsigned count = 1; count < DELIMITER_LENGTH; ++count) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != end_char) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = STRING_CONTENT;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/* This is so if we lex something like
|
||||
"""foo"""
|
||||
^
|
||||
where we are at the `f`, we should quit after
|
||||
reading `foo`, and ascribe it to STRING_CONTENT.
|
||||
|
||||
Then, we restart and try to read the end.
|
||||
This is to prevent `foo` from being absorbed into
|
||||
the STRING_END token.
|
||||
*/
|
||||
if (has_content && lexer->lookahead == end_char) {
|
||||
lexer->result_symbol = STRING_CONTENT;
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Since the string internals are all hidden in the syntax
|
||||
tree anyways, there's no point in going to the effort of
|
||||
specifically separating the string end from string contents.
|
||||
If we see a bunch of quotes in a row, then we just go until
|
||||
they stop appearing, then stop lexing and call it the
|
||||
string's end.
|
||||
*/
|
||||
lexer->result_symbol = STRING_END;
|
||||
lexer->mark_end(lexer);
|
||||
while (lexer->lookahead == end_char) {
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
}
|
||||
stack_pop(stack);
|
||||
return true;
|
||||
}
|
||||
if (has_content) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = STRING_CONTENT;
|
||||
return true;
|
||||
}
|
||||
stack_pop(stack);
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = STRING_END;
|
||||
return true;
|
||||
}
|
||||
advance(lexer);
|
||||
has_content = true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
static bool scan_multiline_comment(TSLexer *lexer) {
|
||||
if (lexer->lookahead != '/') return false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '*') return false;
|
||||
advance(lexer);
|
||||
|
||||
bool after_star = false;
|
||||
unsigned nesting_depth = 1;
|
||||
for (;;) {
|
||||
switch (lexer->lookahead) {
|
||||
case '*':
|
||||
advance(lexer);
|
||||
after_star = true;
|
||||
break;
|
||||
case '/':
|
||||
advance(lexer);
|
||||
if (after_star) {
|
||||
after_star = false;
|
||||
nesting_depth -= 1;
|
||||
if (nesting_depth == 0) {
|
||||
lexer->result_symbol = MULTILINE_COMMENT;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
} else {
|
||||
after_star = false;
|
||||
if (lexer->lookahead == '*') {
|
||||
nesting_depth += 1;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
break;
|
||||
case '\0':
|
||||
return false;
|
||||
default:
|
||||
advance(lexer);
|
||||
after_star = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static bool scan_whitespace_and_comments(TSLexer *lexer) {
|
||||
while (iswspace(lexer->lookahead)) skip(lexer);
|
||||
return lexer->lookahead != '/';
|
||||
}
|
||||
|
||||
static bool scan_for_word(TSLexer *lexer, const char* word, unsigned len) {
|
||||
skip(lexer);
|
||||
for (unsigned i = 0; i < len; ++i) {
|
||||
if (lexer->lookahead != word[i]) return false;
|
||||
skip(lexer);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool scan_automatic_semicolon(TSLexer *lexer) {
|
||||
lexer->result_symbol = AUTOMATIC_SEMICOLON;
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
bool sameline = true;
|
||||
for (;;) {
|
||||
if (lexer->eof(lexer)) return true;
|
||||
|
||||
if (lexer->lookahead == ';') {
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!iswspace(lexer->lookahead)) break;
|
||||
|
||||
if (lexer->lookahead == '\n') {
|
||||
skip(lexer);
|
||||
sameline = false;
|
||||
break;
|
||||
}
|
||||
|
||||
if (lexer->lookahead == '\r') {
|
||||
skip(lexer);
|
||||
|
||||
if (lexer->lookahead == '\n') skip(lexer);
|
||||
|
||||
sameline = false;
|
||||
break;
|
||||
}
|
||||
|
||||
skip(lexer);
|
||||
}
|
||||
|
||||
// Skip whitespace and comments
|
||||
if (!scan_whitespace_and_comments(lexer))
|
||||
return false;
|
||||
|
||||
if (sameline) {
|
||||
switch (lexer->lookahead) {
|
||||
// Don't insert a semicolon before an else
|
||||
case 'e':
|
||||
return !scan_for_word(lexer, "lse", 3);
|
||||
|
||||
case 'i':
|
||||
return scan_for_word(lexer, "mport", 5);
|
||||
|
||||
case ';':
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
switch (lexer->lookahead) {
|
||||
case ',':
|
||||
case '.':
|
||||
case ':':
|
||||
case '*':
|
||||
case '%':
|
||||
case '>':
|
||||
case '<':
|
||||
case '=':
|
||||
case '{':
|
||||
case '[':
|
||||
case '(':
|
||||
case '?':
|
||||
case '|':
|
||||
case '&':
|
||||
case '/':
|
||||
return false;
|
||||
|
||||
// Insert a semicolon before `--` and `++`, but not before binary `+` or `-`.
|
||||
// Insert before +/-Float
|
||||
case '+':
|
||||
skip(lexer);
|
||||
if (lexer->lookahead == '+') return true;
|
||||
return iswdigit(lexer->lookahead);
|
||||
|
||||
case '-':
|
||||
skip(lexer);
|
||||
if (lexer->lookahead == '-') return true;
|
||||
return iswdigit(lexer->lookahead);
|
||||
|
||||
// Don't insert a semicolon before `!=`, but do insert one before a unary `!`.
|
||||
case '!':
|
||||
skip(lexer);
|
||||
return lexer->lookahead != '=';
|
||||
|
||||
// Don't insert a semicolon before an else
|
||||
case 'e':
|
||||
return !scan_for_word(lexer, "lse", 3);
|
||||
|
||||
// Don't insert a semicolon before `in` or `instanceof`, but do insert one
|
||||
// before an identifier or an import.
|
||||
case 'i':
|
||||
skip(lexer);
|
||||
if (lexer->lookahead != 'n') return true;
|
||||
skip(lexer);
|
||||
if (!iswalpha(lexer->lookahead)) return false;
|
||||
return !scan_for_word(lexer, "stanceof", 8);
|
||||
|
||||
case ';':
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
|
||||
default:
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
static bool scan_safe_nav(TSLexer *lexer) {
|
||||
lexer->result_symbol = SAFE_NAV;
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
// skip white space
|
||||
if (!scan_whitespace_and_comments(lexer))
|
||||
return false;
|
||||
|
||||
if (lexer->lookahead != '?')
|
||||
return false;
|
||||
|
||||
advance(lexer);
|
||||
|
||||
if (!scan_whitespace_and_comments(lexer))
|
||||
return false;
|
||||
|
||||
if (lexer->lookahead != '.')
|
||||
return false;
|
||||
|
||||
advance(lexer);
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool scan_line_sep(TSLexer *lexer) {
|
||||
// Line Seps: [ CR, LF, CRLF ]
|
||||
int state = 0;
|
||||
while (true) {
|
||||
switch(lexer->lookahead) {
|
||||
case ' ':
|
||||
case '\t':
|
||||
case '\v':
|
||||
// Skip whitespace
|
||||
advance(lexer);
|
||||
break;
|
||||
|
||||
case '\n':
|
||||
advance(lexer);
|
||||
return true;
|
||||
|
||||
case '\r':
|
||||
if (state == 1)
|
||||
return true;
|
||||
|
||||
state = 1;
|
||||
advance(lexer);
|
||||
break;
|
||||
|
||||
default:
|
||||
// We read a CR
|
||||
if (state == 1)
|
||||
return true;
|
||||
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static bool scan_import_list_delimiter(TSLexer *lexer) {
|
||||
// Import lists are terminated either by an empty line or a non import statement
|
||||
lexer->result_symbol = IMPORT_LIST_DELIMITER;
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
// if eof; return true
|
||||
if (lexer->eof(lexer))
|
||||
return true;
|
||||
|
||||
// Scan for the first line seperator
|
||||
if (!scan_line_sep(lexer))
|
||||
return false;
|
||||
|
||||
// if line.sep line.sep; return true
|
||||
if (scan_line_sep(lexer)) {
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
|
||||
// if line.sep [^import]; return true
|
||||
while (true) {
|
||||
switch (lexer->lookahead) {
|
||||
case ' ':
|
||||
case '\t':
|
||||
case '\v':
|
||||
// Skip whitespace
|
||||
advance(lexer);
|
||||
break;
|
||||
|
||||
case 'i':
|
||||
return !scan_for_word(lexer, "mport", 5);
|
||||
|
||||
default:
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
bool tree_sitter_kotlin_external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
|
||||
if (valid_symbols[AUTOMATIC_SEMICOLON]) {
|
||||
bool ret = scan_automatic_semicolon(lexer);
|
||||
if (!ret && valid_symbols[SAFE_NAV] && lexer->lookahead == '?') {
|
||||
return scan_safe_nav(lexer);
|
||||
}
|
||||
|
||||
// if we fail to find an automatic semicolon, it's still possible that we may
|
||||
// want to lex a string or comment later
|
||||
if (ret) return ret;
|
||||
}
|
||||
|
||||
if (valid_symbols[IMPORT_LIST_DELIMITER]) {
|
||||
return scan_import_list_delimiter(lexer);
|
||||
}
|
||||
|
||||
// content or end
|
||||
if (valid_symbols[STRING_CONTENT] && scan_string_content(lexer, payload)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
// a string might follow after some whitespace, so we can't lookahead
|
||||
// until we get rid of it
|
||||
while (iswspace(lexer->lookahead)) skip(lexer);
|
||||
|
||||
if (valid_symbols[STRING_START] && scan_string_start(lexer, payload)) {
|
||||
lexer->result_symbol = STRING_START;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[MULTILINE_COMMENT] && scan_multiline_comment(lexer)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[SAFE_NAV]) {
|
||||
return scan_safe_nav(lexer);
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
void *tree_sitter_kotlin_external_scanner_create() {
|
||||
Stack *stack = ts_calloc(1, sizeof(Stack));
|
||||
if (stack == NULL) abort();
|
||||
array_init(stack);
|
||||
return stack;
|
||||
}
|
||||
|
||||
void tree_sitter_kotlin_external_scanner_destroy(void *payload) {
|
||||
Stack *stack = (Stack *)payload;
|
||||
array_delete(stack);
|
||||
ts_free(stack);
|
||||
}
|
||||
|
||||
unsigned tree_sitter_kotlin_external_scanner_serialize(void *payload, char *buffer) {
|
||||
Stack *stack = (Stack *)payload;
|
||||
memcpy(buffer, stack->contents, stack->size);
|
||||
return stack->size;
|
||||
}
|
||||
|
||||
void tree_sitter_kotlin_external_scanner_deserialize(void *payload, const char *buffer, unsigned length) {
|
||||
Stack *stack = (Stack *)payload;
|
||||
if (length > 0) {
|
||||
array_reserve(stack, length);
|
||||
memcpy(stack->contents, buffer, length);
|
||||
stack->size = length;
|
||||
} else {
|
||||
array_clear(stack);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
#ifndef TREE_SITTER_ALLOC_H_
|
||||
#define TREE_SITTER_ALLOC_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
// Allow clients to override allocation functions
|
||||
#ifdef TREE_SITTER_REUSE_ALLOCATOR
|
||||
|
||||
extern void *(*ts_current_malloc)(size_t);
|
||||
extern void *(*ts_current_calloc)(size_t, size_t);
|
||||
extern void *(*ts_current_realloc)(void *, size_t);
|
||||
extern void (*ts_current_free)(void *);
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc ts_current_malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc ts_current_calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc ts_current_realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free ts_current_free
|
||||
#endif
|
||||
|
||||
#else
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free free
|
||||
#endif
|
||||
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ALLOC_H_
|
||||
@@ -0,0 +1,290 @@
|
||||
#ifndef TREE_SITTER_ARRAY_H_
|
||||
#define TREE_SITTER_ARRAY_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include "./alloc.h"
|
||||
|
||||
#include <assert.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(disable : 4101)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-variable"
|
||||
#endif
|
||||
|
||||
#define Array(T) \
|
||||
struct { \
|
||||
T *contents; \
|
||||
uint32_t size; \
|
||||
uint32_t capacity; \
|
||||
}
|
||||
|
||||
/// Initialize an array.
|
||||
#define array_init(self) \
|
||||
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
|
||||
|
||||
/// Create an empty array.
|
||||
#define array_new() \
|
||||
{ NULL, 0, 0 }
|
||||
|
||||
/// Get a pointer to the element at a given `index` in the array.
|
||||
#define array_get(self, _index) \
|
||||
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
|
||||
|
||||
/// Get a pointer to the first element in the array.
|
||||
#define array_front(self) array_get(self, 0)
|
||||
|
||||
/// Get a pointer to the last element in the array.
|
||||
#define array_back(self) array_get(self, (self)->size - 1)
|
||||
|
||||
/// Clear the array, setting its size to zero. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_clear(self) ((self)->size = 0)
|
||||
|
||||
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
|
||||
/// less than the array's current capacity, this function has no effect.
|
||||
#define array_reserve(self, new_capacity) \
|
||||
_array__reserve((Array *)(self), array_elem_size(self), new_capacity)
|
||||
|
||||
/// Free any memory allocated for this array. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_delete(self) _array__delete((Array *)(self))
|
||||
|
||||
/// Push a new `element` onto the end of the array.
|
||||
#define array_push(self, element) \
|
||||
(_array__grow((Array *)(self), 1, array_elem_size(self)), \
|
||||
(self)->contents[(self)->size++] = (element))
|
||||
|
||||
/// Increase the array's size by `count` elements.
|
||||
/// New elements are zero-initialized.
|
||||
#define array_grow_by(self, count) \
|
||||
do { \
|
||||
if ((count) == 0) break; \
|
||||
_array__grow((Array *)(self), count, array_elem_size(self)); \
|
||||
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
|
||||
(self)->size += (count); \
|
||||
} while (0)
|
||||
|
||||
/// Append all elements from one array to the end of another.
|
||||
#define array_push_all(self, other) \
|
||||
array_extend((self), (other)->size, (other)->contents)
|
||||
|
||||
/// Append `count` elements to the end of the array, reading their values from the
|
||||
/// `contents` pointer.
|
||||
#define array_extend(self, count, contents) \
|
||||
_array__splice( \
|
||||
(Array *)(self), array_elem_size(self), (self)->size, \
|
||||
0, count, contents \
|
||||
)
|
||||
|
||||
/// Remove `old_count` elements from the array starting at the given `index`. At
|
||||
/// the same index, insert `new_count` new elements, reading their values from the
|
||||
/// `new_contents` pointer.
|
||||
#define array_splice(self, _index, old_count, new_count, new_contents) \
|
||||
_array__splice( \
|
||||
(Array *)(self), array_elem_size(self), _index, \
|
||||
old_count, new_count, new_contents \
|
||||
)
|
||||
|
||||
/// Insert one `element` into the array at the given `index`.
|
||||
#define array_insert(self, _index, element) \
|
||||
_array__splice((Array *)(self), array_elem_size(self), _index, 0, 1, &(element))
|
||||
|
||||
/// Remove one element from the array at the given `index`.
|
||||
#define array_erase(self, _index) \
|
||||
_array__erase((Array *)(self), array_elem_size(self), _index)
|
||||
|
||||
/// Pop the last element off the array, returning the element by value.
|
||||
#define array_pop(self) ((self)->contents[--(self)->size])
|
||||
|
||||
/// Assign the contents of one array to another, reallocating if necessary.
|
||||
#define array_assign(self, other) \
|
||||
_array__assign((Array *)(self), (const Array *)(other), array_elem_size(self))
|
||||
|
||||
/// Swap one array with another
|
||||
#define array_swap(self, other) \
|
||||
_array__swap((Array *)(self), (Array *)(other))
|
||||
|
||||
/// Get the size of the array contents
|
||||
#define array_elem_size(self) (sizeof *(self)->contents)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
///
|
||||
/// If an existing element is found to be equal to `needle`, then the `index`
|
||||
/// out-parameter is set to the existing value's index, and the `exists`
|
||||
/// out-parameter is set to true. Otherwise, `index` is set to an index where
|
||||
/// `needle` should be inserted in order to preserve the sorting, and `exists`
|
||||
/// is set to false.
|
||||
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using integer comparisons
|
||||
/// of a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_with`.
|
||||
#define array_search_sorted_by(self, field, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
#define array_insert_sorted_with(self, compare, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using integer comparisons of
|
||||
/// a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_by`.
|
||||
#define array_insert_sorted_by(self, field, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
// Private
|
||||
|
||||
typedef Array(void) Array;
|
||||
|
||||
/// This is not what you're looking for, see `array_delete`.
|
||||
static inline void _array__delete(Array *self) {
|
||||
if (self->contents) {
|
||||
ts_free(self->contents);
|
||||
self->contents = NULL;
|
||||
self->size = 0;
|
||||
self->capacity = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_erase`.
|
||||
static inline void _array__erase(Array *self, size_t element_size,
|
||||
uint32_t index) {
|
||||
assert(index < self->size);
|
||||
char *contents = (char *)self->contents;
|
||||
memmove(contents + index * element_size, contents + (index + 1) * element_size,
|
||||
(self->size - index - 1) * element_size);
|
||||
self->size--;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_reserve`.
|
||||
static inline void _array__reserve(Array *self, size_t element_size, uint32_t new_capacity) {
|
||||
if (new_capacity > self->capacity) {
|
||||
if (self->contents) {
|
||||
self->contents = ts_realloc(self->contents, new_capacity * element_size);
|
||||
} else {
|
||||
self->contents = ts_malloc(new_capacity * element_size);
|
||||
}
|
||||
self->capacity = new_capacity;
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_assign`.
|
||||
static inline void _array__assign(Array *self, const Array *other, size_t element_size) {
|
||||
_array__reserve(self, element_size, other->size);
|
||||
self->size = other->size;
|
||||
memcpy(self->contents, other->contents, self->size * element_size);
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_swap`.
|
||||
static inline void _array__swap(Array *self, Array *other) {
|
||||
Array swap = *other;
|
||||
*other = *self;
|
||||
*self = swap;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
|
||||
static inline void _array__grow(Array *self, uint32_t count, size_t element_size) {
|
||||
uint32_t new_size = self->size + count;
|
||||
if (new_size > self->capacity) {
|
||||
uint32_t new_capacity = self->capacity * 2;
|
||||
if (new_capacity < 8) new_capacity = 8;
|
||||
if (new_capacity < new_size) new_capacity = new_size;
|
||||
_array__reserve(self, element_size, new_capacity);
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_splice`.
|
||||
static inline void _array__splice(Array *self, size_t element_size,
|
||||
uint32_t index, uint32_t old_count,
|
||||
uint32_t new_count, const void *elements) {
|
||||
uint32_t new_size = self->size + new_count - old_count;
|
||||
uint32_t old_end = index + old_count;
|
||||
uint32_t new_end = index + new_count;
|
||||
assert(old_end <= self->size);
|
||||
|
||||
_array__reserve(self, element_size, new_size);
|
||||
|
||||
char *contents = (char *)self->contents;
|
||||
if (self->size > old_end) {
|
||||
memmove(
|
||||
contents + new_end * element_size,
|
||||
contents + old_end * element_size,
|
||||
(self->size - old_end) * element_size
|
||||
);
|
||||
}
|
||||
if (new_count > 0) {
|
||||
if (elements) {
|
||||
memcpy(
|
||||
(contents + index * element_size),
|
||||
elements,
|
||||
new_count * element_size
|
||||
);
|
||||
} else {
|
||||
memset(
|
||||
(contents + index * element_size),
|
||||
0,
|
||||
new_count * element_size
|
||||
);
|
||||
}
|
||||
}
|
||||
self->size += new_count - old_count;
|
||||
}
|
||||
|
||||
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
|
||||
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
|
||||
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
|
||||
do { \
|
||||
*(_index) = start; \
|
||||
*(_exists) = false; \
|
||||
uint32_t size = (self)->size - *(_index); \
|
||||
if (size == 0) break; \
|
||||
int comparison; \
|
||||
while (size > 1) { \
|
||||
uint32_t half_size = size / 2; \
|
||||
uint32_t mid_index = *(_index) + half_size; \
|
||||
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
|
||||
if (comparison <= 0) *(_index) = mid_index; \
|
||||
size -= half_size; \
|
||||
} \
|
||||
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
|
||||
if (comparison == 0) *(_exists) = true; \
|
||||
else if (comparison < 0) *(_index) += 1; \
|
||||
} while (0)
|
||||
|
||||
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
|
||||
/// parameter by reference in order to work with the generic sorting function above.
|
||||
#define _compare_int(a, b) ((int)*(a) - (int)(b))
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(default : 4101)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ARRAY_H_
|
||||
@@ -0,0 +1,265 @@
|
||||
#ifndef TREE_SITTER_PARSER_H_
|
||||
#define TREE_SITTER_PARSER_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
#define ts_builtin_sym_error ((TSSymbol)-1)
|
||||
#define ts_builtin_sym_end 0
|
||||
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
|
||||
|
||||
#ifndef TREE_SITTER_API_H_
|
||||
typedef uint16_t TSStateId;
|
||||
typedef uint16_t TSSymbol;
|
||||
typedef uint16_t TSFieldId;
|
||||
typedef struct TSLanguage TSLanguage;
|
||||
#endif
|
||||
|
||||
typedef struct {
|
||||
TSFieldId field_id;
|
||||
uint8_t child_index;
|
||||
bool inherited;
|
||||
} TSFieldMapEntry;
|
||||
|
||||
typedef struct {
|
||||
uint16_t index;
|
||||
uint16_t length;
|
||||
} TSFieldMapSlice;
|
||||
|
||||
typedef struct {
|
||||
bool visible;
|
||||
bool named;
|
||||
bool supertype;
|
||||
} TSSymbolMetadata;
|
||||
|
||||
typedef struct TSLexer TSLexer;
|
||||
|
||||
struct TSLexer {
|
||||
int32_t lookahead;
|
||||
TSSymbol result_symbol;
|
||||
void (*advance)(TSLexer *, bool);
|
||||
void (*mark_end)(TSLexer *);
|
||||
uint32_t (*get_column)(TSLexer *);
|
||||
bool (*is_at_included_range_start)(const TSLexer *);
|
||||
bool (*eof)(const TSLexer *);
|
||||
};
|
||||
|
||||
typedef enum {
|
||||
TSParseActionTypeShift,
|
||||
TSParseActionTypeReduce,
|
||||
TSParseActionTypeAccept,
|
||||
TSParseActionTypeRecover,
|
||||
} TSParseActionType;
|
||||
|
||||
typedef union {
|
||||
struct {
|
||||
uint8_t type;
|
||||
TSStateId state;
|
||||
bool extra;
|
||||
bool repetition;
|
||||
} shift;
|
||||
struct {
|
||||
uint8_t type;
|
||||
uint8_t child_count;
|
||||
TSSymbol symbol;
|
||||
int16_t dynamic_precedence;
|
||||
uint16_t production_id;
|
||||
} reduce;
|
||||
uint8_t type;
|
||||
} TSParseAction;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
} TSLexMode;
|
||||
|
||||
typedef union {
|
||||
TSParseAction action;
|
||||
struct {
|
||||
uint8_t count;
|
||||
bool reusable;
|
||||
} entry;
|
||||
} TSParseActionEntry;
|
||||
|
||||
typedef struct {
|
||||
int32_t start;
|
||||
int32_t end;
|
||||
} TSCharacterRange;
|
||||
|
||||
struct TSLanguage {
|
||||
uint32_t version;
|
||||
uint32_t symbol_count;
|
||||
uint32_t alias_count;
|
||||
uint32_t token_count;
|
||||
uint32_t external_token_count;
|
||||
uint32_t state_count;
|
||||
uint32_t large_state_count;
|
||||
uint32_t production_id_count;
|
||||
uint32_t field_count;
|
||||
uint16_t max_alias_sequence_length;
|
||||
const uint16_t *parse_table;
|
||||
const uint16_t *small_parse_table;
|
||||
const uint32_t *small_parse_table_map;
|
||||
const TSParseActionEntry *parse_actions;
|
||||
const char * const *symbol_names;
|
||||
const char * const *field_names;
|
||||
const TSFieldMapSlice *field_map_slices;
|
||||
const TSFieldMapEntry *field_map_entries;
|
||||
const TSSymbolMetadata *symbol_metadata;
|
||||
const TSSymbol *public_symbol_map;
|
||||
const uint16_t *alias_map;
|
||||
const TSSymbol *alias_sequences;
|
||||
const TSLexMode *lex_modes;
|
||||
bool (*lex_fn)(TSLexer *, TSStateId);
|
||||
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
|
||||
TSSymbol keyword_capture_token;
|
||||
struct {
|
||||
const bool *states;
|
||||
const TSSymbol *symbol_map;
|
||||
void *(*create)(void);
|
||||
void (*destroy)(void *);
|
||||
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
|
||||
unsigned (*serialize)(void *, char *);
|
||||
void (*deserialize)(void *, const char *, unsigned);
|
||||
} external_scanner;
|
||||
const TSStateId *primary_state_ids;
|
||||
};
|
||||
|
||||
static inline bool set_contains(TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
|
||||
uint32_t index = 0;
|
||||
uint32_t size = len - index;
|
||||
while (size > 1) {
|
||||
uint32_t half_size = size / 2;
|
||||
uint32_t mid_index = index + half_size;
|
||||
TSCharacterRange *range = &ranges[mid_index];
|
||||
if (lookahead >= range->start && lookahead <= range->end) {
|
||||
return true;
|
||||
} else if (lookahead > range->end) {
|
||||
index = mid_index;
|
||||
}
|
||||
size -= half_size;
|
||||
}
|
||||
TSCharacterRange *range = &ranges[index];
|
||||
return (lookahead >= range->start && lookahead <= range->end);
|
||||
}
|
||||
|
||||
/*
|
||||
* Lexer Macros
|
||||
*/
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#define UNUSED __pragma(warning(suppress : 4101))
|
||||
#else
|
||||
#define UNUSED __attribute__((unused))
|
||||
#endif
|
||||
|
||||
#define START_LEXER() \
|
||||
bool result = false; \
|
||||
bool skip = false; \
|
||||
UNUSED \
|
||||
bool eof = false; \
|
||||
int32_t lookahead; \
|
||||
goto start; \
|
||||
next_state: \
|
||||
lexer->advance(lexer, skip); \
|
||||
start: \
|
||||
skip = false; \
|
||||
lookahead = lexer->lookahead;
|
||||
|
||||
#define ADVANCE(state_value) \
|
||||
{ \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ADVANCE_MAP(...) \
|
||||
{ \
|
||||
static const uint16_t map[] = { __VA_ARGS__ }; \
|
||||
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
|
||||
if (map[i] == lookahead) { \
|
||||
state = map[i + 1]; \
|
||||
goto next_state; \
|
||||
} \
|
||||
} \
|
||||
}
|
||||
|
||||
#define SKIP(state_value) \
|
||||
{ \
|
||||
skip = true; \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ACCEPT_TOKEN(symbol_value) \
|
||||
result = true; \
|
||||
lexer->result_symbol = symbol_value; \
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
#define END_STATE() return result;
|
||||
|
||||
/*
|
||||
* Parse Table Macros
|
||||
*/
|
||||
|
||||
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
|
||||
|
||||
#define STATE(id) id
|
||||
|
||||
#define ACTIONS(id) id
|
||||
|
||||
#define SHIFT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value) \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_REPEAT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value), \
|
||||
.repetition = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_EXTRA() \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.extra = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define REDUCE(symbol_name, children, precedence, prod_id) \
|
||||
{{ \
|
||||
.reduce = { \
|
||||
.type = TSParseActionTypeReduce, \
|
||||
.symbol = symbol_name, \
|
||||
.child_count = children, \
|
||||
.dynamic_precedence = precedence, \
|
||||
.production_id = prod_id \
|
||||
}, \
|
||||
}}
|
||||
|
||||
#define RECOVER() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeRecover \
|
||||
}}
|
||||
|
||||
#define ACCEPT_INPUT() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeAccept \
|
||||
}}
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_PARSER_H_
|
||||
File diff suppressed because it is too large
Load Diff
@@ -13,11 +13,17 @@
|
||||
|
||||
use tree_sitter::Language;
|
||||
|
||||
// Vendored kotlin grammar (build.rs-compiled C — no usable crate exists;
|
||||
// see grammars/kotlin and the kotlin checklist §Grammar prep).
|
||||
extern "C" {
|
||||
fn tree_sitter_kotlin() -> *const ();
|
||||
}
|
||||
|
||||
/// Languages this kernel binary can extract (reported by contractInfo;
|
||||
/// TS-side routing policy decides what actually routes).
|
||||
pub const LANGUAGES: [&str; 14] = [
|
||||
pub const LANGUAGES: [&str; 15] = [
|
||||
"typescript", "tsx", "javascript", "jsx", "java", "python", "go", "c", "cpp", "rust",
|
||||
"csharp", "ruby", "php", "swift",
|
||||
"csharp", "ruby", "php", "swift", "kotlin",
|
||||
];
|
||||
|
||||
pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
@@ -47,6 +53,10 @@ pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
// R7b: crate 0.7.3 — the vendored wasm is built from this crate's own
|
||||
// tarball src/ (table identity by construction; see grammars.ts).
|
||||
"swift" => Some(tree_sitter_swift::LANGUAGE.into()),
|
||||
// R7b: fwcd 0.3.8, vendored C compiled in build.rs (crate unusable).
|
||||
"kotlin" => {
|
||||
Some(unsafe { tree_sitter_language::LanguageFn::from_raw(tree_sitter_kotlin) }.into())
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ mod docstring;
|
||||
mod ids;
|
||||
mod go;
|
||||
mod java;
|
||||
mod kotlin;
|
||||
mod langs;
|
||||
mod php;
|
||||
mod ruby;
|
||||
@@ -221,6 +222,7 @@ pub fn extract_file(file_path: String, content: String, language: String) -> Res
|
||||
"ruby" => ruby::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"php" => php::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"swift" => swift::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
"kotlin" => kotlin::extract(&file_path, &content).map_err(Error::from_reason)?,
|
||||
_ => tsjs::extract(&file_path, &content, &language).map_err(Error::from_reason)?,
|
||||
};
|
||||
Ok(ExtractBuffers {
|
||||
|
||||
@@ -1,6 +1,16 @@
|
||||
# Kotlin kernel port (R7b) — the bug-for-bug checklist
|
||||
|
||||
**Status: SURVEY COMPLETE (2026-07-20) — PORT NOT STARTED.** Survey basis: every
|
||||
**Status: PORT COMPLETE (2026-07-20)** — walker `codegraph-kernel/src/kotlin.rs`
|
||||
+ the vendored-grammar-C build (codegraph-kernel/grammars/kotlin via build.rs
|
||||
cc — the mechanism's first use), all gates passed (bump dumps byte-identical
|
||||
old-vs-new ×3 as predicted; parity sweeps 0-diff okio 299/322 / okhttp
|
||||
531/580 / kotlinx.coroutines 1031/1082 with exactly the predicted 23/49/51
|
||||
deferrals; kernel-arm dumps byte-identical ×3; KMP expect/actual synthesis
|
||||
IDENTICAL — 412 edges both arms; kernel-kotlin-parity suite; DEFAULT_ROUTED
|
||||
+= kotlin — 15 languages). One fixture note: the survey's torture.kt itself
|
||||
tripped the PHANTOM-error class (one-line class/object bodies) and deferred —
|
||||
the checked-in parity fixture reflows those to multi-line, and the phantom
|
||||
shape is pinned in the defer test instead. Survey basis: every
|
||||
TS-side branch a `.kt`/`.kts` file exercises, with file:line anchors as of
|
||||
**`a6c62d7`** (HEAD at survey time, clean main). Every grammar-shape claim below
|
||||
was **probed against both the production tree-sitter-wasms build and a fresh
|
||||
|
||||
@@ -609,7 +609,7 @@ parity before porting the language.
|
||||
| csharp | `languages/csharp.ts` | T1 | crates.io | **DONE (R7b #2, 2026-07-20)** — `csharp.rs` walker; NO grammar bump (the #717 vendored wasm verified table-identical to crate 0.23.5 — first port with no grammar-prep step); the #237 `#if` preParse stays TS-side via the route-point hoist. Parity 0-diff on serilog/Newtonsoft.Json/jellyfin (3,229 files) + dump byte-identical ×3; deferral 0.05–3.3% = both-arm `#if` damage. Quirk list: docs/design/csharp-kernel-port-checklist.md. | ☑ |
|
||||
| rust | `languages/rust.ts` | T1 | crates.io | **DONE (R7b #1, 2026-07-20)** — `rustlang.rs` walker; grammar bumped to v0.24.2 (crate + vendored wasm together). Parity 0-diff on ripgrep/tokio/rust-analyzer + dump byte-identical ×3; rust-analyzer's parser crates defer 18% (token-macro tables, both-arm parse errors — grammar-inherent). Quirk list: docs/design/rust-lang-kernel-port-checklist.md. | ☑ |
|
||||
| dart, scala, lua, luau, r | dedicated files | T1 | crates.io (luau/r/scala: verify crate freshness vs our wasm) | Long-tail T1; port opportunistically after the big five. | ☐ |
|
||||
| kotlin | `languages/kotlin.ts` | T1½ | crates.io | Expect/actual pairing is synthesis-side (fine); extraction is clean but validate against a KMP repo. | ☐ |
|
||||
| kotlin | `languages/kotlin.ts` | T1½ | **vendored C** (crate unusable) | **DONE (R7b #6, 2026-07-20)** — `kotlin.rs` walker; the arc's FIRST vendored-grammar-C language: fwcd 0.3.8's sha-matched parser.c/scanner.c compile inside codegraph-kernel via build.rs + cc (the crates.io crate pins tree-sitter <0.23; tree-sitter-kotlin-ng is a different grammar). Behavior-neutral wasm re-vendor (dumps byte-identical old-vs-new ×3). Two walker firsts: extension-fn receiver QNs + owner-contains, and extractModifiers→decorators (KMP expect/actual — 412 synthesized edges identical both arms on kotlinx.coroutines). Parity 0-diff okio/okhttp/kotlinx.coroutines (1,861 clean files; deferral 4.7–8.5% both-arm incl. PHANTOM hasError files). Quirk list: docs/design/kotlin-kernel-port-checklist.md. | ☑ |
|
||||
| swift | shared + dedicated branch | T1½ | crates.io | **DONE (R7b #5, 2026-07-20)** — `swift.rs` walker incl. the #1020 dedicated property branch (Alamofire's 348 property nodes reproduced exactly on the kernel arm); grammar bumped to crate 0.7.3 (wasm built from the CRATE TARBALL's src — the tag ships an older ABI-14 generation; delta = error-set membership + 2 gate-found categories, all classified). Parity 0-diff Alamofire/vapor/swift-nio (720 clean files; deferral 9–27% both-arm structural — sweeps use --max-deferral 0.3). One walker fix found by the sweep: the shared `assignment` shadow-prune case is swift-live (declared-then-assigned `let X: T`). Quirk list: docs/design/swift-kernel-port-checklist.md. | ☑ |
|
||||
| c, cpp | `languages/c-cpp.ts` | **T2** | crates.io | **DONE (R7a, 2026-07-17)** — `ccpp/` walker; ALL pre-passes stayed TS-side via the route-point preParse hoist (+6 new blanks added during gating — see the checklist doc); content-based `.h` C-vs-C++ detection stays upstream at detectLanguage. Parity 0-diff + dump byte-identical on redis/git/fmt/protobuf/ALS. | ☑ |
|
||||
| metal, cuda | dialects over the cpp grammar | **T2** (rides c/cpp) | crates.io (cpp) | **DONE (rides R7a)** — `.metal`/`.cu`/`.cuh` map to 'cpp' and their blanks run in the hoisted preParse (filePath rides along for the extension gates); hoist-parity pinned in kernel-ccpp-parity.test.ts + the metal/cuda suites. | ☑ |
|
||||
|
||||
@@ -48,7 +48,7 @@ if (paths.length === 0) {
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift']);
|
||||
const KERNEL_LANGS = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'c', 'cpp', 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin']);
|
||||
const EXTS = new Map([
|
||||
['.ts', 'typescript'], ['.mts', 'typescript'], ['.cts', 'typescript'],
|
||||
['.tsx', 'tsx'], ['.js', 'javascript'], ['.mjs', 'javascript'],
|
||||
@@ -63,6 +63,7 @@ const EXTS = new Map([
|
||||
['.rb', 'ruby'], ['.rake', 'ruby'], // R7b
|
||||
['.php', 'php'], ['.module', 'php'], ['.install', 'php'], ['.theme', 'php'], ['.inc', 'php'], // R7b
|
||||
['.swift', 'swift'], // R7b
|
||||
['.kt', 'kotlin'], ['.kts', 'kotlin'], // R7b
|
||||
]);
|
||||
|
||||
/** Collect candidate files. */
|
||||
|
||||
@@ -322,6 +322,14 @@ const VENDORED_WASM_LANGS: ReadonlySet<GrammarLanguage> = new Set([
|
||||
// (ABI 13 → 15). NOT graph-neutral — delta is error-set membership only;
|
||||
// classified list in the swift checklist doc.
|
||||
'swift',
|
||||
// R7b (Kotlin kernel port prep): fwcd tree-sitter-kotlin 0.3.8 (tag
|
||||
// e1a2d5a), parser.c/scanner.c sha-matched crate↔tag; behavior-IDENTICAL
|
||||
// to the tree-sitter-wasms build (0 CST/error disagreements across the
|
||||
// gate repos) — a reproducibility re-vendor, ABI stays 14. The crates.io
|
||||
// crate is UNUSABLE by the kernel (pins tree-sitter <0.23) and
|
||||
// tree-sitter-kotlin-ng is a different grammar — the kernel compiles the
|
||||
// same vendored C sources instead (codegraph-kernel/grammars/kotlin).
|
||||
'kotlin',
|
||||
]);
|
||||
|
||||
/** Absolute path of a language's grammar WASM (vendored or tree-sitter-wasms). */
|
||||
|
||||
@@ -74,6 +74,13 @@ const DEFAULT_ROUTED: ReadonlySet<Language> = new Set<Language>([
|
||||
// is structurally 9–27% on BOTH arms (heavy #if conditionalization) —
|
||||
// sweeps run --max-deferral 0.3; a deferral-rate JUMP is the bug signal.
|
||||
'swift',
|
||||
// R7b (2026-07-20): parity swept 0-diff on okio/okhttp/kotlinx.coroutines
|
||||
// (1,861 clean files byte-parity; KMP expect/actual synthesis identical —
|
||||
// 412 edges both arms on kotlinx.coroutines) + full-init dump-diffs
|
||||
// byte-identical ×3. Deferral 4.7–8.5% both-arm grammar reality
|
||||
// (fun-interface misparses + PHANTOM hasError files) — default sweep guard
|
||||
// holds; a JUMP past ~10% is the bug signal.
|
||||
'kotlin',
|
||||
]);
|
||||
|
||||
/**
|
||||
|
||||
Executable
BIN
Binary file not shown.
Reference in New Issue
Block a user