feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)
The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.
The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.
The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.
Three differences are not cosmetic:
* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
not as string. The call-site overlay refuses to claim a token classed string,
so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
The grammars disagree about whether `string` is a type_identifier or an
anonymous token inside a predefined_type, and TextMate scoped them
inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.
Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.
Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
This commit is contained in:
+2
-6
@@ -20,7 +20,7 @@
|
||||
"ui"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "tsc && npm run copy-assets && npm run build:textmate && npm run build:ui && node -e \"require('fs').chmodSync('dist/bin/codegraph.js', 0o755)\"",
|
||||
"build": "tsc && npm run copy-assets && npm run build:ui && node -e \"require('fs').chmodSync('dist/bin/codegraph.js', 0o755)\"",
|
||||
"build:ui": "npm run build --workspace ui && node scripts/check-ui-build.mjs",
|
||||
"preuninstall": "node dist/bin/uninstall.js",
|
||||
"copy-assets": "node -e \"const fs=require('fs');fs.mkdirSync('dist/db',{recursive:true});fs.copyFileSync('src/db/schema.sql','dist/db/schema.sql');fs.mkdirSync('dist/extraction/wasm',{recursive:true});fs.readdirSync('src/extraction/wasm').filter(f=>f.endsWith('.wasm')).forEach(f=>fs.copyFileSync('src/extraction/wasm/'+f,'dist/extraction/wasm/'+f))\"",
|
||||
@@ -31,8 +31,7 @@
|
||||
"test:watch": "vitest",
|
||||
"test:eval": "vitest run __tests__/evaluation/",
|
||||
"eval": "npm run build && npx tsx __tests__/evaluation/runner.ts",
|
||||
"clean": "node -e \"const fs=require('fs');fs.rmSync('dist',{recursive:true,force:true})\"",
|
||||
"build:textmate": "node scripts/prune-grammars.mjs"
|
||||
"clean": "node -e \"const fs=require('fs');fs.rmSync('dist',{recursive:true,force:true})\""
|
||||
},
|
||||
"keywords": [
|
||||
"code-intelligence",
|
||||
@@ -43,8 +42,6 @@
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@clack/prompts": "^1.3.0",
|
||||
"@shikijs/core": "^4.4.3",
|
||||
"@shikijs/engine-javascript": "^4.4.3",
|
||||
"commander": "^14.0.2",
|
||||
"fast-string-width": "^3.0.2",
|
||||
"fast-wrap-ansi": "^0.2.0",
|
||||
@@ -56,7 +53,6 @@
|
||||
"web-tree-sitter": "^0.25.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@shikijs/langs": "^4.4.3",
|
||||
"@types/better-sqlite3": "^7.6.0",
|
||||
"@types/node": "^20.19.30",
|
||||
"@types/picomatch": "^4.0.2",
|
||||
|
||||
Reference in New Issue
Block a user