feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)

The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.

The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.

The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.

Three differences are not cosmetic:

* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
  not as string. The call-site overlay refuses to claim a token classed string,
  so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
  The grammars disagree about whether `string` is a type_identifier or an
  anonymous token inside a predefined_type, and TextMate scoped them
  inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.

Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.

Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
This commit is contained in:
Colby McHenry
2026-08-27 06:22:11 -05:00
parent 8ac0138940
commit ad91c8fdd8
32 changed files with 1084 additions and 1293 deletions
+2 -6
View File
@@ -20,7 +20,7 @@
"ui"
],
"scripts": {
"build": "tsc && npm run copy-assets && npm run build:textmate && npm run build:ui && node -e \"require('fs').chmodSync('dist/bin/codegraph.js', 0o755)\"",
"build": "tsc && npm run copy-assets && npm run build:ui && node -e \"require('fs').chmodSync('dist/bin/codegraph.js', 0o755)\"",
"build:ui": "npm run build --workspace ui && node scripts/check-ui-build.mjs",
"preuninstall": "node dist/bin/uninstall.js",
"copy-assets": "node -e \"const fs=require('fs');fs.mkdirSync('dist/db',{recursive:true});fs.copyFileSync('src/db/schema.sql','dist/db/schema.sql');fs.mkdirSync('dist/extraction/wasm',{recursive:true});fs.readdirSync('src/extraction/wasm').filter(f=>f.endsWith('.wasm')).forEach(f=>fs.copyFileSync('src/extraction/wasm/'+f,'dist/extraction/wasm/'+f))\"",
@@ -31,8 +31,7 @@
"test:watch": "vitest",
"test:eval": "vitest run __tests__/evaluation/",
"eval": "npm run build && npx tsx __tests__/evaluation/runner.ts",
"clean": "node -e \"const fs=require('fs');fs.rmSync('dist',{recursive:true,force:true})\"",
"build:textmate": "node scripts/prune-grammars.mjs"
"clean": "node -e \"const fs=require('fs');fs.rmSync('dist',{recursive:true,force:true})\""
},
"keywords": [
"code-intelligence",
@@ -43,8 +42,6 @@
"license": "MIT",
"dependencies": {
"@clack/prompts": "^1.3.0",
"@shikijs/core": "^4.4.3",
"@shikijs/engine-javascript": "^4.4.3",
"commander": "^14.0.2",
"fast-string-width": "^3.0.2",
"fast-wrap-ansi": "^0.2.0",
@@ -56,7 +53,6 @@
"web-tree-sitter": "^0.25.3"
},
"devDependencies": {
"@shikijs/langs": "^4.4.3",
"@types/better-sqlite3": "^7.6.0",
"@types/node": "^20.19.30",
"@types/picomatch": "^4.0.2",