feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)

The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.

The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.

The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.

Three differences are not cosmetic:

* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
  not as string. The call-site overlay refuses to claim a token classed string,
  so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
  The grammars disagree about whether `string` is a type_identifier or an
  anonymous token inside a predefined_type, and TextMate scoped them
  inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.

Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.

Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
This commit is contained in:
Colby McHenry
2026-08-27 06:22:11 -05:00
parent 8ac0138940
commit ad91c8fdd8
32 changed files with 1084 additions and 1293 deletions
+7 -5
View File
@@ -11,7 +11,7 @@ Distributed as `@colbymchenry/codegraph` on npm; same binary serves as installer
## Build, Test, Run
```bash
npm run build # tsc + copy schema.sql and *.wasm + prune TextMate grammars + build the viewer into dist/; chmods dist/bin/codegraph.js
npm run build # tsc + copy schema.sql and *.wasm + build the viewer into dist/; chmods dist/bin/codegraph.js
npm run dev # tsc --watch
npm run clean # rm -rf dist
@@ -29,10 +29,12 @@ npx vitest run __tests__/extraction.test.ts -t "TypeScript"
`copy-assets` (called from `build`) copies `src/db/schema.sql` and all `src/extraction/wasm/*.wasm` files into `dist/`. **Any new SQL or grammar wasm must be copied or it won't ship.**
Two other build steps write into `dist/` and are subject to the same rule:
`build:textmate` (`scripts/prune-grammars.mjs`) writes the viewer's syntax grammars to `dist/textmate/`, and
`build:ui` builds the browser viewer into `dist/viewer/` (never `dist/ui/` — that's the terminal ui).
`scripts/check-ui-build.mjs` asserts both trees after every build and inside every release archive.
One other build step writes into `dist/` and is subject to the same rule: `build:ui` builds the
browser viewer into `dist/viewer/` (never `dist/ui/` — that's the terminal ui).
`scripts/check-ui-build.mjs` asserts both `dist/viewer/` and the copied grammars in
`dist/extraction/wasm/` after every build and inside every release archive — the viewer's syntax
highlighting reads a file with the same grammar the engine indexed it with, so a missing wasm is an
unhighlighted screen as well as an extraction gap.
Node engines: `>=20.0.0 <25.0.0`. There is a hard exit on Node 25.x and below 20 (see `src/bin/node-version-check.ts`).