feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)

The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.

The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.

The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.

Three differences are not cosmetic:

* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
  not as string. The call-site overlay refuses to claim a token classed string,
  so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
  The grammars disagree about whether `string` is a type_identifier or an
  anonymous token inside a predefined_type, and TextMate scoped them
  inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.

Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.

Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
This commit is contained in:
Colby McHenry
2026-08-27 06:22:11 -05:00
parent 8ac0138940
commit ad91c8fdd8
32 changed files with 1084 additions and 1293 deletions
+24 -14
View File
@@ -321,22 +321,32 @@ with `src/index.ts` selected, 15 links and 4 dimmed boxes, matching the canvas).
- Svelte 5 (≥ 5.25) + Vite (workspace `ui/`), Svelte Flow `@xyflow/svelte` ^1.6 for the Map and Flow canvases only (custom nodes/edges,
hidden handles for port spreading, local selection state — the pattern in docker-app's `StackGraph.svelte`); `@dagrejs/dagre` only as a
fallback if crossing quality demands it (never ELK). Symbol view = DOM + one SVG overlay (`ResizeObserver` re-layout).
- Shiki (JavaScript regex engine, lazy grammars, custom near-monochrome theme as in §2.2) server-side in `/api/source`; tree-sitter-derived
tokens replace it in phase 3.
- *As built (CG-43).* `@shikijs/core` + `@shikijs/engine-javascript` are runtime dependencies (~5 MB installed, no wasm, no native
module); `@shikijs/langs` is a **devDependency** and `npm run build:textmate` (`scripts/prune-grammars.mjs`) writes only the
closure the engine's 40-odd languages reach — 56 grammars, 2.6 MB — into **`dist/textmate/`**, checked by `scripts/check-ui-build.mjs`.
Shipping all 722 grammars would have been 11 MB.
- The theme classifies rather than colours: its foregrounds are sentinels the server maps to class names (`comment`, `string`,
`keyword`, `number`, `ident`, `other`), and the viewer paints them from the CSS custom properties above — so **one token stream
serves light and dark** with no refetch when `prefers-color-scheme` flips, and the ramp lives only in `ui/src/app.css`.
- Syntax classification comes off **the engine's own tree-sitter parse** — no highlighter dependency, no second grammar set.
- *As built (CG-43, replaced in CG-57).* The first cut ran Shiki with 56 pruned TextMate grammars in `dist/textmate/`. That is
gone: `@shikijs/*` is off the dependency list, `scripts/prune-grammars.mjs` and `npm run build:textmate` are deleted, and
`scripts/check-ui-build.mjs` now asserts the tree-sitter grammars in `dist/extraction/wasm/` instead. A `.ts` file is read by
exactly the grammar that decided what its symbols are, so the viewer and the graph can never disagree about it.
- Eight token classes on the wire: `comment`, `string`, `number`, `keyword`, `type`, `def`, `ident`, `other`. Rules, not scope
tables — a node whose type mentions `comment` is a comment; inside a string every leaf is string *except* below an
interpolation, where code resumes (so `${user.name()}` still links); an **anonymous** leaf is a keyword when its text is a bare
word and punctuation otherwise; a **named** leaf is an identifier, a type name, or — from the extractors' own definition
tables — the name a definition declares. `punct` is folded into `other`: they paint identically and splitting them would
roughly double the token count on a dense line.
- The classification is a class NAME, never a colour, and the viewer paints it from the CSS custom properties above — so **one
token stream serves light and dark** with no refetch when `prefers-color-scheme` flips, and the ramp lives only in
`ui/src/app.css`. `type` is a distinct class painted at plain ink: the colouring is near-monochrome and a type name is not one
of the four things it moves off plain ink.
- Every code token is split into identifier runs before it goes on the wire, so the graph's call-site overlay claims a token the
highlighter produced rather than re-cutting a line — which is what keeps a link landing on the callee's own name whatever
classifier produced rather than re-cutting a line — which is what keeps a link landing on the callee's own name whatever
boundaries a grammar chose, and keeps links working in the plain-text fallback.
- Measured on this machine (Shiki 4.4.3, JS regex engine, 3 000 lines cold): Python 3547 ms, Go 4357 ms, **TypeScript ~700 ms**
the TS TextMate grammar is 57× the cost of any other and the oniguruma-wasm engine would run it in ~120 ms. Slices are therefore
cached by content hash + range, so a re-render (resize, theme flip, stepping back through the trail) is a map lookup (< 10 ms);
a symbol-sized slice (~280 lines of TS) is ~50 ms cold. Phase 1 only ever requests one symbol's range.
- Single-file components (`.svelte`, `.vue`, `.astro`) have no grammar of their own; their `<script>` blocks — where every
indexed symbol in those files lives — are classified as TypeScript or JavaScript, exactly the delegation the extractors
already do. The surrounding markup, and the config formats with file-level extraction only (YAML, XML, Twig, properties),
render plain with their identifiers still split out, so links land there too.
- Measured on this machine, 3 000 lines cold: **TypeScript 2441 ms** (it was ~700 ms under Shiki, whose TS grammar cost 57×
every other one), Go ~30 ms, Python 2529 ms, and Rust/Ruby/PHP/C#/Swift 1427 ms. Slices are still cached by content hash +
range, so a re-render (resize, theme flip, stepping back through the trail) is a map lookup. Side-by-side parity screenshots
for the eight gate languages: `docs/design/cg57-highlighting-parity/`.
- No native modules; no runtime dependency for the UI itself; the CLI serves **`dist/viewer/`** over `node:http`, loopback only.
(Not `dist/ui/``src/ui/` is the engine's *terminal* ui and tsc already compiles it there; see `ui/README.md`.)