feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)

The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.

The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.

The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.

Three differences are not cosmetic:

* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
  not as string. The call-site overlay refuses to claim a token classed string,
  so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
  The grammars disagree about whether `string` is a type_identifier or an
  anonymous token inside a predefined_type, and TextMate scoped them
  inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.

Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.

Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
This commit is contained in:
Colby McHenry
2026-08-27 06:22:11 -05:00
parent 8ac0138940
commit ad91c8fdd8
32 changed files with 1084 additions and 1293 deletions
+95
View File
@@ -0,0 +1,95 @@
# Highlighting parity: Shiki → the engine's own tree-sitter parse (CG-57)
The viewer's code block used to be classified by a second highlighter — Shiki with 56 pruned
TextMate grammars shipped in `dist/textmate/` — over source the engine had already parsed with a
real grammar. CG-57 takes the classification off that tree instead. This file records what the swap
changed, measured rather than asserted, so nobody has to re-derive it from a diff.
Screenshots, one per language, before on the left and after on the right, same stylesheet:
[`cg57-highlighting-parity/`](./cg57-highlighting-parity/) — `typescript.png`, `go.png`,
`python.png`, `rust.png`, `swift.png`, `csharp.png`, `ruby.png`, `php.png`.
## What it costs
3 000 lines, cold, dev Mac (M-series), parse + classify + wire:
| | TypeScript | Go | Python | Rust | Swift | C# | Ruby | PHP |
|---|---|---|---|---|---|---|---|---|
| Shiki + TextMate | ~700 ms | 4357 ms | 3547 ms | — | — | — | — | — |
| Engine tree-sitter | 2441 ms | ~30 ms | 2529 ms | 1819 ms | 2527 ms | 2025 ms | 1416 ms | 2022 ms |
The task's budget was **< 100 ms per 3 000-line file warm**; every language clears it *cold*.
TypeScript is the number that mattered: its TextMate grammar was 57× every other one and the cost
was regex *execution*, not compilation, so nothing about the old module could have fixed it. The
slice cache still exists — a re-render (resize, theme flip, stepping back through the trail) should
cost nothing at all, and the whole-file view pages the same file repeatedly.
## What it changes on screen
Per-character comparison over ~40 lines of realistic source per language, counting only
non-whitespace characters, and treating `ident` / `other` / `type` as one bucket because all three
paint at plain ink:
| language | painted identically | what moved |
|---|---|---|
| TypeScript | 91.3% | 33 `def`, 40 interpolation chars now code, 2 punctuation |
| Go | 91.5% | 37 built-in type words, 13 `def` |
| Python | 93.2% | 26 `def`, 15 keyword (`is not`, `__future__`) |
| Rust | 96.3% | 21 `def`, 3 keyword |
| Swift | 93.3% | 18 `def`, 14 keyword (`throws`/`rethrows`) |
| C# | 88.3% | 30 built-in type words, 23 `def`, 31 interpolation chars now code |
| Ruby | 83.8% | 23 `def`, 35 interpolation chars now code, 14 symbol literals, 3 keyword |
| PHP | 85.9% | 33 built-in type words, 29 `def`, 15 phpdoc tag chars, 12 keyword |
Every remaining difference is one of five deliberate categories:
1. **`ident``def`.** The definition's own name now carries weight 600, everywhere rather than
only on the line the Symbol view opened at. It comes from the extractors' own definition tables
(`functionTypes`, `classTypes`, `methodTypes`, …) plus each language's `nameField`, so it cannot
drift from what indexing considers a definition.
2. **`string` → code, inside an interpolation.** A template literal's `${…}`, an f-string's `{…}`,
Ruby's `#{…}` and C#'s `$"{…}"` are classified as code. This is the one difference that is not
cosmetic: the call-site overlay deliberately refuses to claim a token classed `string`, so
**calls inside interpolated strings now link and did not before.**
3. **`keyword``type`, on built-in type words.** `string`, `int`, `u32`, `void`. The grammars
disagree with each other about what a built-in type is — tree-sitter-go calls `string` a
`type_identifier`, tree-sitter-typescript wraps it in a `predefined_type` whose child is an
anonymous token spelled `string` — and TextMate scoped them inconsistently too (plain in
TypeScript, `storage.type` in Go). They now all paint at plain ink, like a user-defined type
name, in every language.
4. **Keyword-set corrections.** Python's `is not`, Rust's and Swift's modifiers, and Ruby's `new`
(which is a method, not a keyword — TextMate's `keyword.operator.new` matched it anyway).
5. **`keyword``comment`, on phpdoc tags.** `@var` and friends recede with the comment they are
in, which is what the near-monochrome ramp asks for.
## What is no longer highlighted
Nine formats have extraction but no tree-sitter grammar. Three of them — `.svelte`, `.vue`,
`.astro` — are classified through their `<script>` blocks with TypeScript or JavaScript, the same
delegation the extractors do, so every symbol the engine indexed in those files is highlighted and
the surrounding markup is not. The other six (Liquid, Razor, YAML, Twig, XML, `.properties`) render
plain, where Shiki had grammars for them.
That is a real, deliberate loss, and it is the alternative to a worse one. `tree-sitter-wasms`
ships an `html` grammar that would cover most of them, but the ABI-13 builds in that package are
the known cause of a shared-WASM-heap corruption that silently drops edges for *every other*
language in the same process (see `VENDORED_WASM_LANGS` in `src/extraction/grammars.ts`), and the
viewer runs in a process someone leaves open all day. Adding unvetted grammars to buy tag colouring
on config files is not a trade worth making. Identifiers are still split out on those files, so the
graph's call-site links land exactly as they do everywhere else — highlighting is the part that
degrades, never the linking.
## Reproducing this
There is no committed harness: the "before" side needs the deleted Shiki module. Rebuild it from
the last commit that had it —
```
git worktree add /tmp/cg48-baseline <ref-with-shiki>
ln -s "$PWD/node_modules" /tmp/cg48-baseline/node_modules # @shikijs/* must still be installed
( cd /tmp/cg48-baseline && npx tsc && node scripts/prune-grammars.mjs )
```
— then run both `dist/ui-server/highlight/index.js` modules over the same lines and compare
`classes[id]` per character. The screenshots were rendered from the same two token streams through
the viewer's own token CSS at `--force-device-scale-factor=2`.
Binary file not shown.

After

Width:  |  Height:  |  Size: 203 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 194 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 216 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 188 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 172 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 207 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 153 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 254 KiB

+24 -14
View File
@@ -321,22 +321,32 @@ with `src/index.ts` selected, 15 links and 4 dimmed boxes, matching the canvas).
- Svelte 5 (≥ 5.25) + Vite (workspace `ui/`), Svelte Flow `@xyflow/svelte` ^1.6 for the Map and Flow canvases only (custom nodes/edges,
hidden handles for port spreading, local selection state — the pattern in docker-app's `StackGraph.svelte`); `@dagrejs/dagre` only as a
fallback if crossing quality demands it (never ELK). Symbol view = DOM + one SVG overlay (`ResizeObserver` re-layout).
- Shiki (JavaScript regex engine, lazy grammars, custom near-monochrome theme as in §2.2) server-side in `/api/source`; tree-sitter-derived
tokens replace it in phase 3.
- *As built (CG-43).* `@shikijs/core` + `@shikijs/engine-javascript` are runtime dependencies (~5 MB installed, no wasm, no native
module); `@shikijs/langs` is a **devDependency** and `npm run build:textmate` (`scripts/prune-grammars.mjs`) writes only the
closure the engine's 40-odd languages reach — 56 grammars, 2.6 MB — into **`dist/textmate/`**, checked by `scripts/check-ui-build.mjs`.
Shipping all 722 grammars would have been 11 MB.
- The theme classifies rather than colours: its foregrounds are sentinels the server maps to class names (`comment`, `string`,
`keyword`, `number`, `ident`, `other`), and the viewer paints them from the CSS custom properties above — so **one token stream
serves light and dark** with no refetch when `prefers-color-scheme` flips, and the ramp lives only in `ui/src/app.css`.
- Syntax classification comes off **the engine's own tree-sitter parse** — no highlighter dependency, no second grammar set.
- *As built (CG-43, replaced in CG-57).* The first cut ran Shiki with 56 pruned TextMate grammars in `dist/textmate/`. That is
gone: `@shikijs/*` is off the dependency list, `scripts/prune-grammars.mjs` and `npm run build:textmate` are deleted, and
`scripts/check-ui-build.mjs` now asserts the tree-sitter grammars in `dist/extraction/wasm/` instead. A `.ts` file is read by
exactly the grammar that decided what its symbols are, so the viewer and the graph can never disagree about it.
- Eight token classes on the wire: `comment`, `string`, `number`, `keyword`, `type`, `def`, `ident`, `other`. Rules, not scope
tables — a node whose type mentions `comment` is a comment; inside a string every leaf is string *except* below an
interpolation, where code resumes (so `${user.name()}` still links); an **anonymous** leaf is a keyword when its text is a bare
word and punctuation otherwise; a **named** leaf is an identifier, a type name, or — from the extractors' own definition
tables — the name a definition declares. `punct` is folded into `other`: they paint identically and splitting them would
roughly double the token count on a dense line.
- The classification is a class NAME, never a colour, and the viewer paints it from the CSS custom properties above — so **one
token stream serves light and dark** with no refetch when `prefers-color-scheme` flips, and the ramp lives only in
`ui/src/app.css`. `type` is a distinct class painted at plain ink: the colouring is near-monochrome and a type name is not one
of the four things it moves off plain ink.
- Every code token is split into identifier runs before it goes on the wire, so the graph's call-site overlay claims a token the
highlighter produced rather than re-cutting a line — which is what keeps a link landing on the callee's own name whatever
classifier produced rather than re-cutting a line — which is what keeps a link landing on the callee's own name whatever
boundaries a grammar chose, and keeps links working in the plain-text fallback.
- Measured on this machine (Shiki 4.4.3, JS regex engine, 3 000 lines cold): Python 3547 ms, Go 4357 ms, **TypeScript ~700 ms**
the TS TextMate grammar is 57× the cost of any other and the oniguruma-wasm engine would run it in ~120 ms. Slices are therefore
cached by content hash + range, so a re-render (resize, theme flip, stepping back through the trail) is a map lookup (< 10 ms);
a symbol-sized slice (~280 lines of TS) is ~50 ms cold. Phase 1 only ever requests one symbol's range.
- Single-file components (`.svelte`, `.vue`, `.astro`) have no grammar of their own; their `<script>` blocks — where every
indexed symbol in those files lives — are classified as TypeScript or JavaScript, exactly the delegation the extractors
already do. The surrounding markup, and the config formats with file-level extraction only (YAML, XML, Twig, properties),
render plain with their identifiers still split out, so links land there too.
- Measured on this machine, 3 000 lines cold: **TypeScript 2441 ms** (it was ~700 ms under Shiki, whose TS grammar cost 57×
every other one), Go ~30 ms, Python 2529 ms, and Rust/Ruby/PHP/C#/Swift 1427 ms. Slices are still cached by content hash +
range, so a re-render (resize, theme flip, stepping back through the trail) is a map lookup. Side-by-side parity screenshots
for the eight gate languages: `docs/design/cg57-highlighting-parity/`.
- No native modules; no runtime dependency for the UI itself; the CLI serves **`dist/viewer/`** over `node:http`, loopback only.
(Not `dist/ui/``src/ui/` is the engine's *terminal* ui and tsc already compiles it there; see `ui/README.md`.)