feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)
The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.
The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.
The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.
Three differences are not cosmetic:
* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
not as string. The call-site overlay refuses to claim a token classed string,
so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
The grammars disagree about whether `string` is a type_identifier or an
anonymous token inside a predefined_type, and TextMate scoped them
inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.
Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.
Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
@@ -0,0 +1,95 @@
|
||||
# Highlighting parity: Shiki → the engine's own tree-sitter parse (CG-57)
|
||||
|
||||
The viewer's code block used to be classified by a second highlighter — Shiki with 56 pruned
|
||||
TextMate grammars shipped in `dist/textmate/` — over source the engine had already parsed with a
|
||||
real grammar. CG-57 takes the classification off that tree instead. This file records what the swap
|
||||
changed, measured rather than asserted, so nobody has to re-derive it from a diff.
|
||||
|
||||
Screenshots, one per language, before on the left and after on the right, same stylesheet:
|
||||
[`cg57-highlighting-parity/`](./cg57-highlighting-parity/) — `typescript.png`, `go.png`,
|
||||
`python.png`, `rust.png`, `swift.png`, `csharp.png`, `ruby.png`, `php.png`.
|
||||
|
||||
## What it costs
|
||||
|
||||
3 000 lines, cold, dev Mac (M-series), parse + classify + wire:
|
||||
|
||||
| | TypeScript | Go | Python | Rust | Swift | C# | Ruby | PHP |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| Shiki + TextMate | ~700 ms | 43–57 ms | 35–47 ms | — | — | — | — | — |
|
||||
| Engine tree-sitter | 24–41 ms | ~30 ms | 25–29 ms | 18–19 ms | 25–27 ms | 20–25 ms | 14–16 ms | 20–22 ms |
|
||||
|
||||
The task's budget was **< 100 ms per 3 000-line file warm**; every language clears it *cold*.
|
||||
TypeScript is the number that mattered: its TextMate grammar was 5–7× every other one and the cost
|
||||
was regex *execution*, not compilation, so nothing about the old module could have fixed it. The
|
||||
slice cache still exists — a re-render (resize, theme flip, stepping back through the trail) should
|
||||
cost nothing at all, and the whole-file view pages the same file repeatedly.
|
||||
|
||||
## What it changes on screen
|
||||
|
||||
Per-character comparison over ~40 lines of realistic source per language, counting only
|
||||
non-whitespace characters, and treating `ident` / `other` / `type` as one bucket because all three
|
||||
paint at plain ink:
|
||||
|
||||
| language | painted identically | what moved |
|
||||
|---|---|---|
|
||||
| TypeScript | 91.3% | 33 `def`, 40 interpolation chars now code, 2 punctuation |
|
||||
| Go | 91.5% | 37 built-in type words, 13 `def` |
|
||||
| Python | 93.2% | 26 `def`, 15 keyword (`is not`, `__future__`) |
|
||||
| Rust | 96.3% | 21 `def`, 3 keyword |
|
||||
| Swift | 93.3% | 18 `def`, 14 keyword (`throws`/`rethrows`) |
|
||||
| C# | 88.3% | 30 built-in type words, 23 `def`, 31 interpolation chars now code |
|
||||
| Ruby | 83.8% | 23 `def`, 35 interpolation chars now code, 14 symbol literals, 3 keyword |
|
||||
| PHP | 85.9% | 33 built-in type words, 29 `def`, 15 phpdoc tag chars, 12 keyword |
|
||||
|
||||
Every remaining difference is one of five deliberate categories:
|
||||
|
||||
1. **`ident` → `def`.** The definition's own name now carries weight 600, everywhere rather than
|
||||
only on the line the Symbol view opened at. It comes from the extractors' own definition tables
|
||||
(`functionTypes`, `classTypes`, `methodTypes`, …) plus each language's `nameField`, so it cannot
|
||||
drift from what indexing considers a definition.
|
||||
2. **`string` → code, inside an interpolation.** A template literal's `${…}`, an f-string's `{…}`,
|
||||
Ruby's `#{…}` and C#'s `$"{…}"` are classified as code. This is the one difference that is not
|
||||
cosmetic: the call-site overlay deliberately refuses to claim a token classed `string`, so
|
||||
**calls inside interpolated strings now link and did not before.**
|
||||
3. **`keyword` → `type`, on built-in type words.** `string`, `int`, `u32`, `void`. The grammars
|
||||
disagree with each other about what a built-in type is — tree-sitter-go calls `string` a
|
||||
`type_identifier`, tree-sitter-typescript wraps it in a `predefined_type` whose child is an
|
||||
anonymous token spelled `string` — and TextMate scoped them inconsistently too (plain in
|
||||
TypeScript, `storage.type` in Go). They now all paint at plain ink, like a user-defined type
|
||||
name, in every language.
|
||||
4. **Keyword-set corrections.** Python's `is not`, Rust's and Swift's modifiers, and Ruby's `new`
|
||||
(which is a method, not a keyword — TextMate's `keyword.operator.new` matched it anyway).
|
||||
5. **`keyword` → `comment`, on phpdoc tags.** `@var` and friends recede with the comment they are
|
||||
in, which is what the near-monochrome ramp asks for.
|
||||
|
||||
## What is no longer highlighted
|
||||
|
||||
Nine formats have extraction but no tree-sitter grammar. Three of them — `.svelte`, `.vue`,
|
||||
`.astro` — are classified through their `<script>` blocks with TypeScript or JavaScript, the same
|
||||
delegation the extractors do, so every symbol the engine indexed in those files is highlighted and
|
||||
the surrounding markup is not. The other six (Liquid, Razor, YAML, Twig, XML, `.properties`) render
|
||||
plain, where Shiki had grammars for them.
|
||||
|
||||
That is a real, deliberate loss, and it is the alternative to a worse one. `tree-sitter-wasms`
|
||||
ships an `html` grammar that would cover most of them, but the ABI-13 builds in that package are
|
||||
the known cause of a shared-WASM-heap corruption that silently drops edges for *every other*
|
||||
language in the same process (see `VENDORED_WASM_LANGS` in `src/extraction/grammars.ts`), and the
|
||||
viewer runs in a process someone leaves open all day. Adding unvetted grammars to buy tag colouring
|
||||
on config files is not a trade worth making. Identifiers are still split out on those files, so the
|
||||
graph's call-site links land exactly as they do everywhere else — highlighting is the part that
|
||||
degrades, never the linking.
|
||||
|
||||
## Reproducing this
|
||||
|
||||
There is no committed harness: the "before" side needs the deleted Shiki module. Rebuild it from
|
||||
the last commit that had it —
|
||||
|
||||
```
|
||||
git worktree add /tmp/cg48-baseline <ref-with-shiki>
|
||||
ln -s "$PWD/node_modules" /tmp/cg48-baseline/node_modules # @shikijs/* must still be installed
|
||||
( cd /tmp/cg48-baseline && npx tsc && node scripts/prune-grammars.mjs )
|
||||
```
|
||||
|
||||
— then run both `dist/ui-server/highlight/index.js` modules over the same lines and compare
|
||||
`classes[id]` per character. The screenshots were rendered from the same two token streams through
|
||||
the viewer's own token CSS at `--force-device-scale-factor=2`.
|
||||
|
After Width: | Height: | Size: 203 KiB |
|
After Width: | Height: | Size: 194 KiB |
|
After Width: | Height: | Size: 216 KiB |
|
After Width: | Height: | Size: 188 KiB |
|
After Width: | Height: | Size: 172 KiB |
|
After Width: | Height: | Size: 207 KiB |
|
After Width: | Height: | Size: 153 KiB |
|
After Width: | Height: | Size: 254 KiB |
@@ -321,22 +321,32 @@ with `src/index.ts` selected, 15 links and 4 dimmed boxes, matching the canvas).
|
||||
- Svelte 5 (≥ 5.25) + Vite (workspace `ui/`), Svelte Flow `@xyflow/svelte` ^1.6 for the Map and Flow canvases only (custom nodes/edges,
|
||||
hidden handles for port spreading, local selection state — the pattern in docker-app's `StackGraph.svelte`); `@dagrejs/dagre` only as a
|
||||
fallback if crossing quality demands it (never ELK). Symbol view = DOM + one SVG overlay (`ResizeObserver` re-layout).
|
||||
- Shiki (JavaScript regex engine, lazy grammars, custom near-monochrome theme as in §2.2) server-side in `/api/source`; tree-sitter-derived
|
||||
tokens replace it in phase 3.
|
||||
- *As built (CG-43).* `@shikijs/core` + `@shikijs/engine-javascript` are runtime dependencies (~5 MB installed, no wasm, no native
|
||||
module); `@shikijs/langs` is a **devDependency** and `npm run build:textmate` (`scripts/prune-grammars.mjs`) writes only the
|
||||
closure the engine's 40-odd languages reach — 56 grammars, 2.6 MB — into **`dist/textmate/`**, checked by `scripts/check-ui-build.mjs`.
|
||||
Shipping all 722 grammars would have been 11 MB.
|
||||
- The theme classifies rather than colours: its foregrounds are sentinels the server maps to class names (`comment`, `string`,
|
||||
`keyword`, `number`, `ident`, `other`), and the viewer paints them from the CSS custom properties above — so **one token stream
|
||||
serves light and dark** with no refetch when `prefers-color-scheme` flips, and the ramp lives only in `ui/src/app.css`.
|
||||
- Syntax classification comes off **the engine's own tree-sitter parse** — no highlighter dependency, no second grammar set.
|
||||
- *As built (CG-43, replaced in CG-57).* The first cut ran Shiki with 56 pruned TextMate grammars in `dist/textmate/`. That is
|
||||
gone: `@shikijs/*` is off the dependency list, `scripts/prune-grammars.mjs` and `npm run build:textmate` are deleted, and
|
||||
`scripts/check-ui-build.mjs` now asserts the tree-sitter grammars in `dist/extraction/wasm/` instead. A `.ts` file is read by
|
||||
exactly the grammar that decided what its symbols are, so the viewer and the graph can never disagree about it.
|
||||
- Eight token classes on the wire: `comment`, `string`, `number`, `keyword`, `type`, `def`, `ident`, `other`. Rules, not scope
|
||||
tables — a node whose type mentions `comment` is a comment; inside a string every leaf is string *except* below an
|
||||
interpolation, where code resumes (so `${user.name()}` still links); an **anonymous** leaf is a keyword when its text is a bare
|
||||
word and punctuation otherwise; a **named** leaf is an identifier, a type name, or — from the extractors' own definition
|
||||
tables — the name a definition declares. `punct` is folded into `other`: they paint identically and splitting them would
|
||||
roughly double the token count on a dense line.
|
||||
- The classification is a class NAME, never a colour, and the viewer paints it from the CSS custom properties above — so **one
|
||||
token stream serves light and dark** with no refetch when `prefers-color-scheme` flips, and the ramp lives only in
|
||||
`ui/src/app.css`. `type` is a distinct class painted at plain ink: the colouring is near-monochrome and a type name is not one
|
||||
of the four things it moves off plain ink.
|
||||
- Every code token is split into identifier runs before it goes on the wire, so the graph's call-site overlay claims a token the
|
||||
highlighter produced rather than re-cutting a line — which is what keeps a link landing on the callee's own name whatever
|
||||
classifier produced rather than re-cutting a line — which is what keeps a link landing on the callee's own name whatever
|
||||
boundaries a grammar chose, and keeps links working in the plain-text fallback.
|
||||
- Measured on this machine (Shiki 4.4.3, JS regex engine, 3 000 lines cold): Python 35–47 ms, Go 43–57 ms, **TypeScript ~700 ms** —
|
||||
the TS TextMate grammar is 5–7× the cost of any other and the oniguruma-wasm engine would run it in ~120 ms. Slices are therefore
|
||||
cached by content hash + range, so a re-render (resize, theme flip, stepping back through the trail) is a map lookup (< 10 ms);
|
||||
a symbol-sized slice (~280 lines of TS) is ~50 ms cold. Phase 1 only ever requests one symbol's range.
|
||||
- Single-file components (`.svelte`, `.vue`, `.astro`) have no grammar of their own; their `<script>` blocks — where every
|
||||
indexed symbol in those files lives — are classified as TypeScript or JavaScript, exactly the delegation the extractors
|
||||
already do. The surrounding markup, and the config formats with file-level extraction only (YAML, XML, Twig, properties),
|
||||
render plain with their identifiers still split out, so links land there too.
|
||||
- Measured on this machine, 3 000 lines cold: **TypeScript 24–41 ms** (it was ~700 ms under Shiki, whose TS grammar cost 5–7×
|
||||
every other one), Go ~30 ms, Python 25–29 ms, and Rust/Ruby/PHP/C#/Swift 14–27 ms. Slices are still cached by content hash +
|
||||
range, so a re-render (resize, theme flip, stepping back through the trail) is a map lookup. Side-by-side parity screenshots
|
||||
for the eight gate languages: `docs/design/cg57-highlighting-parity/`.
|
||||
- No native modules; no runtime dependency for the UI itself; the CLI serves **`dist/viewer/`** over `node:http`, loopback only.
|
||||
(Not `dist/ui/` — `src/ui/` is the engine's *terminal* ui and tsc already compiles it there; see `ui/README.md`.)
|
||||
|
||||
|
||||