Colby McHenry
ad91c8fdd8
The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.
The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.
The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.
Three differences are not cosmetic:
* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
not as string. The call-site overlay refuses to claim a token classed string,
so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
The grammars disagree about whether `string` is a type_identifier or an
anonymous token inside a predefined_type, and TextMate scoped them
inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.
Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.
Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
96 lines
5.8 KiB
Markdown
96 lines
5.8 KiB
Markdown
# Highlighting parity: Shiki → the engine's own tree-sitter parse (CG-57)
|
||
|
||
The viewer's code block used to be classified by a second highlighter — Shiki with 56 pruned
|
||
TextMate grammars shipped in `dist/textmate/` — over source the engine had already parsed with a
|
||
real grammar. CG-57 takes the classification off that tree instead. This file records what the swap
|
||
changed, measured rather than asserted, so nobody has to re-derive it from a diff.
|
||
|
||
Screenshots, one per language, before on the left and after on the right, same stylesheet:
|
||
[`cg57-highlighting-parity/`](./cg57-highlighting-parity/) — `typescript.png`, `go.png`,
|
||
`python.png`, `rust.png`, `swift.png`, `csharp.png`, `ruby.png`, `php.png`.
|
||
|
||
## What it costs
|
||
|
||
3 000 lines, cold, dev Mac (M-series), parse + classify + wire:
|
||
|
||
| | TypeScript | Go | Python | Rust | Swift | C# | Ruby | PHP |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| Shiki + TextMate | ~700 ms | 43–57 ms | 35–47 ms | — | — | — | — | — |
|
||
| Engine tree-sitter | 24–41 ms | ~30 ms | 25–29 ms | 18–19 ms | 25–27 ms | 20–25 ms | 14–16 ms | 20–22 ms |
|
||
|
||
The task's budget was **< 100 ms per 3 000-line file warm**; every language clears it *cold*.
|
||
TypeScript is the number that mattered: its TextMate grammar was 5–7× every other one and the cost
|
||
was regex *execution*, not compilation, so nothing about the old module could have fixed it. The
|
||
slice cache still exists — a re-render (resize, theme flip, stepping back through the trail) should
|
||
cost nothing at all, and the whole-file view pages the same file repeatedly.
|
||
|
||
## What it changes on screen
|
||
|
||
Per-character comparison over ~40 lines of realistic source per language, counting only
|
||
non-whitespace characters, and treating `ident` / `other` / `type` as one bucket because all three
|
||
paint at plain ink:
|
||
|
||
| language | painted identically | what moved |
|
||
|---|---|---|
|
||
| TypeScript | 91.3% | 33 `def`, 40 interpolation chars now code, 2 punctuation |
|
||
| Go | 91.5% | 37 built-in type words, 13 `def` |
|
||
| Python | 93.2% | 26 `def`, 15 keyword (`is not`, `__future__`) |
|
||
| Rust | 96.3% | 21 `def`, 3 keyword |
|
||
| Swift | 93.3% | 18 `def`, 14 keyword (`throws`/`rethrows`) |
|
||
| C# | 88.3% | 30 built-in type words, 23 `def`, 31 interpolation chars now code |
|
||
| Ruby | 83.8% | 23 `def`, 35 interpolation chars now code, 14 symbol literals, 3 keyword |
|
||
| PHP | 85.9% | 33 built-in type words, 29 `def`, 15 phpdoc tag chars, 12 keyword |
|
||
|
||
Every remaining difference is one of five deliberate categories:
|
||
|
||
1. **`ident` → `def`.** The definition's own name now carries weight 600, everywhere rather than
|
||
only on the line the Symbol view opened at. It comes from the extractors' own definition tables
|
||
(`functionTypes`, `classTypes`, `methodTypes`, …) plus each language's `nameField`, so it cannot
|
||
drift from what indexing considers a definition.
|
||
2. **`string` → code, inside an interpolation.** A template literal's `${…}`, an f-string's `{…}`,
|
||
Ruby's `#{…}` and C#'s `$"{…}"` are classified as code. This is the one difference that is not
|
||
cosmetic: the call-site overlay deliberately refuses to claim a token classed `string`, so
|
||
**calls inside interpolated strings now link and did not before.**
|
||
3. **`keyword` → `type`, on built-in type words.** `string`, `int`, `u32`, `void`. The grammars
|
||
disagree with each other about what a built-in type is — tree-sitter-go calls `string` a
|
||
`type_identifier`, tree-sitter-typescript wraps it in a `predefined_type` whose child is an
|
||
anonymous token spelled `string` — and TextMate scoped them inconsistently too (plain in
|
||
TypeScript, `storage.type` in Go). They now all paint at plain ink, like a user-defined type
|
||
name, in every language.
|
||
4. **Keyword-set corrections.** Python's `is not`, Rust's and Swift's modifiers, and Ruby's `new`
|
||
(which is a method, not a keyword — TextMate's `keyword.operator.new` matched it anyway).
|
||
5. **`keyword` → `comment`, on phpdoc tags.** `@var` and friends recede with the comment they are
|
||
in, which is what the near-monochrome ramp asks for.
|
||
|
||
## What is no longer highlighted
|
||
|
||
Nine formats have extraction but no tree-sitter grammar. Three of them — `.svelte`, `.vue`,
|
||
`.astro` — are classified through their `<script>` blocks with TypeScript or JavaScript, the same
|
||
delegation the extractors do, so every symbol the engine indexed in those files is highlighted and
|
||
the surrounding markup is not. The other six (Liquid, Razor, YAML, Twig, XML, `.properties`) render
|
||
plain, where Shiki had grammars for them.
|
||
|
||
That is a real, deliberate loss, and it is the alternative to a worse one. `tree-sitter-wasms`
|
||
ships an `html` grammar that would cover most of them, but the ABI-13 builds in that package are
|
||
the known cause of a shared-WASM-heap corruption that silently drops edges for *every other*
|
||
language in the same process (see `VENDORED_WASM_LANGS` in `src/extraction/grammars.ts`), and the
|
||
viewer runs in a process someone leaves open all day. Adding unvetted grammars to buy tag colouring
|
||
on config files is not a trade worth making. Identifiers are still split out on those files, so the
|
||
graph's call-site links land exactly as they do everywhere else — highlighting is the part that
|
||
degrades, never the linking.
|
||
|
||
## Reproducing this
|
||
|
||
There is no committed harness: the "before" side needs the deleted Shiki module. Rebuild it from
|
||
the last commit that had it —
|
||
|
||
```
|
||
git worktree add /tmp/cg48-baseline <ref-with-shiki>
|
||
ln -s "$PWD/node_modules" /tmp/cg48-baseline/node_modules # @shikijs/* must still be installed
|
||
( cd /tmp/cg48-baseline && npx tsc && node scripts/prune-grammars.mjs )
|
||
```
|
||
|
||
— then run both `dist/ui-server/highlight/index.js` modules over the same lines and compare
|
||
`classes[id]` per character. The screenshots were rendered from the same two token streams through
|
||
the viewer's own token CSS at `--force-device-scale-factor=2`.
|