The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.
The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.
The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.
Three differences are not cosmetic:
* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
not as string. The call-site overlay refuses to claim a token classed string,
so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
The grammars disagree about whether `string` is a type_identifier or an
anonymous token inside a predefined_type, and TextMate scoped them
inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.
Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.
Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
The viewer's code block stops lexing with a hand-rolled dialect table and
reads real TextMate grammars instead, run once in `/api/source`.
Three things make that safe to depend on:
* Highlighting never fails a request. A missing grammar, an oversized
slice, an ESM import that did not resolve — every one of them answers
`engine: 'plain'` with a reason and the source still goes out.
* Identifiers survive whatever token boundaries a grammar chose. Every
code token is split into identifier runs before it goes on the wire, so
the graph's call-site overlay claims a token the highlighter produced
rather than re-cutting the line. `assignRefs` now matches on a token's
text rather than on the class a grammar gave it, so a language that
scopes type names as `storage.type` still links.
* The theme classifies rather than colours: its foregrounds are sentinels
the server maps back to class names, and the viewer paints them from
CSS custom properties — one token stream serves light and dark with no
refetch, and the ramp lives only in app.css.
Comments move from --ink-3 to a new --code-comment. --ink-3 measures
3.46:1 on paper and 3.00:1 on the hot-line tint, both under AA for 12.5px
text; --code-comment is the smallest step along the same ramp that clears
4.5:1 on every background a code line can have, and stays quieter than
the strings and numbers above it.
Shipping: @shikijs/core and @shikijs/engine-javascript are runtime
dependencies (no wasm, no native module); @shikijs/langs stays a
devDependency and `npm run build:textmate` writes only the closure the
engine's 40-odd languages reach — 56 grammars, 2.6 MB, against 11 MB for
all 722. check-ui-build.mjs asserts the tree after every build and inside
every release archive.
Colby McHenry