feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)
The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.
The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.
The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.
Three differences are not cosmetic:
* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
not as string. The call-site overlay refuses to claim a token classed string,
so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
The grammars disagree about whether `string` is a type_identifier or an
anonymous token inside a predefined_type, and TextMate scoped them
inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.
Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.
Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
This commit is contained in:
@@ -484,7 +484,7 @@ describe('showsBody', () => {
|
||||
|
||||
describe('client-side token decoding', () => {
|
||||
// The classification itself is the server's job (`src/ui-server/highlight/`,
|
||||
// real TextMate grammars); what is worth pinning here is the decoding — the
|
||||
// the engine's own tree-sitter parse); what is worth pinning here is the decoding — the
|
||||
// columns the call-site overlay matches against, and the plain fallback that
|
||||
// has to keep links working when no grammar covers a file.
|
||||
const CLASSES = ['other', 'ident', 'comment', 'string', 'keyword', 'number'];
|
||||
@@ -552,7 +552,7 @@ describe('client-side token decoding', () => {
|
||||
|
||||
it('keys a slice by real file line, not by offset into the slice', () => {
|
||||
const byLine = tokensByLine(['a();', 'b();'], 120, {
|
||||
engine: 'shiki',
|
||||
engine: 'tree-sitter',
|
||||
grammar: 'typescript',
|
||||
classes: CLASSES,
|
||||
lines: [
|
||||
|
||||
Reference in New Issue
Block a user