codegraph/docs/design/cg57-highlighting-parity.md

Highlighting parity: Shiki → the engine's own tree-sitter parse (CG-57)

The viewer's code block used to be classified by a second highlighter — Shiki with 56 pruned TextMate grammars shipped in dist/textmate/ — over source the engine had already parsed with a real grammar. CG-57 takes the classification off that tree instead. This file records what the swap changed, measured rather than asserted, so nobody has to re-derive it from a diff.

Screenshots, one per language, before on the left and after on the right, same stylesheet: cg57-highlighting-parity/ — typescript.png, go.png, python.png, rust.png, swift.png, csharp.png, ruby.png, php.png.

What it costs

3 000 lines, cold, dev Mac (M-series), parse + classify + wire:

	TypeScript	Go	Python	Rust	Swift	C#	Ruby	PHP
Shiki + TextMate	~700 ms	43–57 ms	35–47 ms	—	—	—	—	—
Engine tree-sitter	24–41 ms	~30 ms	25–29 ms	18–19 ms	25–27 ms	20–25 ms	14–16 ms	20–22 ms

The task's budget was < 100 ms per 3 000-line file warm; every language clears it cold. TypeScript is the number that mattered: its TextMate grammar was 5–7× every other one and the cost was regex execution, not compilation, so nothing about the old module could have fixed it. The slice cache still exists — a re-render (resize, theme flip, stepping back through the trail) should cost nothing at all, and the whole-file view pages the same file repeatedly.

What it changes on screen

Per-character comparison over ~40 lines of realistic source per language, counting only non-whitespace characters, and treating ident / other / type as one bucket because all three paint at plain ink:

language	painted identically	what moved
TypeScript	91.3%	33 def, 40 interpolation chars now code, 2 punctuation
Go	91.5%	37 built-in type words, 13 def
Python	93.2%	26 def, 15 keyword (is not, __future__)
Rust	96.3%	21 def, 3 keyword
Swift	93.3%	18 def, 14 keyword (throws/rethrows)
C#	88.3%	30 built-in type words, 23 def, 31 interpolation chars now code
Ruby	83.8%	23 def, 35 interpolation chars now code, 14 symbol literals, 3 keyword
PHP	85.9%	33 built-in type words, 29 def, 15 phpdoc tag chars, 12 keyword

Every remaining difference is one of five deliberate categories:

1. ident → def. The definition's own name now carries weight 600, everywhere rather than only on the line the Symbol view opened at. It comes from the extractors' own definition tables (functionTypes, classTypes, methodTypes, …) plus each language's nameField, so it cannot drift from what indexing considers a definition.
2. string → code, inside an interpolation. A template literal's ${…}, an f-string's {…}, Ruby's #{…} and C#'s $"{…}" are classified as code. This is the one difference that is not cosmetic: the call-site overlay deliberately refuses to claim a token classed string, so calls inside interpolated strings now link and did not before.
3. keyword → type, on built-in type words. string, int, u32, void. The grammars disagree with each other about what a built-in type is — tree-sitter-go calls string a type_identifier, tree-sitter-typescript wraps it in a predefined_type whose child is an anonymous token spelled string — and TextMate scoped them inconsistently too (plain in TypeScript, storage.type in Go). They now all paint at plain ink, like a user-defined type name, in every language.
4. Keyword-set corrections. Python's is not, Rust's and Swift's modifiers, and Ruby's new (which is a method, not a keyword — TextMate's keyword.operator.new matched it anyway).
5. keyword → comment, on phpdoc tags. @var and friends recede with the comment they are in, which is what the near-monochrome ramp asks for.

What is no longer highlighted

Nine formats have extraction but no tree-sitter grammar. Three of them — .svelte, .vue, .astro — are classified through their <script> blocks with TypeScript or JavaScript, the same delegation the extractors do, so every symbol the engine indexed in those files is highlighted and the surrounding markup is not. The other six (Liquid, Razor, YAML, Twig, XML, .properties) render plain, where Shiki had grammars for them.

That is a real, deliberate loss, and it is the alternative to a worse one. tree-sitter-wasms ships an html grammar that would cover most of them, but the ABI-13 builds in that package are the known cause of a shared-WASM-heap corruption that silently drops edges for every other language in the same process (see VENDORED_WASM_LANGS in src/extraction/grammars.ts), and the viewer runs in a process someone leaves open all day. Adding unvetted grammars to buy tag colouring on config files is not a trade worth making. Identifiers are still split out on those files, so the graph's call-site links land exactly as they do everywhere else — highlighting is the part that degrades, never the linking.

Reproducing this

There is no committed harness: the "before" side needs the deleted Shiki module. Rebuild it from the last commit that had it —

git worktree add /tmp/cg48-baseline <ref-with-shiki>
ln -s "$PWD/node_modules" /tmp/cg48-baseline/node_modules   # @shikijs/* must still be installed
( cd /tmp/cg48-baseline && npx tsc && node scripts/prune-grammars.mjs )

— then run both dist/ui-server/highlight/index.js modules over the same lines and compare classes[id] per character. The screenshots were rendered from the same two token streams through the viewer's own token CSS at --force-device-scale-factor=2.