github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)

Browse Source 
The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.

The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.

The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.

Three differences are not cosmetic:

* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
  not as string. The call-site overlay refuses to claim a token classed string,
  so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
  The grammars disagree about whether `string` is a type_identifier or an
  anonymous token inside a predefined_type, and TextMate scoped them
  inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.

Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.

Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.
This commit is contained in:
Colby McHenry
2026-08-27 06:22:11 -05:00
parent 8ac0138940
commit ad91c8fdd8
32 changed files with 1084 additions and 1293 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/check-ui-build.mjs
+50 -23
View File
@@ -13,16 +13,17 @@
* where tsc puts the TERMINAL ui, so a mis-pointed outDir silently deletes
* modules the CLI requires at startup.
*
* The pruned TextMate grammars in dist/textmate/ are checked the same way and
* for the same reason: without them every file the viewer shows falls back to
* unhighlighted text, which looks like a styling bug rather than a missing
* build step.
* The tree-sitter grammars in dist/extraction/wasm/ are checked the same way
* and for the same reason. They are copied by `npm run copy-assets`, they are
* what both indexing and the viewer's syntax classification parse with, and
* their absence is survivable at runtime — source is served unhighlighted —
* which is exactly why it has to fail here: nothing downstream would complain.
*
* Usage: node scripts/check-ui-build.mjs [--root <dir>]
* --root directory holding dist/ (default: the repo root). The release
* bundler points this at its staging dir to verify the copy.
*/
import { existsSync, readFileSync, statSync } from 'node:fs';
import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs';
import { dirname, join, resolve, sep } from 'node:path';
import { fileURLToPath } from 'node:url';
@@ -95,36 +96,62 @@ for (const compiled of [join('bin', 'codegraph.js'), 'index.js', join('ui', 'shi
}
}
// The pruned syntax grammars (scripts/prune-grammars.mjs). Their absence is
// survivable at runtime — source is served unhighlighted — which is exactly why
// it has to fail here: nothing downstream would ever complain.
const textmateDir = join(root, 'dist', 'textmate');
const manifestPath = join(textmateDir, 'manifest.json');
if (!existsSync(manifestPath)) {
// The vendored tree-sitter grammars (`npm run copy-assets`). The viewer reads
// every file with the same grammar the engine indexed it with, so a missing
// wasm is both an extraction gap and a silently unhighlighted screen.
const wasmDir = join(root, 'dist', 'extraction', 'wasm');
/**
* The grammars the syntax classification is gated on — the eight languages
* CG-57 measured parity against, plus the two the TS family needs. Every one is
* vendored (see VENDORED_WASM_LANGS), so all of them must be in this directory
* rather than resolved out of node_modules.
*/
const GATE_GRAMMARS = [
'tree-sitter-typescript.wasm',
'tree-sitter-tsx.wasm',
'tree-sitter-javascript.wasm',
'tree-sitter-go.wasm',
'tree-sitter-python.wasm',
'tree-sitter-rust.wasm',
'tree-sitter-swift.wasm',
'tree-sitter-c_sharp.wasm',
'tree-sitter-ruby.wasm',
'tree-sitter-php.wasm',
];
if (!existsSync(wasmDir)) {
fail(
`missing ${manifestPath}`,
`missing ${wasmDir}`,
staged
? 'this bundle was assembled before the syntax grammars were added, or dist/textmate was not copied'
: 'run `npm run build:textmate` (it needs @shikijs/langs from devDependencies)'
? 'dist/extraction/wasm was not copied into the bundle — re-run scripts/build-bundle.sh'
: 'run `npm run copy-assets` (it copies src/extraction/wasm/*.wasm into dist/)'
);
}
const manifest = JSON.parse(readFileSync(manifestPath, 'utf8'));
const languages = Object.keys(manifest.languages ?? {});
if (languages.length === 0) fail('dist/textmate/manifest.json lists no languages');
// Against the source tree, the source directory IS the list — nothing to drift.
// Inside a staged bundle there is no src/, so the gate list carries it.
const expectedGrammars = new Set(GATE_GRAMMARS);
const srcWasmDir = join(root, 'src', 'extraction', 'wasm');
if (!staged && existsSync(srcWasmDir)) {
for (const name of readdirSync(srcWasmDir)) {
if (name.endsWith('.wasm')) expectedGrammars.add(name);
}
}
const grammarFiles = new Set(Object.values(manifest.languages).flat());
const missingGrammars = [...grammarFiles].filter(
(name) => !existsSync(join(textmateDir, `${name}.json`))
const missingGrammars = [...expectedGrammars].filter(
(name) => !existsSync(join(wasmDir, name))
);
if (missingGrammars.length > 0) {
fail(
`dist/textmate is missing ${missingGrammars.length} grammar file(s): ${missingGrammars.join(', ')}`,
'the prune step was interrupted or dist/textmate was copied incompletely'
`dist/extraction/wasm is missing ${missingGrammars.length} grammar(s): ${missingGrammars.join(', ')}`,
'the copy-assets step was interrupted or dist/extraction/wasm was copied incompletely'
);
}
const grammarCount = readdirSync(wasmDir).filter((n) => n.endsWith('.wasm')).length;
console.log(
`[check-ui-build] dist/viewer ok (index.html + ${assets} referenced asset(s)); ` +
`dist/textmate ok (${languages.length} languages, ${grammarFiles.size} grammars); dist/ engine intact`
`dist/extraction/wasm ok (${grammarCount} grammars); dist/ engine intact`
);
scripts/prune-grammars.mjs
-110
View File
@@ -1,110 +0,0 @@
#!/usr/bin/env node
/**
* Write the TextMate grammars the viewer needs into `dist/textmate/`.
*
* Shiki carries about 700 grammars, 11 MB of JSON. The engine indexes about 40
* languages. Shipping the other 660 to every user of a code-intelligence CLI is
* not a trade worth making, so `@shikijs/langs` stays a devDependency and this
* step copies out exactly the closure the viewer can reach: every grammar named
* in `src/ui-server/highlight/languages.ts`, plus every grammar those embed
* (`vue` needs html, css, typescript, json and four Vue-specific ones before it
* will highlight a single-file component).
*
* Run from `npm run build`, after `tsc`, because the language table is read
* from the compiled `dist/ui-server/highlight/languages.js` rather than being
* duplicated here — one source of truth for what ships.
*
* Output:
* dist/textmate/manifest.json grammar id -> files to load, deps first
* dist/textmate/<name>.json one TextMate grammar, verbatim
*/
import { createRequire } from 'node:module';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { fileURLToPath } from 'node:url';
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
const OUT = path.join(ROOT, 'dist', 'textmate');
const require = createRequire(import.meta.url);
function fail(message) {
process.stderr.write(`[prune-grammars] ${message}\n`);
process.exit(1);
}
const languagesModule = path.join(ROOT, 'dist', 'ui-server', 'highlight', 'languages.js');
if (!fs.existsSync(languagesModule)) {
fail(`${path.relative(ROOT, languagesModule)} is missing — run tsc before this script.`);
}
const { REQUIRED_GRAMMARS } = require(languagesModule);
if (!Array.isArray(REQUIRED_GRAMMARS) || REQUIRED_GRAMMARS.length === 0) {
fail('REQUIRED_GRAMMARS is empty — the language table did not compile as expected.');
}
const shikiVersion = JSON.parse(
fs.readFileSync(path.join(ROOT, 'node_modules', '@shikijs', 'langs', 'package.json'), 'utf-8')
).version;
/**
* Load one Shiki language module and return its registrations.
*
* The default export is already the flattened chain — embedded grammars first,
* the language itself last — which is exactly the order Shiki's registry needs
* to resolve `embeddedLangs`. Keeping that order is the whole reason the
* manifest stores a list rather than a single filename.
*/
async function loadChain(id) {
const mod = await import(`@shikijs/langs/${id}`);
const chain = mod.default;
if (!Array.isArray(chain) || chain.length === 0) {
fail(`@shikijs/langs/${id} did not export a grammar array.`);
}
return chain;
}
fs.rmSync(OUT, { recursive: true, force: true });
fs.mkdirSync(OUT, { recursive: true });
const manifest = { shikiVersion, languages: {} };
const written = new Map();
let bytes = 0;
for (const id of REQUIRED_GRAMMARS) {
let chain;
try {
chain = await loadChain(id);
} catch (err) {
fail(`could not load the ${id} grammar: ${err?.message ?? err}`);
}
const files = [];
for (const grammar of chain) {
// A chain can name the same dependency more than once (Vue reaches
// JavaScript four different ways). Registering it twice is wasted work and
// a confusing manifest; the FIRST occurrence is the one that keeps the
// dependencies-before-dependents ordering intact.
// `name` is the grammar's own id and is unique across the bundle, so two
// languages that embed html write (and share) exactly one html.json.
const file = grammar.name;
if (typeof file !== 'string' || !/^[\w.+-]+$/.test(file)) {
fail(`the ${id} chain contains a grammar with an unusable name: ${JSON.stringify(file)}`);
}
if (files.includes(file)) continue;
if (!written.has(file)) {
const json = JSON.stringify(grammar);
fs.writeFileSync(path.join(OUT, `${file}.json`), json);
written.set(file, json.length);
bytes += json.length;
}
files.push(file);
}
manifest.languages[id] = files;
}
fs.writeFileSync(path.join(OUT, 'manifest.json'), JSON.stringify(manifest, null, 2));
process.stdout.write(
`[prune-grammars] ${REQUIRED_GRAMMARS.length} languages -> ${written.size} grammars, ` +
`${(bytes / 1024 / 1024).toFixed(1)} MB in dist/textmate (shiki ${shikiVersion})\n`
);