Colby McHenry
2ad836d935
The viewer's code block stops lexing with a hand-rolled dialect table and reads real TextMate grammars instead, run once in `/api/source`. Three things make that safe to depend on: * Highlighting never fails a request. A missing grammar, an oversized slice, an ESM import that did not resolve — every one of them answers `engine: 'plain'` with a reason and the source still goes out. * Identifiers survive whatever token boundaries a grammar chose. Every code token is split into identifier runs before it goes on the wire, so the graph's call-site overlay claims a token the highlighter produced rather than re-cutting the line. `assignRefs` now matches on a token's text rather than on the class a grammar gave it, so a language that scopes type names as `storage.type` still links. * The theme classifies rather than colours: its foregrounds are sentinels the server maps back to class names, and the viewer paints them from CSS custom properties — one token stream serves light and dark with no refetch, and the ramp lives only in app.css. Comments move from --ink-3 to a new --code-comment. --ink-3 measures 3.46:1 on paper and 3.00:1 on the hot-line tint, both under AA for 12.5px text; --code-comment is the smallest step along the same ramp that clears 4.5:1 on every background a code line can have, and stays quieter than the strings and numbers above it. Shipping: @shikijs/core and @shikijs/engine-javascript are runtime dependencies (no wasm, no native module); @shikijs/langs stays a devDependency and `npm run build:textmate` writes only the closure the engine's 40-odd languages reach — 56 grammars, 2.6 MB, against 11 MB for all 722. check-ui-build.mjs asserts the tree after every build and inside every release archive.
87 lines
3.5 KiB
TypeScript
87 lines
3.5 KiB
TypeScript
/**
|
|
* Finding and reading the pruned TextMate grammars on disk.
|
|
*
|
|
* Shiki ships 700-odd grammars; the engine indexes 40-odd languages. The build
|
|
* writes only the closure those 40 need — including the grammars they embed, so
|
|
* a `.vue` file still gets its `<script lang="ts">` — into `dist/textmate/`,
|
|
* and `@shikijs/langs` stays a devDependency that never reaches a user's disk.
|
|
* See `scripts/prune-grammars.mjs`.
|
|
*
|
|
* They are located the way `db/index.ts` finds `schema.sql` and `assets.ts`
|
|
* finds the viewer: relative to `__dirname`, never to `process.cwd()`, which is
|
|
* whatever directory the user happened to be standing in.
|
|
*
|
|
* `dist/textmate`, not `dist/highlight` — `src/ui-server/highlight/` is this
|
|
* module and tsc already owns `dist/ui-server/highlight/`. The same collision
|
|
* that put the viewer in `dist/viewer` rather than `dist/ui`.
|
|
*/
|
|
|
|
import * as fs from 'fs';
|
|
import * as path from 'path';
|
|
|
|
/** Overrides where the grammars are read from. For tests and for packagers. */
|
|
export const TEXTMATE_PATH_ENV = 'CODEGRAPH_TEXTMATE_PATH';
|
|
|
|
/** What `scripts/prune-grammars.mjs` writes beside the grammar files. */
|
|
export interface GrammarManifest {
|
|
/** Shiki version the grammars were pruned from — surfaced when one fails. */
|
|
shikiVersion: string;
|
|
/** Grammar id → the files to load, dependencies first, the grammar itself last. */
|
|
languages: Record<string, string[]>;
|
|
}
|
|
|
|
/**
|
|
* Candidate locations, most-specific first.
|
|
*
|
|
* 1. The `CODEGRAPH_TEXTMATE_PATH` override.
|
|
* 2. `<__dirname>/../../textmate` — the shipped layout
|
|
* (`dist/ui-server/highlight/` → `dist/textmate/`).
|
|
* 3. `<__dirname>/../../../dist/textmate` — running the TypeScript straight out
|
|
* of `src/` (vitest, tsx), where `__dirname` is `src/ui-server/highlight/`.
|
|
*/
|
|
export function grammarDirCandidates(): string[] {
|
|
const override = process.env[TEXTMATE_PATH_ENV]?.trim();
|
|
const candidates = [
|
|
path.join(__dirname, '..', '..', 'textmate'),
|
|
path.join(__dirname, '..', '..', '..', 'dist', 'textmate'),
|
|
];
|
|
return override ? [path.resolve(override), ...candidates] : candidates;
|
|
}
|
|
|
|
/**
|
|
* The grammar directory and its manifest, or null when the build did not run.
|
|
*
|
|
* Null is a normal outcome, not an error: a source checkout that has only had
|
|
* `tsc` run against it has no `dist/textmate`, and the right answer there is
|
|
* plain text, not a 500 on a request for source.
|
|
*/
|
|
export function loadManifest(): { dir: string; manifest: GrammarManifest } | null {
|
|
for (const dir of grammarDirCandidates()) {
|
|
try {
|
|
const raw = fs.readFileSync(path.join(dir, 'manifest.json'), 'utf-8');
|
|
const manifest = JSON.parse(raw) as GrammarManifest;
|
|
if (manifest && typeof manifest === 'object' && manifest.languages) return { dir, manifest };
|
|
} catch {
|
|
// Not here — try the next candidate.
|
|
}
|
|
}
|
|
return null;
|
|
}
|
|
|
|
/**
|
|
* Read the grammar registrations one language needs, dependencies first.
|
|
*
|
|
* Shiki resolves a grammar's `embeddedLangs` against what is already in its
|
|
* registry, so the order the manifest records matters: `vue` must arrive after
|
|
* the `html`, `css` and `typescript` it embeds, or the embedded blocks come
|
|
* back unhighlighted.
|
|
*/
|
|
export function readGrammarChain(dir: string, manifest: GrammarManifest, id: string): unknown[] {
|
|
const files = manifest.languages[id];
|
|
if (!files) return [];
|
|
return files.map((file) => {
|
|
const full = path.join(dir, `${file}.json`);
|
|
return JSON.parse(fs.readFileSync(full, 'utf-8')) as unknown;
|
|
});
|
|
}
|