fix: Lazy grammar loading and quantized embeddings to prevent V8 WASM OOM
Fixes #54 — `codegraph init -i` crashes with "Fatal process out of memory: Zone" on large codebases because all 16 tree-sitter WASM grammar modules were compiled upfront by V8, exhausting the WASM Zone allocator. Changes: - initGrammars() now only initializes the tree-sitter WASM runtime (Parser.init()), no longer eagerly loads all grammar files - New loadGrammarsForLanguages() loads only grammars for languages actually present in the project (e.g. a Dart project loads ~2-3 grammars instead of 16) - Orchestrator detects needed languages after file scan, before parsing begins - Embedding pipeline now uses quantized model (~67MB vs ~270MB) to further reduce WASM memory pressure when embeddings are enabled
This commit is contained in:
@@ -10,12 +10,13 @@ import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import { CodeGraph } from '../src';
|
||||
import { extractFromSource, scanDirectory, shouldIncludeFile } from '../src/extraction';
|
||||
import { detectLanguage, isLanguageSupported, getSupportedLanguages, initGrammars } from '../src/extraction/grammars';
|
||||
import { detectLanguage, isLanguageSupported, getSupportedLanguages, initGrammars, loadAllGrammars } from '../src/extraction/grammars';
|
||||
import { normalizePath } from '../src/utils';
|
||||
import { DEFAULT_CONFIG } from '../src/types';
|
||||
|
||||
beforeAll(async () => {
|
||||
await initGrammars();
|
||||
await loadAllGrammars();
|
||||
});
|
||||
|
||||
// Create a temporary directory for each test
|
||||
|
||||
Reference in New Issue
Block a user