fix: Lazy grammar loading and quantized embeddings to prevent V8 WASM OOM
Fixes #54 — `codegraph init -i` crashes with "Fatal process out of memory: Zone" on large codebases because all 16 tree-sitter WASM grammar modules were compiled upfront by V8, exhausting the WASM Zone allocator. Changes: - initGrammars() now only initializes the tree-sitter WASM runtime (Parser.init()), no longer eagerly loads all grammar files - New loadGrammarsForLanguages() loads only grammars for languages actually present in the project (e.g. a Dart project loads ~2-3 grammars instead of 16) - Orchestrator detects needed languages after file scan, before parsing begins - Embedding pipeline now uses quantized model (~67MB vs ~270MB) to further reduce WASM memory pressure when embeddings are enabled
This commit is contained in:
@@ -127,8 +127,10 @@ export class TextEmbedder {
|
||||
env.allowRemoteModels = false;
|
||||
}
|
||||
|
||||
// Load the pipeline
|
||||
// Load the pipeline with quantized model to reduce WASM memory pressure.
|
||||
// Quantized (int8/uint8) is ~4x smaller than FP32 with minimal quality loss.
|
||||
this.pipeline = await pipeline('feature-extraction', this.modelId, {
|
||||
quantized: true,
|
||||
progress_callback: this.showProgress
|
||||
? (progress: { status: string; file?: string; progress?: number }) => {
|
||||
if (progress.status === 'progress' && progress.file && progress.progress) {
|
||||
|
||||
Reference in New Issue
Block a user