* perf(index): ~34% faster fresh indexing, byte-identical graphs
Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed
the main thread as the critical path: per-row SQLite statement calls,
repeated import-resolution walks, and per-row FTS trigger firings, with
the parse workers ~75% idle behind it. This lands the semantics-preserving
tranche of fixes:
- Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments)
behind cached per-batch-size prepared statements; row order preserved,
so rowid-based resolution determinism (#1015) is unchanged.
- storeFileBundle: one transaction per file instead of four; nested
transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no
caller depended on nested rollback).
- Dedicated store-writer thread for the fresh-DB bulk path (bundles
applied in file order on a single writer connection; main thread does
no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1.
- Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load,
rebuild once at the end; crash inside the window self-heals on the
next open.
- Per-context memos for resolveImportPath/findExportedSymbol + a per-file
exported-symbol index, invalidated exactly where clearCaches() already
resets the resolver's own caches.
- Fast-init on completely fresh DBs (journal in memory, no fsync until
the index completes; interrupted init re-runs from scratch). Kill
switch: CODEGRAPH_NO_FAST_INIT=1.
- MaybeYield returns undefined on the not-due path so per-ref yield
checks stop paying a promise + microtask hop each.
- Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and
worker entry points.
Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series).
Graph dumps byte-identical across init, re-index, and sync paths; full
suite green (2403 passed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perf(resolution): parallel reference resolution with canonical admission
Fan resolution batches across a pool of read-only worker threads, each
hosting a full ReferenceResolver over its own SQLite connection; results
are admitted on the main thread in chunk order, so edge insertion order,
row cleanup, failure parking, and deferred post-pass queues are exactly
the sequence the single-threaded loop produces. Per-ref inputs match the
baseline because the sequential path already resolves each batch against
the state committed BEFORE that batch.
Validated byte-identical on excalidraw (pool forced on) and apache/dubbo
(4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical
graph dumps (91,495 nodes / 223,953 edges).
The pool only engages when total pending refs clear a threshold (default
150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1
to disable): measured on a ~58k-ref repo the workers' boot CPU contends
with resolution on the same cores and makes indexing slower, so small
repos keep the sequential path. When fast-init left the DB in
memory-journal mode, WAL is restored before resolution only when the pool
will run (readers + rollback-journal writers don't mix).
Also: sqlite adapter readOnly open support.
TreeCursor spine rewrite of the body walker was built, measured neutral
on real repos and equal in a 20k-child microbench (web-tree-sitter's
namedChild(i) is not quadratic in this binding), and rejected — per-node
JS<->WASM marshaling is the floor, which a traversal swap cannot remove.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parse timeouts are now judged by the worker's own clock: the base timer
only marks a job late (after a long synchronous store stall, Node runs the
timers phase before the poll phase, so the timer fired before an
already-delivered result was processed — killing workers over parses that
took milliseconds, even on 0-byte files); a result arriving before a 3×
hard-kill backstop is accepted, timed-out files are retried, and
CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are
pre-read once on the main thread and handed to every worker, so
spawns/respawns load grammars from memory instead of re-reading a
saturated disk.
Bulk indexing defers WAL auto-checkpointing for the whole run: the default
1000-page interval re-writes hot B-tree/FTS pages into the main DB file
over and over — ~95% of all disk I/O under throttled measurement. A
WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill
passes (never blocking the writer or the #850 watchdog heartbeat), pauses
the writer for a full backfill if the disk truly can't keep up, and folds
the WAL at the parse→resolution boundary so post-parse reads never page a
bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with
CODEGRAPH_WAL_VALVE_MB.
Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped
files (was 8); guava-scale completes in 7.6 min with a full graph where
v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The extension → language table was hardcoded, so a codebase using a
non-standard extension for a supported language (e.g. `.dota_lua` for Lua)
had those files silently skipped — no way to opt them in short of patching
the source.
Add an opt-in, project-scoped `codegraph.json` at the repo root:
{ "extensions": { ".dota_lua": "lua", ".tpl": "php" } }
Mappings merge on top of the built-in defaults and take precedence (so a
built-in can be re-pointed, e.g. `.h` → `cpp`). Absent or malformed config
is the zero-config default — byte-identical to prior behavior; an invalid
target language or unparseable file is warned-and-skipped, never fatal.
Implementation:
- New `src/project-config.ts` — `loadExtensionOverrides(rootDir)`, validated
against `isLanguageSupported`, mtime-cached per root.
- `detectLanguage` / `isSourceFile` gain an optional `overrides` arg
(omitting it is the existing behavior).
- Overrides threaded per-operation through every extraction call site
(scan/walk gates, git change-detection, grammar selection, extraction,
the file watcher), resolved from the project root — no process-global
state, so the multi-project daemon stays isolated. The parse worker
receives the resolved language in its message.
Tests: 13 new cases (unit, loader validation/normalization/caching, and a
full-index integration proving a custom-extension file is extracted while
the zero-config path indexes nothing). Worker path smoke-tested via the
built CLI.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add framework extract wiring plan
* feat(resolution): replace extractNodes with extract() returning nodes and references
* feat(resolution): add getApplicableFrameworks helper for per-language dispatch
* feat(django): emit route nodes and route->view references in extract()
* feat(flask,fastapi): emit route nodes and route->handler references
* feat(express): emit route nodes and route->handler references
* feat(laravel): emit route nodes and route->handler references
* feat(rails): emit route nodes and route->handler references
* feat(spring): emit route nodes and route->handler references
* feat(go): emit route nodes and route->handler references
* feat(rust): emit route nodes and route->handler references
* feat(aspnet): emit route nodes and route->handler references
* feat(swift,vapor): emit route nodes and route->handler references
* chore(react,svelte): migrate resolvers to extract() interface
* feat(extraction): run framework extractors after tree-sitter parse
* docs: document framework route extraction
* feat(strip-comments): add per-language comment stripper for framework extractors
Replaces comment characters and string-literal contents with spaces (not
removal) so source offsets stay valid for downstream regex match index ->
line number conversion. Handles Python triple-quoted docstrings, Ruby
=begin/=end, Rust nested block comments, and the standard //, #, /* */
forms across the supported languages.
This is consumed by framework extract() methods in a follow-up commit so
that commented-out / docstring routing examples don't surface as phantom
route nodes in the graph.
* feat(frameworks): strip comments before regex extraction (prevents phantom routes)
Pipes the per-language stripCommentsForRegex helper into every framework
extract() that scans raw source: django/flask/fastapi (python.ts),
express, laravel, rails, spring, go, rust, aspnet, vapor, plus
swiftui/uikit struct extraction in swift.ts.
Without this, examples like:
# path('/admin/', AdminPanel.as_view())
""" path('/users/', UserListView.as_view()) """
urlpatterns = [path('/real/', RealView.as_view())]
produced 3 phantom route nodes. Now only the real one is extracted.
Each framework gets a regression test in __tests__/frameworks.test.ts
asserting that line-, block-, docstring- and (where relevant)
heredoc-style commented-out routes do not surface as nodes.
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): surface lock-acquisition errors and silence Emscripten Aborted() spam
Two unrelated cosmetic but actively misleading bugs that surface when
the indexer is under load.
1) printIndexResult fell through to "No files found to index" whenever
the IndexResult had filesIndexed=0 AND filesErrored=0. The
lock-acquisition path returns success:false with a generic
"Could not acquire file lock" entry in result.errors[] (severity
'error'), but filesErrored counts only file-level parse failures,
so the user saw "No files found to index" — actively wrong.
Add a top-of-function check for the !success && !hasErrors case
that surfaces the first severity:'error' message instead.
2) parse-worker.ts let Emscripten's stderr "Aborted()" lines (plus
their "Build with -sASSERTIONS for more info" follow-ups) leak to
the parent's terminal whenever a WASM tree-sitter parser crashed
on a pathological file. Even after the JS layer caught and recovered,
the user saw dozens of `Aborted()` lines spammed to stderr. Install
a stderr filter at worker startup that drops only those specific
Emscripten internal lines; everything we log ourselves passes
through unchanged.
Verified live against ollama/ollama@v0.22.0:
- second concurrent `codegraph index` now shows
"Could not acquire file lock - another process may be indexing"
instead of "No files found to index"
- WASM-crash-prone re-index produced 0 Aborted() lines (down from 68+).
* fix(cli): null-safe error surfacing + clearer stderr-filter contract docs
Two reviewer findings on PR #128:
- printIndexResult: when result.success is false but result.errors
contains no severity:'error' entry (degenerate case but possible
if the result shape ever drifts), the find() returned undefined
and the previous if-guard fell through to the misleading
'No files found to index' branch. Now always surfaces a clear
failure message via clack.log.error, defaulting to 'Indexing
failed — no further details available' when no specific error
is in the errors list.
- parse-worker stderr filter: callback handling was already correct
but the comment didn't document it; expand the comment to spell
out the Writable-stream-contract obligation, the per-call match
semantics (split-chunk caveat), and the substring-exactness
trade-off so future readers understand the deliberate trade-offs.
Addresses C++ classes missing from .h files where extension-based detection defaults to 'c' language which has no class extraction support. Adds looksLikeCpp() heuristic that scans first 8KB for C++-specific patterns (namespace, class, template, access specifiers) to promote .h files to 'cpp' language when C++ constructs are detected. Ensures cpp grammar is loaded alongside c to handle potential .h promotion during parsing.
Implements file-size-based timeouts (base 10s + 10s per 100KB), more frequent worker recycling (250 files), and automatic retry logic for WASM memory corruption failures. Workers now crash immediately on memory errors to prevent cascading failures, with failed files automatically retried on fresh workers with clean heaps.
Offloads tree-sitter parsing to a dedicated worker thread, keeping the main thread unblocked so shimmer progress animations render smoothly during indexing. Refactors shimmer progress renderer into separate worker for consistent 50ms animation updates. Falls back to in-process parsing when worker compilation unavailable (e.g., tests).
andreinknvandGitHub