`isGeneratedFile` was path-only, but Go's own convention is a CONTENT
marker (`// Code generated by <tool>. DO NOT EDIT.`), not a filename one.
A Go monorepo with generated CRUD in ordinarily-named files sitting beside
hand-written use-cases was therefore invisible to every generated-file
down-rank in the codebase — that is #1500.
Measured on kubernetes/client-go (2,453 Go files): the canonical banner
appears in 2,001 of them, the path check flags 0, the new content check
flags exactly those 2,001 — no false positives, no misses.
Design: decide at INDEX time (content is already in memory for parsing),
persist on `files.generated`, read from the DB. Explore never reads file
headers per request.
- `hasGeneratedHeader(content)` recognizes the standard banners — Go's,
protoc's, `@generated`, `<auto-generated>`, Thrift, OpenAPI Generator,
FlatBuffers, bindgen, ANTLR. Precision-first and fenced three ways: an
8KB/60-line header window, a comment-line requirement (leader or open
block comment), and markers tight enough that prose can't trip them. A
generator's own source, holding the banner as a string constant in its
body, is not flagged; neither is this module itself (pinned by test).
- `isGeneratedFile(path)` is unchanged — cheap, sync, still the fallback.
- Schema v9 adds `files.generated` + a PARTIAL index. DDL only, no
backfill: the flag derives from content the migration cannot see, so
rows stay 0 until a re-index and every reader unions the flag with the
path check — an un-migrated index keeps pre-#1500 behavior rather than
regressing. Re-index required; noted in the CHANGELOG.
- `generatedPredicateFor(paths)` gives ranking a bounded probe + O(1)
lookups. Bounded, not cached: no invalidation, so a ranking call can
never serve a verdict the last sync already replaced. Wired into explore
ranking, findSymbolMatches, findAllSymbols, search (MCP + CLI), the
context formatter, and the dominant-file/route-file hygiene filters.
Cost (acceptance bar was no measurable index-time regression): a single
unanchored `/generat/i` test over the header rejects ~every hand-written
file before any line splitting. 4.6 µs/file on client-go (worst case —
82% generated). End-to-end `codegraph init` on client-go, n=3 alternating
arms: 5.73s median with detection vs 5.76s path-only baseline; the arms
cross over between runs, so the difference is inside run-to-run noise.
Scope note: generated status remains a stable TIEBREAK at equal score,
exactly where it was. Making it a strong negative signal is CG-10, which
this unblocks by making the signal correct and available.
Two pre-existing tests hard-coded schema version 8; both now track
CURRENT_SCHEMA_VERSION (or the migration table) so future migrations
don't require editing them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore: ignore .kommandr/ directory
* fix(sync): resolve cross-file refs when an edit adds or removes the satisfying symbol (#1240)
Incremental sync scoped reference resolution to the changed files' own
refs, and a completed pass deleted every ref it failed to resolve — so
a symbol change in one file could never repair references in UNCHANGED
files, in either direction, until a full re-index:
- New-export case: a.ts imports/calls `greet` before b.ts defines it.
The failed refs were deleted at index time; when b.ts later gained
`greet`, nothing revisited a.ts — the calls/imports edges stayed
missing while status reported a clean index.
- Removal case: when a re-index (or file deletion) dropped a symbol,
the incoming edges cascade-deleted and the callers — whose resolved
refs had been consumed — never got a chance to rebind to an
alternative definition or reconnect when the symbol returned.
Fix, sharing one lifecycle:
- Schema v8: unresolved_refs gains status ('pending'/'failed') and
name_tail (last dotted segment, so `h.greet` is findable by `greet`).
Both resolver persist paths now park unresolvable refs as failed
instead of deleting them. All pending-work readers (batched drain,
non-progress guard, #1187 orphan sweep, status pendingRefs) filter to
pending, preserving their invariants and keeping status honest.
- Sync retry: after scoped resolution, failed refs whose name tail
matches a symbol name now present in the changed files are re-resolved
through a per-ref-yielding path (watchdog-safe, #1091 class). Names
matching >500 failed refs are skipped as external/builtin noise (#999
rationale).
- Removal side: createEdges stamps each resolution edge with its
originating reference (metadata.refName, + refKind when kind promotion
rewrote it). When the #899 restore misses a target or sync deletes a
file, the dropped edge is resurrected as exactly that ref — re-resolved
in the same sync (rebinding to an alternative definition) or parked
failed until the symbol reappears. Edges without the stamp (pre-upgrade,
synthesized) still drop silently: reconstructing from the target's plain
name would strip receiver context and risk a rebind a full re-index
would never make.
- Pure-removal syncs clear resolver caches so a long-lived daemon can't
resolve resurrected refs against the pre-removal graph.
Validated: issue repro now yields a graph byte-identical to a full
re-index; move/remove-readd/file-deletion scenarios all rebind or heal;
baseline-vs-new A/B on express and gin shows identical node/edge counts
and no timing regression (DB grows ~25% from the parked ref rows — pure
cache, reset by any full re-index). 8 regression tests added.
Fixes#1240
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.
The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.
- All synthesis passes stream node-kind scans (cursors, not arrays) and
yield on time-budgeted checkpoints; language gates skip passes whose
filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
units stream one file at a time, and the all-functions array +
write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
per-file store commits chunk with yields behind a serialized flush
chain (preserving #1015 file-order determinism); resolver warm-up
streams the DISTINCT name set; resolution batch-tail and merged-edge
inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
session permanently silent (client-hello tail unshifted into a
flowing stream with zero listeners — the long-standing #662 test
flake was this real bug); first tool call no longer queues behind
the query pool's cold start (pool.ready gate).
Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The keyword gate (#1126) can never know a repo's domain nouns. This adds
the graph-derived tier the design discussion converged on: symbol names
are split into prose segments at index time (name_segment_vocab, riding
the insertNode write path), and the hook verifies a prompt's plain words
against them — "the state machine des commandes" → OrderStateMachine, in
any language whose technical nouns are Latin script.
Confidence now decides HOW MUCH to inject, not just whether:
- HIGH (keyword, or index-verified code token): full explore injection,
unchanged — the validated adoption lever.
- MEDIUM (segment matches only): a ~500-byte pointer naming the matching
symbols; the AGENT writes the explore query. Never runs explore, so a
fuzzy match can't inject 16KB of wrong-feature context.
- Silent otherwise, as before.
Precision is derived from the repo's own naming statistics plus measured
FP fixes: co-occurrence (≥2 words on one name) always qualifies; a single
word must be ≥5 chars, cluster across 2–25 names (singletons are prose
coincidence: "deploy to production" → matchesNonProductionDir), match a
multi-segment name, and not be an English function/filler word (the one
place a word list is honest: identifiers are English, so only English
prose collides). Every candidate is re-verified against nodes before
being surfaced — vocab rows are proposals, deletions leave orphans by
design, a full index rebuilds from scratch, and sync heals pre-upgrade
databases (batched + yielding; emptiness captured at sync ENTRY so the
sync's own writes can't mask the backfill).
Schema v7 migration is DDL-only (instant; none of the #1067 row-churn
hazards). Gate outcomes roll up as anonymous usage counters
(prompt-hook-gate-<outcome>, names only, never content) through the
existing telemetry pipeline — recall becomes measurable, and the counters
are the agreed kill-criterion data for ever revisiting a local classifier.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`insertEdge` has always used `INSERT OR IGNORE`, but the edges table carried
no UNIQUE constraint — only an autoincrement PK and non-unique indexes — so
`OR IGNORE` had nothing to conflict on and behaved like a plain INSERT.
Whenever two extraction/resolution passes emitted the same edge (e.g. a
return type captured by both a type-reference and a value-reference pass),
the graph stored byte-identical duplicate rows: ~527 on this repo, inflating
edge counts and letting callers/impact list the same relationship twice.
Add a UNIQUE identity index on (source, target, kind, IFNULL(line,-1),
IFNULL(col,-1)) — in schema.sql for fresh databases and migration v6 (dedup
existing rows, then create the index) for existing ones. IFNULL folds the
nullable line/col so coordinate-less edges (synthesized / file-level) dedup
too; SQLite otherwise treats each NULL as distinct. Distinct call sites
(same source/target/kind, different line/col) are preserved — only
byte-identical structural duplicates collapse. This is the storage-layer
invariant the reporter identified: it makes OR IGNORE keep its promise and
catches every double-emit, present and future, rather than chasing each
emitting pass.
Migration v6 is deterministic (keeps the lowest id per identity group) and
idempotent (IF NOT EXISTS index; no-op DELETE once unique). The DELETE's
GROUP BY matches the index expression exactly so creation can't fail on a
leftover pair.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deleteResolvedReferences bound every id into a single unbounded
`IN (...)`, so a list longer than SQLITE_MAX_VARIABLE_NUMBER (32766 on
the bundled node:sqlite) threw "too many SQL variables" — the one IN-list
in queries.ts that #540 missed. It's reachable only through the exported
QueryBuilder (library use): the internal resolution path uses
deleteSpecificResolvedReferences, which binds per-row and is immune, so
the CLI/MCP indexing pipeline was never affected. Wrap it in the same
SQLITE_PARAM_CHUNK_SIZE loop every sibling query uses, and add a
regression test (33k ids, past the real 32766 ceiling) that throws
without the fix.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Batch getNodesByIds to collapse N+1 reads in graph traversal, invalidate the
insertNode LRU cache so INSERT OR REPLACE doesn't serve a stale row, and run
incremental PRAGMA optimize + passive WAL checkpoint after bulk writes.
Closes#108
Colby McHenryandClaude Opus 5
NandhisandGitHub
andreinknvandGitHub