From 4c15f84aa498cba03672601e0721576cfd224efc Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 6 Jul 2026 14:20:19 -0500 Subject: [PATCH] fix(resolution): sweep orphaned unresolved refs so an interrupted index heals on sync (#1187) (#1191) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An indexing run killed mid-"Resolving refs" (crash, Ctrl-C, the #1122 watchdog kill) left the refs it never reached parked in unresolved_refs. The git-scoped sync fast path only re-resolves changed files' refs, so those files' call edges were missing permanently — a too-small blast radius clustering by package/module (the #1187 field report: 3 of 10 caller files for a Spring @Resource-injected method) — until a full re-index. - sync() now sweeps leftover unresolved refs with the batched resolver after its scoped pass, including on no-change syncs, so a bare `codegraph sync` recovers a wedged index (and heals pre-fix indexes on the first post-upgrade sync) - the scoped pass deletes unresolvable rows too (parity with the batched path), making "rows at rest" a sound orphan signal - drop the batched loop's early break that abandoned all later batches when one batch was all-unresolvable (its rows WERE consumed — that early stop could orphan the rest of the table at init) - surface the state: `codegraph status` warns, `status --json` gains index.pendingRefs, and MCP codegraph_status tells agents the blast radius is incomplete until the next sync Verified end-to-end on a 2,414-file synthetic Spring repo: SIGKILL mid-resolution reproduces the reporter's exact 3-of-10-callers state; a bare sync now heals it to 10/10 with the edge count converging to the clean-init total; a healthy-index sync stays a no-op. Co-authored-by: Claude Fable 5 --- CHANGELOG.md | 1 + __tests__/orphaned-refs-sweep.test.ts | 231 ++++++++++++++++++++++++++ src/bin/codegraph.ts | 10 ++ src/index.ts | 46 ++++- src/mcp/tools.ts | 13 ++ src/resolution/index.ts | 29 +++- 6 files changed, 323 insertions(+), 7 deletions(-) create mode 100644 __tests__/orphaned-refs-sweep.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0eb4d4d..34a1606 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes - `codegraph init` and `codegraph index` no longer get killed by the safety watchdog at the "Resolving refs" step on large method-name-heavy codebases (big Java/enterprise monorepos were the main victims, especially on slower machines). Resolution used to come up for air only every 500 references, so a dense stretch of expensive ones could starve the watchdog long enough for it to assume the process was stuck and kill a perfectly healthy index. Resolution now checkpoints after every reference, and two of the expensive steps got much cheaper: repeated method lookups on the same type are now cached, and source files are no longer re-split line-by-line for every call being resolved — indexing such repos is several times faster as a result. Generated or minified single-line files are also skipped during receiver-type inference instead of being scanned per call. Thanks @UchihaYong and @wangmeng-95 for the reports. (#1122) +- An index left incomplete by an interrupted run now heals itself on the next sync instead of silently staying wrong forever. If indexing died partway through resolving references (a crash, Ctrl-C, or the watchdog kill fixed above), the affected files still looked indexed but their caller/impact edges were missing — a too-small blast radius clustering by package or module, e.g. a Spring `@Resource`-injected method reporting 3 of its 10 real caller files — and because incremental syncs only re-resolve files that changed, the damage was permanent until a full re-index. Any sync (a watched file change, or a bare `codegraph sync`) now detects the leftover references and finishes resolving them, `codegraph status` warns when an index is in that state instead of passing it off as healthy, and a rare early-stop that could abandon resolution on repos whose first files reference only external libraries is fixed too. Thanks @KnifeOfLife for the report and the package-correlation observation that pinned it down. (#1187) - The automatic context hook for Claude Code now fires for structural questions asked in nearly thirty languages — French, Spanish, Portuguese, German, Italian, Dutch, Polish, Czech, Romanian, Hungarian, Greek, Swedish, Danish, Norwegian, Finnish, Russian, Ukrainian, Turkish, Indonesian, Vietnamese, Thai, Hindi, Arabic, Farsi, Hebrew, Japanese, Korean, and both simplified and traditional Chinese — instead of just English and simplified Chinese. Previously a natural question like "comment marche la state machine des commandes ?" injected nothing unless it happened to contain a code-shaped symbol name, making the hook look broken for non-English teams. English questions phrased with derived word forms ("explain the architecture…", "what are the dependencies…") now fire too, and prompts in any other language still fire when they name a symbol from the index. Thanks @anthonyle-roy-lgtm for the report. (#1126) - Lua and Luau method calls with capitalized names (`obj:Method()` — the standard Roblox convention) now link to the right method. Because Lua's method-call syntax looks identical to a Luau type annotation, a capitalized call like `lg:Log()` was misread as declaring the variable's type, so whenever two or more classes shared a method name (`Init`, `Update`, `Destroy`, …) the call was silently dropped from callers, impact/blast-radius, and flow traces. Lowercase method names were unaffected. Thanks @inth3shadows for the precise root-cause analysis and repro. (#1124) - Removed dead code left behind by the discontinued managed-reasoning feature. Its `codegraph login` flow was unplugged before ever shipping in a release, but the unused module still shipped inside the platform bundles, and a security review flagged its Windows browser-open step (it routed the login URL through `cmd`, which would have been unsafe had the flow ever been wired back up). The leftover module and its tests are now fully deleted. Thanks @inth3shadows for the report. (#1114) diff --git a/__tests__/orphaned-refs-sweep.test.ts b/__tests__/orphaned-refs-sweep.test.ts new file mode 100644 index 0000000..7864ab9 --- /dev/null +++ b/__tests__/orphaned-refs-sweep.test.ts @@ -0,0 +1,231 @@ +/** + * Orphaned unresolved-refs sweep (#1187) + * + * A resolution pass that dies mid-run (watchdog SIGKILL, Ctrl-C, crash) + * leaves the refs it never reached in unresolved_refs. The git-scoped sync + * fast path only ever reads the changed files' rows, so those orphans — and + * the call edges they represent — used to be missing permanently until a + * full re-index. Field report: a Spring monorepo where blast radius showed + * 3 of 10 caller files for a method behind @Resource field injection. + * + * These tests pin the healing behavior: a completed pass consumes every row + * it processes (resolved or not), and sync sweeps any leftovers even when + * no files changed. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; + +describe('Orphaned refs sweep (#1187)', () => { + let testDir: string; + let cg: CodeGraph; + + beforeEach(() => { + testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-orphan-sweep-')); + }); + + afterEach(() => { + if (cg) { + cg.destroy(); + } + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }); + } + }); + + /** Distinct files with a `calls` edge into the node. */ + function callerFiles(target: { id: string }): string[] { + return [...new Set(cg.getCallers(target.id).map((c) => c.node.filePath))].sort(); + } + + /** + * Leave `relPath` in the exact on-disk state a resolution pass killed + * mid-run leaves behind: content re-extracted (nodes + refs re-inserted, + * old edges cascade-deleted, content hash stamped current) but resolution + * never run. The content tweak is needed because re-extraction of + * byte-identical content is a no-op; the hash stamp means a later sync + * sees NO changed files. + */ + async function interruptAfterExtraction(relPath: string): Promise { + fs.appendFileSync(path.join(testDir, relPath), '\n// interrupted-run edit\n'); + await cg.indexFiles([relPath]); + } + + function findMethod(name: string) { + const hit = cg + .searchNodes(name) + .find((r) => (r.node.kind === 'method' || r.node.kind === 'function') && r.node.name === name); + expect(hit, `expected an indexed definition of ${name}`).toBeDefined(); + return hit!.node; + } + + describe('sync() heals an interrupted resolution run', () => { + beforeEach(async () => { + // The #1187 shape: a concrete @Component class called through Spring + // @Resource field injection from another package. + const supportDir = path.join(testDir, 'src', 'support'); + const notifyDir = path.join(testDir, 'src', 'notify'); + fs.mkdirSync(supportDir, { recursive: true }); + fs.mkdirSync(notifyDir, { recursive: true }); + + fs.writeFileSync( + path.join(supportDir, 'MemberDescriptionSupport.java'), + [ + 'package com.demo.support;', + '', + 'public class MemberDescriptionSupport {', + ' public String getSuperVipName() {', + ' return "SVIP";', + ' }', + '}', + '', + ].join('\n') + ); + + fs.writeFileSync( + path.join(notifyDir, 'NotifyBuilder.java'), + [ + 'package com.demo.notify;', + '', + 'import com.demo.support.MemberDescriptionSupport;', + '', + 'public class NotifyBuilder {', + ' private MemberDescriptionSupport memberDescriptionSupport;', + '', + ' public String buildParams() {', + ' return memberDescriptionSupport.getSuperVipName();', + ' }', + '}', + '', + ].join('\n') + ); + + cg = CodeGraph.initSync(testDir); + await cg.indexAll(); + }); + + it('resolves leftover refs on a sync with NO file changes', async () => { + const target = findMethod('getSuperVipName'); + + // Healthy baseline: the caller edge exists, no refs pending. + expect(callerFiles(target)).toContain('src/notify/NotifyBuilder.java'); + expect(cg.getPendingReferenceCount()).toBe(0); + + // Simulate the interrupted run: re-extract the caller (cascade-deleting + // its old nodes and edges, re-inserting its refs) and stop before + // resolution — exactly the state a killed "Resolving refs" phase + // leaves behind. + await interruptAfterExtraction('src/notify/NotifyBuilder.java'); + expect(cg.getPendingReferenceCount()).toBeGreaterThan(0); + expect(callerFiles(target)).not.toContain('src/notify/NotifyBuilder.java'); + + // The file on disk is unchanged, so this sync re-extracts nothing — + // pre-fix it returned without touching resolution and the edge stayed + // missing forever. + const result = await cg.sync(); + expect(result.filesAdded).toBe(0); + expect(result.filesModified).toBe(0); + + expect(cg.getPendingReferenceCount()).toBe(0); + expect(callerFiles(target)).toContain('src/notify/NotifyBuilder.java'); + }); + + it('is idempotent: a second no-change sync stays clean', async () => { + await interruptAfterExtraction('src/notify/NotifyBuilder.java'); + await cg.sync(); + const target = findMethod('getSuperVipName'); + const healed = callerFiles(target); + + const again = await cg.sync(); + expect(again.filesAdded + again.filesModified + again.filesRemoved).toBe(0); + expect(cg.getPendingReferenceCount()).toBe(0); + expect(callerFiles(target)).toEqual(healed); + }); + }); + + describe('completed passes consume every processed row', () => { + it('resolveReferences() deletes unresolvable rows (parity with the batched path)', async () => { + const srcDir = path.join(testDir, 'src'); + fs.mkdirSync(srcDir, { recursive: true }); + fs.writeFileSync( + path.join(srcDir, 'app.ts'), + [ + 'export function helper() { return 1; }', + 'export function main() {', + ' helper();', + ' totallyUndefinedCall();', // resolves to nothing anywhere + '}', + '', + ].join('\n') + ); + + cg = CodeGraph.initSync(testDir); + await cg.indexAll(); + expect(cg.getPendingReferenceCount()).toBe(0); + + // Re-extract without resolving: both the resolvable helper() ref and + // the unresolvable one are back in the table. + await interruptAfterExtraction('src/app.ts'); + expect(cg.getPendingReferenceCount()).toBeGreaterThan(0); + + // The non-batched full pass (which also backs the git-scoped sync + // path) must consume BOTH: pre-fix it deleted only resolved rows, so + // unresolvable ones parked forever and defeated the orphan sweep's + // "non-empty table means interrupted run" invariant. + cg.resolveReferences(); + expect(cg.getPendingReferenceCount()).toBe(0); + }); + + it('batched resolution does not stop at an all-unresolvable batch', async () => { + const srcDir = path.join(testDir, 'src'); + fs.mkdirSync(srcDir, { recursive: true }); + // File A: only unresolvable refs. Extracted first, so its rows sort + // first and fill the whole first batch. + fs.writeFileSync( + path.join(srcDir, 'a.ts'), + [ + 'export function a() {', + ' ghostOne();', + ' ghostTwo();', + ' ghostThree();', + '}', + '', + ].join('\n') + ); + // File B: a resolvable ref whose rows sort after A's. + fs.writeFileSync( + path.join(srcDir, 'b.ts'), + [ + "import { target } from './c';", + 'export function b() { target(); }', + '', + ].join('\n') + ); + fs.writeFileSync( + path.join(srcDir, 'c.ts'), + 'export function target() { return 2; }\n' + ); + + cg = CodeGraph.initSync(testDir); + await cg.indexAll(); + + // Re-queue A's refs then B's, in that order. + await interruptAfterExtraction('src/a.ts'); + await interruptAfterExtraction('src/b.ts'); + expect(cg.getPendingReferenceCount()).toBeGreaterThan(0); + + // Batch size 2 puts only A's unresolvable refs in the first batch. + // The old early break ended the whole run there, leaving B's ref an + // orphan even though the batch's rows WERE consumed (progress). + const resolver = (cg as unknown as { resolver: { resolveAndPersistBatched(p?: unknown, b?: number): Promise } }).resolver; + await resolver.resolveAndPersistBatched(undefined, 2); + + expect(cg.getPendingReferenceCount()).toBe(0); + const target = findMethod('target'); + expect(callerFiles(target)).toContain('src/b.ts'); + }); + }); +}); diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 561163b..07c51c5 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -807,6 +807,9 @@ program const buildInfo = cg.getIndexBuildInfo(); const reindexRecommended = cg.isIndexStale(); const indexState = cg.getIndexState(); + // Zero on a healthy index; non-zero at rest means a resolution pass was + // interrupted, so some files' call edges are missing (#1187). + const pendingRefs = cg.getPendingReferenceCount(); // JSON output mode if (options.json) { @@ -842,6 +845,10 @@ program // (a run was killed mid-index — the index is truncated) | // 'failed' | null (predates the marker). state: indexState, + // References awaiting resolution. Non-zero at rest means an + // interrupted resolution pass left edges missing; the next + // sync sweeps them (#1187). + pendingRefs, }, })); cg.destroy(); @@ -862,6 +869,9 @@ program } else if (indexState === 'failed') { warn('The last index run failed — results may be incomplete. Re-run "codegraph index".'); } + if (pendingRefs > 0) { + warn(`${formatNumber(pendingRefs)} references from an interrupted run are awaiting resolution — some callers/impact edges are missing. Run "codegraph sync" to resolve them.`); + } console.log(); // Index stats diff --git a/src/index.ts b/src/index.ts index c464e01..7430369 100644 --- a/src/index.ts +++ b/src/index.ts @@ -608,7 +608,8 @@ export class CodeGraph { } // Resolve references if files were updated - if (result.filesAdded > 0 || result.filesModified > 0) { + const filesChanged = result.filesAdded > 0 || result.filesModified > 0; + if (filesChanged) { if (result.changedFilePaths) { // Scope resolution to changed files (git fast path — bounded set) const unresolvedRefs = this.queries.getUnresolvedReferencesByFiles(result.changedFilePaths); @@ -644,7 +645,38 @@ export class CodeGraph { }); }); } + } + // Orphan sweep (#1187). A resolution pass that dies mid-run — the #850 + // daemon liveness watchdog's SIGKILL (#1122), Ctrl-C, a crash — leaves + // the refs it never reached in unresolved_refs, and the git-scoped fast + // path above never revisits them (it reads only the changed files' + // rows). Those files' call edges were then missing PERMANENTLY, with + // nothing to see except a too-small blast radius, until a full + // re-index. A completed pass deletes every row it processed (resolved + // or not), so any row still present now is such an orphan — or a row + // parked by an older engine whose scoped pass kept unresolvable refs. + // Grind them down with the batched resolver; this also makes a bare + // `codegraph sync` the recovery command for a wedged index. On a + // healthy index this is one COUNT query. + const orphanCount = this.queries.getUnresolvedReferencesCount(); + if (orphanCount > 0) { + options.onProgress?.({ + phase: 'resolving', + current: 0, + total: orphanCount, + }); + + await this.resolveReferencesBatched((current, total) => { + options.onProgress?.({ + phase: 'resolving', + current, + total, + }); + }); + } + + if (filesChanged || orphanCount > 0) { // Second pass: chained calls whose method lives on a supertype the // receiver conforms to (protocol-extension / inherited). Needs the // implements/extends edges built above (#750). @@ -655,7 +687,7 @@ export class CodeGraph { } // Refresh planner stats + checkpoint the WAL after bulk writes. - if (result.filesAdded > 0 || result.filesModified > 0 || result.filesRemoved > 0) { + if (filesChanged || result.filesRemoved > 0 || orphanCount > 0) { this.db.runMaintenance(); } @@ -873,6 +905,16 @@ export class CodeGraph { return this.resolver.resolveAndPersistBatched(onProgress); } + /** + * References extracted but not yet resolved into edges. Zero on a healthy + * index — a completed resolution pass consumes every row. Non-zero at rest + * means a pass was interrupted mid-run (killed indexer, crash — #1187), so + * some files' call edges are missing; the next `sync` sweeps them. + */ + getPendingReferenceCount(): number { + return this.queries.getUnresolvedReferencesCount(); + } + /** * Get detected frameworks in the project */ diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 55ef1e5..eb17577 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -4038,6 +4038,19 @@ export class ToolHandler { ); } + // Non-zero at rest means a resolution pass was interrupted mid-run, so + // some files' call/impact edges are missing until the next sync sweeps + // the leftovers (#1187). Surface it — an agent trusting an incomplete + // blast radius is worse than one that knows to re-sync. + const pendingRefs = cg.getPendingReferenceCount(); + if (pendingRefs > 0) { + lines.push( + `**Pending resolution:** ⚠ ${pendingRefs} references from an interrupted ` + + `index run — some caller/impact edges are missing until the next sync ` + + `(any file change triggers it, or run \`codegraph sync\`)` + ); + } + lines.push('', '**Nodes by Kind:**'); for (const [kind, count] of Object.entries(stats.nodesByKind)) { diff --git a/src/resolution/index.ts b/src/resolution/index.ts index ac82c5e..1edae51 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -968,6 +968,23 @@ export class ReferenceResolver { ); } + // Delete unresolvable refs too — parity with resolveAndPersistBatched. + // Keeping them bought nothing: a ref is only ever retried when its file + // is re-extracted, which cascade-deletes and re-inserts its rows anyway. + // And it broke the #1187 orphan sweep's invariant — after a COMPLETED + // pass the table must hold nothing that pass processed, so that any row + // still present belongs to an interrupted run and the sweep can key off + // a bare row count. + if (result.unresolved.length > 0) { + this.queries.deleteSpecificResolvedReferences( + result.unresolved.map((r) => ({ + fromNodeId: r.fromNodeId, + referenceName: r.referenceName, + referenceKind: r.referenceKind, + })) + ); + } + return result; } @@ -1152,11 +1169,13 @@ export class ReferenceResolver { // Yield so progress UI can render between batches await new Promise(resolve => setImmediate(resolve)); - // If nothing was resolved or removed in this batch, we'd loop forever - // on the same rows. Break to avoid infinite loop. - if (result.resolved.length === 0 && result.unresolved.length === batch.length) { - break; - } + // NOTE: there used to be an extra early break here when a batch resolved + // nothing (`result.unresolved.length === batch.length`). That was wrong: + // an all-unresolvable batch still DELETES its rows (progress), yet the + // break abandoned every batch after it in the same run — on a repo whose + // first 5000 refs are all external/stdlib calls, resolution stopped at + // batch one and left the rest of the table as permanent orphans (#1187). + // The count-based guard below catches the true no-progress case. // Non-progress guard (defense-in-depth). Because we re-read from offset 0 // each pass, the unresolved_refs table MUST shrink every iteration — both