fix(sync): resolve cross-file refs when an edit adds or removes the satisfying symbol (#1240) (#1249)
* chore: ignore .kommandr/ directory * fix(sync): resolve cross-file refs when an edit adds or removes the satisfying symbol (#1240) Incremental sync scoped reference resolution to the changed files' own refs, and a completed pass deleted every ref it failed to resolve — so a symbol change in one file could never repair references in UNCHANGED files, in either direction, until a full re-index: - New-export case: a.ts imports/calls `greet` before b.ts defines it. The failed refs were deleted at index time; when b.ts later gained `greet`, nothing revisited a.ts — the calls/imports edges stayed missing while status reported a clean index. - Removal case: when a re-index (or file deletion) dropped a symbol, the incoming edges cascade-deleted and the callers — whose resolved refs had been consumed — never got a chance to rebind to an alternative definition or reconnect when the symbol returned. Fix, sharing one lifecycle: - Schema v8: unresolved_refs gains status ('pending'/'failed') and name_tail (last dotted segment, so `h.greet` is findable by `greet`). Both resolver persist paths now park unresolvable refs as failed instead of deleting them. All pending-work readers (batched drain, non-progress guard, #1187 orphan sweep, status pendingRefs) filter to pending, preserving their invariants and keeping status honest. - Sync retry: after scoped resolution, failed refs whose name tail matches a symbol name now present in the changed files are re-resolved through a per-ref-yielding path (watchdog-safe, #1091 class). Names matching >500 failed refs are skipped as external/builtin noise (#999 rationale). - Removal side: createEdges stamps each resolution edge with its originating reference (metadata.refName, + refKind when kind promotion rewrote it). When the #899 restore misses a target or sync deletes a file, the dropped edge is resurrected as exactly that ref — re-resolved in the same sync (rebinding to an alternative definition) or parked failed until the symbol reappears. Edges without the stamp (pre-upgrade, synthesized) still drop silently: reconstructing from the target's plain name would strip receiver context and risk a rebind a full re-index would never make. - Pure-removal syncs clear resolver caches so a long-lived daemon can't resolve resurrected refs against the pre-removal graph. Validated: issue repro now yields a graph byte-identical to a full re-index; move/remove-readd/file-deletion scenarios all rebind or heal; baseline-vs-new A/B on express and gin shows identical node/edge counts and no timing regression (DB grows ~25% from the parked ref rows — pure cache, reset by any full re-index). 8 regression tests added. Fixes #1240 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
386bff0f84
commit
9d0cd3a7d1
+32
-1
@@ -9,7 +9,7 @@ import { SqliteDatabase } from './sqlite-adapter';
|
||||
/**
|
||||
* Current schema version
|
||||
*/
|
||||
export const CURRENT_SCHEMA_VERSION = 7;
|
||||
export const CURRENT_SCHEMA_VERSION = 8;
|
||||
|
||||
/**
|
||||
* Migration definition
|
||||
@@ -119,6 +119,37 @@ const migrations: Migration[] = [
|
||||
`);
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 8,
|
||||
description:
|
||||
'Track attempted-but-unresolvable refs as status=failed so sync can retry them when a changed file adds a matching symbol (#1240)',
|
||||
up: (db) => {
|
||||
// DDL only — instant on any size database. No backfill needed: rows are
|
||||
// only ever queried by name_tail once they carry status='failed', and
|
||||
// both fields are written together by markReferencesFailed. Legacy rows
|
||||
// (all 'pending' after this migration) are orphans from interrupted runs
|
||||
// that the #1187 sweep grinds down on the next sync, marking survivors
|
||||
// failed with their tails as it goes. The tail index is partial: on a
|
||||
// healthy index the pending set is empty and the failed set is the only
|
||||
// population worth indexing. Keep the definitions in lockstep with
|
||||
// schema.sql. ALTER TABLE has no IF NOT EXISTS, so guard each column for
|
||||
// idempotency — a database created from current schema.sql already has
|
||||
// both (matters when migrations are re-run from an older recorded
|
||||
// version, as the v6 regression test does).
|
||||
const cols = db.prepare('PRAGMA table_info(unresolved_refs)').all() as Array<{ name: string }>;
|
||||
const hasColumn = (name: string) => cols.some((c) => c.name === name);
|
||||
if (!hasColumn('status')) {
|
||||
db.exec("ALTER TABLE unresolved_refs ADD COLUMN status TEXT NOT NULL DEFAULT 'pending'");
|
||||
}
|
||||
if (!hasColumn('name_tail')) {
|
||||
db.exec("ALTER TABLE unresolved_refs ADD COLUMN name_tail TEXT NOT NULL DEFAULT ''");
|
||||
}
|
||||
db.exec(`
|
||||
CREATE INDEX IF NOT EXISTS idx_unresolved_status ON unresolved_refs(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_unresolved_failed_tail ON unresolved_refs(name_tail) WHERE status = 'failed';
|
||||
`);
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
|
||||
+127
-9
@@ -109,6 +109,20 @@ interface UnresolvedRefRow {
|
||||
candidates: string | null;
|
||||
file_path: string;
|
||||
language: string;
|
||||
status: string;
|
||||
name_tail: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Last segment of a (possibly dotted/qualified) reference name — the part a
|
||||
* new symbol's plain node name could match: 'util.greet' → 'greet',
|
||||
* 'mod::fn' → 'fn', 'greet' → 'greet'. Written to unresolved_refs.name_tail
|
||||
* when a ref is marked failed, so the #1240 retry lookup can match dotted
|
||||
* refs against newly-added node names.
|
||||
*/
|
||||
function referenceNameTail(referenceName: string): string {
|
||||
const idx = Math.max(referenceName.lastIndexOf('.'), referenceName.lastIndexOf(':'));
|
||||
return idx >= 0 ? referenceName.slice(idx + 1) : referenceName;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1627,19 +1641,26 @@ export class QueryBuilder {
|
||||
* re-index (issue #899). Same edge-kind rules as
|
||||
* {@link getDependentFilePaths}: all kinds except `contains`.
|
||||
*/
|
||||
getCrossFileIncomingEdgesWithTarget(filePath: string): Array<Edge & { targetName: string; targetKind: NodeKind }> {
|
||||
const sql = `SELECT e.*, tgt.name AS target_name, tgt.kind AS target_kind
|
||||
getCrossFileIncomingEdgesWithTarget(
|
||||
filePath: string
|
||||
): Array<Edge & { targetName: string; targetKind: NodeKind; sourceFilePath: string; sourceLanguage: Language }> {
|
||||
const sql = `SELECT e.*, tgt.name AS target_name, tgt.kind AS target_kind,
|
||||
src.file_path AS source_file_path, src.language AS source_language
|
||||
FROM edges e
|
||||
JOIN nodes tgt ON tgt.id = e.target
|
||||
JOIN nodes src ON src.id = e.source
|
||||
WHERE tgt.file_path = ?
|
||||
AND e.kind != 'contains'
|
||||
AND src.file_path != ?`;
|
||||
const rows = this.db.prepare(sql).all(filePath, filePath) as Array<EdgeRow & { target_name: string; target_kind: NodeKind }>;
|
||||
const rows = this.db.prepare(sql).all(filePath, filePath) as Array<
|
||||
EdgeRow & { target_name: string; target_kind: NodeKind; source_file_path: string; source_language: Language }
|
||||
>;
|
||||
return rows.map(row => ({
|
||||
...rowToEdge(row),
|
||||
targetName: row.target_name,
|
||||
targetKind: row.target_kind,
|
||||
sourceFilePath: row.source_file_path,
|
||||
sourceLanguage: row.source_language,
|
||||
}));
|
||||
}
|
||||
|
||||
@@ -1827,12 +1848,16 @@ export class QueryBuilder {
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the count of unresolved references without loading them into memory
|
||||
* Get the count of PENDING (never-attempted) references without loading
|
||||
* them into memory. Rows marked status='failed' — attempted by a completed
|
||||
* pass, no match — are excluded: they are not outstanding work, only retry
|
||||
* candidates for the #1240 sweep, so they must not trip the #1187 orphan
|
||||
* sweep or the `status` pending-refs warning.
|
||||
*/
|
||||
getUnresolvedReferencesCount(): number {
|
||||
if (!this.stmts.getUnresolvedCount) {
|
||||
this.stmts.getUnresolvedCount = this.db.prepare(
|
||||
'SELECT COUNT(*) as count FROM unresolved_refs'
|
||||
"SELECT COUNT(*) as count FROM unresolved_refs WHERE status = 'pending'"
|
||||
);
|
||||
}
|
||||
const row = this.stmts.getUnresolvedCount.get() as { count: number };
|
||||
@@ -1840,13 +1865,15 @@ export class QueryBuilder {
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a batch of unresolved references using LIMIT/OFFSET pagination.
|
||||
* Used to process references in bounded memory chunks.
|
||||
* Get a batch of PENDING unresolved references using LIMIT/OFFSET
|
||||
* pagination. Used to process references in bounded memory chunks; failed
|
||||
* rows are excluded so the batched drain loop terminates once every row
|
||||
* has been attempted.
|
||||
*/
|
||||
getUnresolvedReferencesBatch(offset: number, limit: number): UnresolvedReference[] {
|
||||
if (!this.stmts.getUnresolvedBatch) {
|
||||
this.stmts.getUnresolvedBatch = this.db.prepare(
|
||||
'SELECT * FROM unresolved_refs LIMIT ? OFFSET ?'
|
||||
"SELECT * FROM unresolved_refs WHERE status = 'pending' LIMIT ? OFFSET ?"
|
||||
);
|
||||
}
|
||||
const rows = this.stmts.getUnresolvedBatch.all(limit, offset) as UnresolvedRefRow[];
|
||||
@@ -1913,7 +1940,7 @@ export class QueryBuilder {
|
||||
const chunk = filePaths.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const placeholders = chunk.map(() => '?').join(',');
|
||||
const chunkRows = this.db
|
||||
.prepare(`SELECT * FROM unresolved_refs WHERE file_path IN (${placeholders})`)
|
||||
.prepare(`SELECT * FROM unresolved_refs WHERE status = 'pending' AND file_path IN (${placeholders})`)
|
||||
.all(...chunk) as UnresolvedRefRow[];
|
||||
rows.push(...chunkRows);
|
||||
}
|
||||
@@ -1971,6 +1998,97 @@ export class QueryBuilder {
|
||||
deleteMany(refs);
|
||||
}
|
||||
|
||||
/**
|
||||
* Mark refs a completed resolution pass could not resolve as status='failed'
|
||||
* instead of deleting them (#1240). Failed rows are invisible to the pending
|
||||
* count/batch readers (so drain loops and the #1187 orphan sweep still
|
||||
* terminate) but stay queryable by name_tail so a later sync can retry them
|
||||
* when a changed file introduces a symbol that could satisfy them. name_tail
|
||||
* is (re)written here so rows inserted before the v8 migration get their
|
||||
* tail the first time they're attempted.
|
||||
*/
|
||||
markReferencesFailed(refs: Array<{ fromNodeId: string; referenceName: string; referenceKind: string }>): void {
|
||||
if (refs.length === 0) return;
|
||||
const stmt = this.db.prepare(
|
||||
"UPDATE unresolved_refs SET status = 'failed', name_tail = ? WHERE from_node_id = ? AND reference_name = ? AND reference_kind = ?"
|
||||
);
|
||||
const markMany = this.db.transaction((items: typeof refs) => {
|
||||
for (const ref of items) {
|
||||
stmt.run(referenceNameTail(ref.referenceName), ref.fromNodeId, ref.referenceName, ref.referenceKind);
|
||||
}
|
||||
});
|
||||
markMany(refs);
|
||||
}
|
||||
|
||||
/**
|
||||
* Failed refs whose name tail matches one of the given symbol names — the
|
||||
* candidates a sync should retry after files carrying those names changed
|
||||
* (#1240). Names matching more than `perNameCeiling` failed refs are
|
||||
* skipped entirely: at that population a name is external/builtin noise
|
||||
* (`get`, `map`, …) that one new definition won't resolve — the same
|
||||
* rationale as resolution's AMBIGUOUS_NAME_CEILING (#999) — and retrying an
|
||||
* arbitrary subset would be both wasted work and incoherent coverage.
|
||||
*/
|
||||
getRetryableFailedReferences(names: string[], perNameCeiling: number = 500): UnresolvedReference[] {
|
||||
if (names.length === 0) return [];
|
||||
|
||||
// Pass 1: per-tail counts, chunked under the SQLite parameter limit.
|
||||
const retryNames: string[] = [];
|
||||
for (let i = 0; i < names.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = names.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const placeholders = chunk.map(() => '?').join(',');
|
||||
const counts = this.db
|
||||
.prepare(
|
||||
`SELECT name_tail, COUNT(*) as count FROM unresolved_refs WHERE status = 'failed' AND name_tail IN (${placeholders}) GROUP BY name_tail`
|
||||
)
|
||||
.all(...chunk) as Array<{ name_tail: string; count: number }>;
|
||||
for (const row of counts) {
|
||||
if (row.count <= perNameCeiling) retryNames.push(row.name_tail);
|
||||
}
|
||||
}
|
||||
if (retryNames.length === 0) return [];
|
||||
|
||||
// Pass 2: load the surviving rows.
|
||||
const rows: UnresolvedRefRow[] = [];
|
||||
for (let i = 0; i < retryNames.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = retryNames.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const placeholders = chunk.map(() => '?').join(',');
|
||||
const chunkRows = this.db
|
||||
.prepare(`SELECT * FROM unresolved_refs WHERE status = 'failed' AND name_tail IN (${placeholders})`)
|
||||
.all(...chunk) as UnresolvedRefRow[];
|
||||
rows.push(...chunkRows);
|
||||
}
|
||||
|
||||
return rows.map((row) => ({
|
||||
fromNodeId: row.from_node_id,
|
||||
referenceName: row.reference_name,
|
||||
referenceKind: row.reference_kind as EdgeKind,
|
||||
line: row.line,
|
||||
column: row.col,
|
||||
candidates: row.candidates ? safeJsonParse(row.candidates, undefined) : undefined,
|
||||
filePath: row.file_path,
|
||||
language: row.language as Language,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Distinct node names present in the given files — the symbol names a sync
|
||||
* pass uses to look up retryable failed refs after those files changed.
|
||||
*/
|
||||
getNodeNamesByFiles(filePaths: string[]): string[] {
|
||||
if (filePaths.length === 0) return [];
|
||||
const names = new Set<string>();
|
||||
for (let i = 0; i < filePaths.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = filePaths.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const placeholders = chunk.map(() => '?').join(',');
|
||||
const rows = this.db
|
||||
.prepare(`SELECT DISTINCT name FROM nodes WHERE file_path IN (${placeholders})`)
|
||||
.all(...chunk) as Array<{ name: string }>;
|
||||
for (const row of rows) names.add(row.name);
|
||||
}
|
||||
return [...names];
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// Statistics
|
||||
// ===========================================================================
|
||||
|
||||
+13
-1
@@ -67,7 +67,15 @@ CREATE TABLE IF NOT EXISTS files (
|
||||
errors TEXT -- JSON array
|
||||
);
|
||||
|
||||
-- Unresolved References: References that need resolution after full indexing
|
||||
-- Unresolved References: References that need resolution after full indexing.
|
||||
-- status lifecycle: rows are inserted 'pending' by extraction; a completed
|
||||
-- resolution pass either deletes a row (resolved) or marks it 'failed'
|
||||
-- (attempted, no match — kept so a later sync can retry it when a changed
|
||||
-- file introduces a symbol that could satisfy it, #1240). name_tail is the
|
||||
-- last segment of reference_name ('util.greet' → 'greet'), written when a
|
||||
-- row is marked failed, so the retry lookup matches new node names against
|
||||
-- dotted refs too. Rows follow their from_node via ON DELETE CASCADE, so
|
||||
-- re-extracting or deleting a file clears its stale rows in any status.
|
||||
CREATE TABLE IF NOT EXISTS unresolved_refs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
from_node_id TEXT NOT NULL,
|
||||
@@ -78,6 +86,8 @@ CREATE TABLE IF NOT EXISTS unresolved_refs (
|
||||
candidates TEXT, -- JSON array
|
||||
file_path TEXT NOT NULL DEFAULT '',
|
||||
language TEXT NOT NULL DEFAULT 'unknown',
|
||||
status TEXT NOT NULL DEFAULT 'pending',
|
||||
name_tail TEXT NOT NULL DEFAULT '',
|
||||
FOREIGN KEY (from_node_id) REFERENCES nodes(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
@@ -172,6 +182,8 @@ CREATE INDEX IF NOT EXISTS idx_unresolved_from_node ON unresolved_refs(from_node
|
||||
CREATE INDEX IF NOT EXISTS idx_unresolved_name ON unresolved_refs(reference_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_unresolved_file_path ON unresolved_refs(file_path);
|
||||
CREATE INDEX IF NOT EXISTS idx_unresolved_from_name ON unresolved_refs(from_node_id, reference_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_unresolved_status ON unresolved_refs(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_unresolved_failed_tail ON unresolved_refs(name_tail) WHERE status = 'failed';
|
||||
CREATE INDEX IF NOT EXISTS idx_edges_provenance ON edges(provenance);
|
||||
|
||||
-- Project metadata for version/provenance tracking
|
||||
|
||||
+70
-4
@@ -16,6 +16,8 @@ import {
|
||||
ExtractionResult,
|
||||
ExtractionError,
|
||||
Edge,
|
||||
UnresolvedReference,
|
||||
ReferenceKind,
|
||||
} from '../types';
|
||||
import { QueryBuilder } from '../db/queries';
|
||||
import { extractFromSource } from './tree-sitter';
|
||||
@@ -1360,6 +1362,38 @@ function scanDirectoryWalk(
|
||||
return files;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resurrect a resolution edge that is about to be dropped (its target symbol
|
||||
* was removed, renamed, or its whole file deleted) as the ORIGINAL unresolved
|
||||
* reference that created it, read from the refName/refKind stamp
|
||||
* `createEdges` writes into edge metadata. Inserted as status='pending', the
|
||||
* ref is consumed by the same sync's resolution sweep: it rebinds to an
|
||||
* alternative definition if one exists, or parks as status='failed' where the
|
||||
* #1240 retry finds it if the symbol later reappears.
|
||||
*
|
||||
* Returns null — drop silently, the pre-#1240 behavior — for edges without a
|
||||
* refName stamp (created before the stamp existed, or synthesized): rebuilding
|
||||
* a ref from the target's plain node name would strip the receiver/qualifier
|
||||
* context the original text carried (`h.greet` → `greet`) and could rebind
|
||||
* somewhere a full re-index never would. Silent beats wrong.
|
||||
*/
|
||||
function resurrectRefFromDroppedEdge(
|
||||
e: Edge & { sourceFilePath: string; sourceLanguage: Language }
|
||||
): UnresolvedReference | null {
|
||||
const refName = e.metadata?.refName;
|
||||
if (typeof refName !== 'string' || refName.length === 0) return null;
|
||||
const refKind = typeof e.metadata?.refKind === 'string' ? (e.metadata.refKind as ReferenceKind) : e.kind;
|
||||
return {
|
||||
fromNodeId: e.source,
|
||||
referenceName: refName,
|
||||
referenceKind: refKind,
|
||||
line: e.line ?? 0,
|
||||
column: e.column ?? 0,
|
||||
filePath: e.sourceFilePath,
|
||||
language: e.sourceLanguage,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Extraction orchestrator
|
||||
*/
|
||||
@@ -2193,25 +2227,41 @@ export class ExtractionOrchestrator {
|
||||
// (filePath, kind, name). Node ids include the source line, so any line
|
||||
// shift in the callee file (e.g. a docstring-only edit above the symbol)
|
||||
// changes every target id and a naive re-insert by old id would drop them
|
||||
// all. `insertEdges` still filters to endpoints that exist, so edges whose
|
||||
// caller (source) was deleted, or whose callee (target) was renamed/removed
|
||||
// during the re-index (no match in `newTargetIds`), are dropped. This
|
||||
// closes the #899 edge-drop on `sync`.
|
||||
// all. `insertEdges` still filters to endpoints that exist. This closes
|
||||
// the #899 edge-drop on `sync`.
|
||||
//
|
||||
// Edges whose callee (target) was renamed/removed during the re-index (no
|
||||
// match in `newNodesByKindName`) are not silently dropped anymore: each is
|
||||
// resurrected as its ORIGINAL unresolved ref (stamped on the edge as
|
||||
// metadata.refName/refKind at creation) so the same sync's resolution
|
||||
// sweep can rebind it to an alternative definition elsewhere, or park it
|
||||
// as status='failed' to be retried when the symbol reappears — the
|
||||
// removal-side counterpart of #1240. Edges without refName (built before
|
||||
// the stamp existed, or synthesized) still drop silently: reconstructing
|
||||
// a ref from the target's plain name would strip receiver/qualifier
|
||||
// context and risk a rebind a full re-index would never make.
|
||||
if (crossFileIncomingEdges.length > 0) {
|
||||
const newNodesByKindName = new Map<string, string>();
|
||||
for (const n of validNodes) {
|
||||
newNodesByKindName.set(`${n.kind}\0${n.name}`, n.id);
|
||||
}
|
||||
const reinserted: Edge[] = [];
|
||||
const resurrected: UnresolvedReference[] = [];
|
||||
for (const e of crossFileIncomingEdges) {
|
||||
const newTargetId = newNodesByKindName.get(`${e.targetKind}\0${e.targetName}`);
|
||||
if (newTargetId) {
|
||||
reinserted.push({ source: e.source, target: newTargetId, kind: e.kind, metadata: e.metadata, line: e.line, column: e.column, provenance: e.provenance });
|
||||
} else {
|
||||
const ref = resurrectRefFromDroppedEdge(e);
|
||||
if (ref) resurrected.push(ref);
|
||||
}
|
||||
}
|
||||
if (reinserted.length > 0) {
|
||||
this.queries.insertEdges(reinserted);
|
||||
}
|
||||
if (resurrected.length > 0) {
|
||||
this.queries.insertUnresolvedRefsBatch(resurrected);
|
||||
}
|
||||
}
|
||||
|
||||
// Insert unresolved references in batch with denormalized filePath/language
|
||||
@@ -2300,6 +2350,22 @@ export class ExtractionOrchestrator {
|
||||
let reconcileChecks = 0;
|
||||
for (const tracked of trackedFiles) {
|
||||
if (!currentSet.has(tracked.path) || !fs.existsSync(path.join(this.rootDir, tracked.path))) {
|
||||
// Before the cascade deletes them, resurrect incoming cross-file
|
||||
// resolution edges as their original refs (#1240 removal case): the
|
||||
// callers live in files this sync will NOT revisit, so this is their
|
||||
// only chance to rebind to an alternative definition — or to park as
|
||||
// failed until the symbol reappears somewhere. (A deleted file whose
|
||||
// CALLERS are also being deleted is fine: their nodes cascade later
|
||||
// in this loop and take the resurrected rows with them.)
|
||||
const incoming = this.queries.getCrossFileIncomingEdgesWithTarget(tracked.path);
|
||||
if (incoming.length > 0) {
|
||||
const resurrected = incoming
|
||||
.map((e) => resurrectRefFromDroppedEdge(e))
|
||||
.filter((r): r is UnresolvedReference => r !== null);
|
||||
if (resurrected.length > 0) {
|
||||
this.queries.insertUnresolvedRefsBatch(resurrected);
|
||||
}
|
||||
}
|
||||
this.queries.deleteFile(tracked.path);
|
||||
filesRemoved++;
|
||||
}
|
||||
|
||||
+46
-7
@@ -664,6 +664,15 @@ export class CodeGraph {
|
||||
// (regex over *.module.ts only).
|
||||
if (result.filesAdded > 0 || result.filesModified > 0) {
|
||||
this.resolver.runPostExtract();
|
||||
} else if (result.filesRemoved > 0) {
|
||||
// A pure-removal sync still resolves refs below — the deletion path
|
||||
// resurrects the removed file's incoming edges as pending refs
|
||||
// (#1240 removal case) and the orphan sweep consumes them. In a
|
||||
// long-lived process (daemon) the resolver's name caches were
|
||||
// warmed against the pre-removal graph; drop them so resolution
|
||||
// sees the post-removal state. (runPostExtract above clears caches
|
||||
// itself, so the changed-files branch is already covered.)
|
||||
this.resolver.clearCaches();
|
||||
}
|
||||
|
||||
// Resolve references if files were updated
|
||||
@@ -688,6 +697,34 @@ export class CodeGraph {
|
||||
total,
|
||||
});
|
||||
});
|
||||
|
||||
// Retry previously-failed refs the changed files may now satisfy
|
||||
// (#1240). Scoped resolution above only re-resolves refs FROM the
|
||||
// changed files — but when a changed file gains an export/symbol,
|
||||
// refs in UNCHANGED files that failed against the old graph can
|
||||
// now resolve, and nothing else ever revisits them (their rows
|
||||
// were parked as status='failed' by an earlier completed pass).
|
||||
// Look them up by the symbol names the changed files now carry
|
||||
// and re-resolve just that set. On a sync where no failed ref
|
||||
// matches, this is one indexed lookup.
|
||||
const tRetry = Date.now();
|
||||
const retryable = this.queries.getRetryableFailedReferences(
|
||||
this.queries.getNodeNamesByFiles(result.changedFilePaths)
|
||||
);
|
||||
if (retryable.length > 0) {
|
||||
options.onProgress?.({
|
||||
phase: 'resolving',
|
||||
current: 0,
|
||||
total: retryable.length,
|
||||
});
|
||||
await this.resolver.resolveAndPersistListYielding(retryable);
|
||||
options.onProgress?.({
|
||||
phase: 'resolving',
|
||||
current: retryable.length,
|
||||
total: retryable.length,
|
||||
});
|
||||
}
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-failed-ref-retry: ${Date.now() - tRetry}ms (${retryable.length} refs)`);
|
||||
} else {
|
||||
// No git info — use batched resolution to avoid OOM
|
||||
const unresolvedCount = this.queries.getUnresolvedReferencesCount();
|
||||
@@ -714,9 +751,10 @@ export class CodeGraph {
|
||||
// path above never revisits them (it reads only the changed files'
|
||||
// rows). Those files' call edges were then missing PERMANENTLY, with
|
||||
// nothing to see except a too-small blast radius, until a full
|
||||
// re-index. A completed pass deletes every row it processed (resolved
|
||||
// or not), so any row still present now is such an orphan — or a row
|
||||
// parked by an older engine whose scoped pass kept unresolvable refs.
|
||||
// re-index. A completed pass takes every row it processed out of the
|
||||
// PENDING set (resolved rows are deleted, unresolvable ones parked as
|
||||
// status='failed' for the #1240 retry above), so any pending row now
|
||||
// is such an orphan — or a row from an older engine's scoped pass.
|
||||
// Grind them down with the batched resolver; this also makes a bare
|
||||
// `codegraph sync` the recovery command for a wedged index. On a
|
||||
// healthy index this is one COUNT query.
|
||||
@@ -968,10 +1006,11 @@ export class CodeGraph {
|
||||
}
|
||||
|
||||
/**
|
||||
* References extracted but not yet resolved into edges. Zero on a healthy
|
||||
* index — a completed resolution pass consumes every row. Non-zero at rest
|
||||
* means a pass was interrupted mid-run (killed indexer, crash — #1187), so
|
||||
* some files' call edges are missing; the next `sync` sweeps them.
|
||||
* References extracted but never attempted by a resolution pass. Zero on a
|
||||
* healthy index — a completed pass consumes every pending row (resolving it
|
||||
* or parking it as failed, #1240). Non-zero at rest means a pass was
|
||||
* interrupted mid-run (killed indexer, crash — #1187), so some files' call
|
||||
* edges are missing; the next `sync` sweeps them.
|
||||
*/
|
||||
getPendingReferenceCount(): number {
|
||||
return this.queries.getUnresolvedReferencesCount();
|
||||
|
||||
+81
-18
@@ -974,6 +974,18 @@ export class ReferenceResolver {
|
||||
metadata: {
|
||||
confidence: ref.confidence,
|
||||
resolvedBy: ref.resolvedBy,
|
||||
// The ORIGINAL reference text (and kind, when edge-kind promotion
|
||||
// rewrote it — calls→instantiates, extends→implements,
|
||||
// function_ref→references). If this edge's target is later removed
|
||||
// by a re-index, the edge is resurrected as exactly this ref and
|
||||
// re-resolved (#1240 removal case) — a faithful resurrection, so
|
||||
// re-resolution can never bind anywhere a full re-index wouldn't.
|
||||
// Reconstruction from the target node's name instead would strip
|
||||
// receiver/qualifier context (`h.greet` → `greet`) and risk a
|
||||
// wrong rebind; edges without refName (pre-#1240, synthesized) are
|
||||
// deliberately NOT resurrected for the same reason.
|
||||
refName: ref.original.referenceName,
|
||||
...(ref.original.referenceKind !== kind ? { refKind: ref.original.referenceKind } : {}),
|
||||
// Uniform marker for function-as-value edges (#756), regardless of
|
||||
// which strategy resolved them (import vs matchFunctionRef) — lets
|
||||
// tooling label "callback registration" and lets validation diff
|
||||
@@ -1012,15 +1024,17 @@ export class ReferenceResolver {
|
||||
);
|
||||
}
|
||||
|
||||
// Delete unresolvable refs too — parity with resolveAndPersistBatched.
|
||||
// Keeping them bought nothing: a ref is only ever retried when its file
|
||||
// is re-extracted, which cascade-deletes and re-inserts its rows anyway.
|
||||
// And it broke the #1187 orphan sweep's invariant — after a COMPLETED
|
||||
// pass the table must hold nothing that pass processed, so that any row
|
||||
// still present belongs to an interrupted run and the sweep can key off
|
||||
// a bare row count.
|
||||
// Park unresolvable refs as status='failed' — parity with
|
||||
// resolveAndPersistBatched. Deleting them was wrong (#1240): a ref whose
|
||||
// own file never changes is otherwise gone forever, so when a DIFFERENT
|
||||
// file later gains the export/symbol that would satisfy it, no sync can
|
||||
// recreate the edge — only a full re-index. Failed rows are excluded from
|
||||
// the pending readers, which preserves the #1187 orphan sweep's
|
||||
// invariant in status form: after a COMPLETED pass nothing it processed
|
||||
// is still 'pending', so any pending row at rest belongs to an
|
||||
// interrupted run and the sweep can key off the pending count.
|
||||
if (result.unresolved.length > 0) {
|
||||
this.queries.deleteSpecificResolvedReferences(
|
||||
this.queries.markReferencesFailed(
|
||||
result.unresolved.map((r) => ({
|
||||
fromNodeId: r.fromNodeId,
|
||||
referenceName: r.referenceName,
|
||||
@@ -1032,6 +1046,49 @@ export class ReferenceResolver {
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Yielding counterpart of {@link resolveAndPersist} for a caller-supplied
|
||||
* ref list — used by sync's failed-ref retry pass (#1240). Same persistence
|
||||
* semantics: resolved refs become edges and their rows are deleted;
|
||||
* still-unresolvable refs are (re-)marked failed (a no-op for rows already
|
||||
* in that status). Yields per-ref because sync can run on the daemon's
|
||||
* liveness-watchdog thread (#850/#1091) and a retry set is unbounded when
|
||||
* a large edit lands many popular symbol names at once.
|
||||
*/
|
||||
async resolveAndPersistListYielding(refs: UnresolvedReference[]): Promise<ResolutionResult> {
|
||||
const maybeYield = createYielder();
|
||||
const result = await this.resolveBatchYielding(refs, maybeYield);
|
||||
|
||||
const PERSIST_CHUNK = 1000;
|
||||
const edges = this.createEdges(result.resolved);
|
||||
for (let i = 0; i < edges.length; i += PERSIST_CHUNK) {
|
||||
this.queries.insertEdges(edges.slice(i, i + PERSIST_CHUNK));
|
||||
await maybeYield();
|
||||
}
|
||||
|
||||
const resolvedKeys = result.resolved.map((r) => ({
|
||||
fromNodeId: r.original.fromNodeId,
|
||||
referenceName: r.original.referenceName,
|
||||
referenceKind: r.original.referenceKind,
|
||||
}));
|
||||
for (let i = 0; i < resolvedKeys.length; i += PERSIST_CHUNK) {
|
||||
this.queries.deleteSpecificResolvedReferences(resolvedKeys.slice(i, i + PERSIST_CHUNK));
|
||||
await maybeYield();
|
||||
}
|
||||
|
||||
const unresolvedKeys = result.unresolved.map((r) => ({
|
||||
fromNodeId: r.fromNodeId,
|
||||
referenceName: r.referenceName,
|
||||
referenceKind: r.referenceKind,
|
||||
}));
|
||||
for (let i = 0; i < unresolvedKeys.length; i += PERSIST_CHUNK) {
|
||||
this.queries.markReferencesFailed(unresolvedKeys.slice(i, i + PERSIST_CHUNK));
|
||||
await maybeYield();
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Second resolution pass for chained static-factory / fluent calls whose
|
||||
* chained method is defined on a SUPERTYPE the receiver's type conforms to —
|
||||
@@ -1162,8 +1219,10 @@ export class ReferenceResolver {
|
||||
byMethod: {} as Record<string, number>,
|
||||
};
|
||||
|
||||
// Process in batches. We always read from offset 0 because resolved refs
|
||||
// are deleted after each batch, shifting the remaining rows forward.
|
||||
// Process in batches. We always read from offset 0 because every ref the
|
||||
// batch processed leaves the pending set (resolved rows are deleted,
|
||||
// unresolvable ones flip to status='failed'), shifting the remaining
|
||||
// pending rows forward.
|
||||
let prevRemaining = Number.POSITIVE_INFINITY;
|
||||
while (true) {
|
||||
const batch = this.queries.getUnresolvedReferencesBatch(0, batchSize);
|
||||
@@ -1198,14 +1257,17 @@ export class ReferenceResolver {
|
||||
await maybeYield();
|
||||
}
|
||||
|
||||
// Delete unresolvable refs from this batch to avoid re-processing them
|
||||
// Park unresolvable refs from this batch as status='failed' so they
|
||||
// leave the pending set (the batch reader and non-progress guard below
|
||||
// only see pending rows) but stay retryable when a later sync adds a
|
||||
// symbol that could satisfy them (#1240).
|
||||
const unresolvedKeys = result.unresolved.map((r) => ({
|
||||
fromNodeId: r.fromNodeId,
|
||||
referenceName: r.referenceName,
|
||||
referenceKind: r.referenceKind,
|
||||
}));
|
||||
for (let i = 0; i < unresolvedKeys.length; i += PERSIST_CHUNK) {
|
||||
this.queries.deleteSpecificResolvedReferences(unresolvedKeys.slice(i, i + PERSIST_CHUNK));
|
||||
this.queries.markReferencesFailed(unresolvedKeys.slice(i, i + PERSIST_CHUNK));
|
||||
await maybeYield();
|
||||
}
|
||||
|
||||
@@ -1232,13 +1294,14 @@ export class ReferenceResolver {
|
||||
// The count-based guard below catches the true no-progress case.
|
||||
|
||||
// Non-progress guard (defense-in-depth). Because we re-read from offset 0
|
||||
// each pass, the unresolved_refs table MUST shrink every iteration — both
|
||||
// resolved and unresolved refs are deleted above. If it didn't shrink, a
|
||||
// each pass, the PENDING population MUST shrink every iteration — resolved
|
||||
// refs are deleted and unresolvable ones are marked failed above, and both
|
||||
// leave the pending set the batch reader sees. If it didn't shrink, a
|
||||
// resolver returned a match whose `original.referenceName` differs from the
|
||||
// stored row, so the keyed delete no-ops, and we'd re-read + re-resolve +
|
||||
// re-insert the same rows forever (the runaway that grew a 99-file repo to
|
||||
// 5M edges / 1.4 GB before the Go-fallback fix). Stop rather than grow the
|
||||
// graph without bound.
|
||||
// stored row, so the keyed delete/update no-ops, and we'd re-read +
|
||||
// re-resolve + re-insert the same rows forever (the runaway that grew a
|
||||
// 99-file repo to 5M edges / 1.4 GB before the Go-fallback fix). Stop
|
||||
// rather than grow the graph without bound.
|
||||
const remaining = this.queries.getUnresolvedReferencesCount();
|
||||
if (remaining >= prevRemaining) break;
|
||||
prevRemaining = remaining;
|
||||
|
||||
Reference in New Issue
Block a user