perf(resolution): fix O(K²) import-node blowup in "Resolving refs" (#915) (#965)

* perf(resolution): resolve imports to definitions, not sibling import nodes (#915)

"Resolving refs" crawled (tens of minutes) on large projects — most painfully
ones mixing a big front-end and back-end. An external package or module imported
across hundreds/thousands of files (react, a shared UI package, Python
logging/typing) is re-declared as an `import` node in every importing file, so
its unresolved import ref fell through to the exact-name matcher, which scored
all K same-named import nodes via findBestMatch — K refs x K candidates = O(K^2)
per package, producing only meaningless import->import edges.

Fix: exclude `import`-kind nodes as name-match targets (they're statements, not
definitions; real import->definition resolution is the import resolver's job).
Plus two safe constant-factor wins in findBestMatch: hoist the per-candidate
ref.filePath split, and skip cross-language candidates when a same-language one
exists (provably the same winner — same-language scores >=50, cross-language
maxes at 35).

Measured: superset (Py+TS) candidates scored 7.5M -> 833K (9x), non-import edges
preserved (+1618 now resolve to real defs), ~22K useless import->import edges
removed; kubernetes (Go) computePathProximity 37.2s -> 5.0s; synthetic 8k-file
mixed repo (K=4000) resolution 16.0s -> 1.7s. Full suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: correct stale better-sqlite3/wasm references to node:sqlite

The SQLite backend has been Node's built-in node:sqlite (real SQLite, WAL + FTS5,
from the bundled runtime) for a while — there is no native build step and no
node-sqlite3-wasm fallback. README and the docs site were already updated; this
catches the stragglers:

- CLAUDE.md: the src/db/ backend description and the sqlite-backend test note.
- src/db/index.ts, src/mcp/tools.ts: two code comments that still blamed "the
  wasm backend" for non-WAL behavior (reworded to "when WAL isn't in effect").

Leaves tree-sitter grammar wasm (web-tree-sitter / --liftoff-only) untouched —
that's a different, still-current use of wasm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(telemetry): drop the dead sqlite_backend field (schema v2)

node:sqlite is now the only backend, so the `index` event's `sqlite_backend`
field was a constant ("native") carrying no signal — and the `install` event
never actually sent it. Remove the field and the backendKind() helper, bump the
telemetry SCHEMA_VERSION 1 -> 2, and update TELEMETRY.md + docs/design/telemetry.md.

The ingest worker is deliberately left tolerant: `index` doesn't require the
field and schema_version validates as nonNegInt(99), so v2 events ingest fine and
old clients still sending v1 + sqlite_backend keep validating too. Added a legacy
comment there explaining it's safe to drop once old-client share is negligible.

telemetry.test.ts: the assertion pinning schema_version and a stale-claim fixture
line updated 1 -> 2. All telemetry tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-06-23 11:26:23 -05:00
committed by GitHub
co-authored by Claude Opus 4.8
parent a89315645d
commit 0a91d0f512
10 changed files with 73 additions and 29 deletions
+51 -8
View File
@@ -317,7 +317,17 @@ export function matchByExactName(
ref: UnresolvedRef,
context: ResolutionContext
): ResolvedRef | null {
const candidates = applyLanguageGate(context.getNodesByName(ref.referenceName), ref);
// `import`-kind nodes are import STATEMENTS, not definitions, so a reference
// resolving to a sibling file's `import` is a meaningless edge — the real
// import→definition resolution is the import resolver's job (resolveViaImport),
// never name-matching here. Excluding them also removes a quadratic blow-up:
// a ubiquitous package (`react`, `@superset-ui/core`, Python `logging`/`typing`)
// is re-declared as an `import` node in every file that imports it, so K
// unresolved import refs each scored K same-named import candidates through
// findBestMatch — O(K²) per package, the dominant cost of "Resolving refs" on
// large import-heavy (front-end + back-end) repos (#915).
const candidates = applyLanguageGate(context.getNodesByName(ref.referenceName), ref)
.filter((n) => n.kind !== 'import');
if (candidates.length === 0) {
return null;
@@ -1119,16 +1129,22 @@ function splitCamelCase(str: string): string[] {
}
/**
* Compute directory proximity between two file paths.
* Returns a score based on the number of shared directory segments.
* Compute directory proximity from a pre-split list of directory segments
* (`filePath1` minus its filename) and a second file path.
* Returns a score based on the number of shared leading directory segments.
* Higher score = closer in directory tree.
*
* Split into a pre-split variant because findBestMatch scores every candidate
* against the SAME `ref.filePath`; re-splitting it per candidate was a hot spot
* on large repos (#915), so the caller splits it once and passes the segments.
*/
function computePathProximity(filePath1: string, filePath2: string): number {
const dir1 = filePath1.split('/').slice(0, -1);
const dir2 = filePath2.split('/').slice(0, -1);
function pathProximityFromDirs(dir1: string[], filePath2: string): number {
const dir2 = filePath2.split('/');
dir2.pop(); // drop filename — matches the original slice(0, -1) on both paths
let shared = 0;
for (let i = 0; i < Math.min(dir1.length, dir2.length); i++) {
const limit = Math.min(dir1.length, dir2.length);
for (let i = 0; i < limit; i++) {
if (dir1[i] === dir2[i]) {
shared++;
} else {
@@ -1140,6 +1156,16 @@ function computePathProximity(filePath1: string, filePath2: string): number {
return Math.min(shared * 15, 80);
}
/**
* Compute directory proximity between two file paths.
* Returns a score based on the number of shared directory segments.
*/
function computePathProximity(filePath1: string, filePath2: string): number {
const dir1 = filePath1.split('/');
dir1.pop();
return pathProximityFromDirs(dir1, filePath2);
}
/**
* Find the best matching node when there are multiple candidates
*/
@@ -1158,7 +1184,24 @@ function findBestMatch(
let bestScore = -1;
let bestNode: Node | null = null;
// Split the ref's path once (it's the same across every candidate) instead of
// re-splitting it inside computePathProximity per candidate (#915 hot spot).
const refDirs = ref.filePath.split('/');
refDirs.pop();
// A same-language candidate ALWAYS outscores a cross-language one: same-language
// scores at least +50 (language bonus), while a cross-language candidate maxes
// out at +35 (80 language, +80 proximity, +25 kind, +10 exported; it can never
// be in the same file). So when any same-language candidate exists, skip the
// cross-language ones — provably the same winner, without paying the per-candidate
// scoring. Cuts the candidate set to same-language size on mixed front-end +
// back-end repos (#915). When ALL candidates are cross-language (a legitimate
// cross-language `calls` bridge), none are skipped and behavior is unchanged.
const hasSameLanguage = candidates.some((c) => c.language === ref.language);
for (const candidate of candidates) {
if (hasSameLanguage && candidate.language !== ref.language) continue;
let score = 0;
// Same file bonus
@@ -1167,7 +1210,7 @@ function findBestMatch(
}
// Directory proximity bonus — strongly prefer same module/package
score += computePathProximity(ref.filePath, candidate.filePath);
score += pathProximityFromDirs(refDirs, candidate.filePath);
// Language matching: strongly prefer same language, penalize cross-language
if (candidate.language === ref.language) {