* perf(resolution): resolve imports to definitions, not sibling import nodes (#915) "Resolving refs" crawled (tens of minutes) on large projects — most painfully ones mixing a big front-end and back-end. An external package or module imported across hundreds/thousands of files (react, a shared UI package, Python logging/typing) is re-declared as an `import` node in every importing file, so its unresolved import ref fell through to the exact-name matcher, which scored all K same-named import nodes via findBestMatch — K refs x K candidates = O(K^2) per package, producing only meaningless import->import edges. Fix: exclude `import`-kind nodes as name-match targets (they're statements, not definitions; real import->definition resolution is the import resolver's job). Plus two safe constant-factor wins in findBestMatch: hoist the per-candidate ref.filePath split, and skip cross-language candidates when a same-language one exists (provably the same winner — same-language scores >=50, cross-language maxes at 35). Measured: superset (Py+TS) candidates scored 7.5M -> 833K (9x), non-import edges preserved (+1618 now resolve to real defs), ~22K useless import->import edges removed; kubernetes (Go) computePathProximity 37.2s -> 5.0s; synthetic 8k-file mixed repo (K=4000) resolution 16.0s -> 1.7s. Full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: correct stale better-sqlite3/wasm references to node:sqlite The SQLite backend has been Node's built-in node:sqlite (real SQLite, WAL + FTS5, from the bundled runtime) for a while — there is no native build step and no node-sqlite3-wasm fallback. README and the docs site were already updated; this catches the stragglers: - CLAUDE.md: the src/db/ backend description and the sqlite-backend test note. - src/db/index.ts, src/mcp/tools.ts: two code comments that still blamed "the wasm backend" for non-WAL behavior (reworded to "when WAL isn't in effect"). Leaves tree-sitter grammar wasm (web-tree-sitter / --liftoff-only) untouched — that's a different, still-current use of wasm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(telemetry): drop the dead sqlite_backend field (schema v2) node:sqlite is now the only backend, so the `index` event's `sqlite_backend` field was a constant ("native") carrying no signal — and the `install` event never actually sent it. Remove the field and the backendKind() helper, bump the telemetry SCHEMA_VERSION 1 -> 2, and update TELEMETRY.md + docs/design/telemetry.md. The ingest worker is deliberately left tolerant: `index` doesn't require the field and schema_version validates as nonNegInt(99), so v2 events ingest fine and old clients still sending v1 + sqlite_backend keep validating too. Added a legacy comment there explaining it's safe to drop once old-client share is negligible. telemetry.test.ts: the assertion pinning schema_version and a stale-claim fixture line updated 1 -> 2. All telemetry tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
a89315645d
commit
0a91d0f512
+1
-1
@@ -143,7 +143,7 @@ export class DatabaseConnection {
|
||||
*
|
||||
* SQLite silently keeps the prior mode if WAL can't be enabled — e.g. on
|
||||
* filesystems without shared-memory support (some network/virtualized mounts,
|
||||
* WSL2 /mnt), and always on the wasm backend. So the effective mode can differ
|
||||
* WSL2 /mnt). So the effective mode can differ
|
||||
* from what `configureConnection` requested. Surfaced in `codegraph status` so
|
||||
* a "database is locked" report is triageable: 'wal' ⇒ readers never block on a
|
||||
* writer; anything else ⇒ they can. See issue #238.
|
||||
|
||||
+2
-1
@@ -936,7 +936,8 @@ export class ToolHandler {
|
||||
// If the path resolves to the default project, reuse the already-open
|
||||
// default instance rather than opening a SECOND connection to the same DB.
|
||||
// A duplicate connection serializes reads against the watcher's auto-sync
|
||||
// writes; on the wasm backend (no WAL) that surfaces as intermittent
|
||||
// writes; when WAL isn't in effect (e.g. a filesystem without shared-memory
|
||||
// support) that surfaces as intermittent
|
||||
// "database is locked" on concurrent tool calls. See issue #238. The
|
||||
// default instance is owned/closed by the server, so it's never cached.
|
||||
if (this.cg && this.cg.getProjectRoot() === resolvedRoot) {
|
||||
|
||||
@@ -317,7 +317,17 @@ export function matchByExactName(
|
||||
ref: UnresolvedRef,
|
||||
context: ResolutionContext
|
||||
): ResolvedRef | null {
|
||||
const candidates = applyLanguageGate(context.getNodesByName(ref.referenceName), ref);
|
||||
// `import`-kind nodes are import STATEMENTS, not definitions, so a reference
|
||||
// resolving to a sibling file's `import` is a meaningless edge — the real
|
||||
// import→definition resolution is the import resolver's job (resolveViaImport),
|
||||
// never name-matching here. Excluding them also removes a quadratic blow-up:
|
||||
// a ubiquitous package (`react`, `@superset-ui/core`, Python `logging`/`typing`)
|
||||
// is re-declared as an `import` node in every file that imports it, so K
|
||||
// unresolved import refs each scored K same-named import candidates through
|
||||
// findBestMatch — O(K²) per package, the dominant cost of "Resolving refs" on
|
||||
// large import-heavy (front-end + back-end) repos (#915).
|
||||
const candidates = applyLanguageGate(context.getNodesByName(ref.referenceName), ref)
|
||||
.filter((n) => n.kind !== 'import');
|
||||
|
||||
if (candidates.length === 0) {
|
||||
return null;
|
||||
@@ -1119,16 +1129,22 @@ function splitCamelCase(str: string): string[] {
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute directory proximity between two file paths.
|
||||
* Returns a score based on the number of shared directory segments.
|
||||
* Compute directory proximity from a pre-split list of directory segments
|
||||
* (`filePath1` minus its filename) and a second file path.
|
||||
* Returns a score based on the number of shared leading directory segments.
|
||||
* Higher score = closer in directory tree.
|
||||
*
|
||||
* Split into a pre-split variant because findBestMatch scores every candidate
|
||||
* against the SAME `ref.filePath`; re-splitting it per candidate was a hot spot
|
||||
* on large repos (#915), so the caller splits it once and passes the segments.
|
||||
*/
|
||||
function computePathProximity(filePath1: string, filePath2: string): number {
|
||||
const dir1 = filePath1.split('/').slice(0, -1);
|
||||
const dir2 = filePath2.split('/').slice(0, -1);
|
||||
function pathProximityFromDirs(dir1: string[], filePath2: string): number {
|
||||
const dir2 = filePath2.split('/');
|
||||
dir2.pop(); // drop filename — matches the original slice(0, -1) on both paths
|
||||
|
||||
let shared = 0;
|
||||
for (let i = 0; i < Math.min(dir1.length, dir2.length); i++) {
|
||||
const limit = Math.min(dir1.length, dir2.length);
|
||||
for (let i = 0; i < limit; i++) {
|
||||
if (dir1[i] === dir2[i]) {
|
||||
shared++;
|
||||
} else {
|
||||
@@ -1140,6 +1156,16 @@ function computePathProximity(filePath1: string, filePath2: string): number {
|
||||
return Math.min(shared * 15, 80);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute directory proximity between two file paths.
|
||||
* Returns a score based on the number of shared directory segments.
|
||||
*/
|
||||
function computePathProximity(filePath1: string, filePath2: string): number {
|
||||
const dir1 = filePath1.split('/');
|
||||
dir1.pop();
|
||||
return pathProximityFromDirs(dir1, filePath2);
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the best matching node when there are multiple candidates
|
||||
*/
|
||||
@@ -1158,7 +1184,24 @@ function findBestMatch(
|
||||
let bestScore = -1;
|
||||
let bestNode: Node | null = null;
|
||||
|
||||
// Split the ref's path once (it's the same across every candidate) instead of
|
||||
// re-splitting it inside computePathProximity per candidate (#915 hot spot).
|
||||
const refDirs = ref.filePath.split('/');
|
||||
refDirs.pop();
|
||||
|
||||
// A same-language candidate ALWAYS outscores a cross-language one: same-language
|
||||
// scores at least +50 (language bonus), while a cross-language candidate maxes
|
||||
// out at +35 (−80 language, +80 proximity, +25 kind, +10 exported; it can never
|
||||
// be in the same file). So when any same-language candidate exists, skip the
|
||||
// cross-language ones — provably the same winner, without paying the per-candidate
|
||||
// scoring. Cuts the candidate set to same-language size on mixed front-end +
|
||||
// back-end repos (#915). When ALL candidates are cross-language (a legitimate
|
||||
// cross-language `calls` bridge), none are skipped and behavior is unchanged.
|
||||
const hasSameLanguage = candidates.some((c) => c.language === ref.language);
|
||||
|
||||
for (const candidate of candidates) {
|
||||
if (hasSameLanguage && candidate.language !== ref.language) continue;
|
||||
|
||||
let score = 0;
|
||||
|
||||
// Same file bonus
|
||||
@@ -1167,7 +1210,7 @@ function findBestMatch(
|
||||
}
|
||||
|
||||
// Directory proximity bonus — strongly prefer same module/package
|
||||
score += computePathProximity(ref.filePath, candidate.filePath);
|
||||
score += pathProximityFromDirs(refDirs, candidate.filePath);
|
||||
|
||||
// Language matching: strongly prefer same language, penalize cross-language
|
||||
if (candidate.language === ref.language) {
|
||||
|
||||
@@ -30,7 +30,10 @@ import { randomUUID } from 'crypto';
|
||||
export const TELEMETRY_ENDPOINT = 'https://telemetry.getcodegraph.com/v1/events';
|
||||
export const TELEMETRY_DOCS = 'https://github.com/colbymchenry/codegraph/blob/main/TELEMETRY.md';
|
||||
|
||||
const SCHEMA_VERSION = 1;
|
||||
// v2: dropped the `sqlite_backend` field from the `index` event — node:sqlite is
|
||||
// now the only backend (the better-sqlite3-native / wasm-fallback split is gone),
|
||||
// so the value was a constant carrying no signal. See TELEMETRY.md.
|
||||
const SCHEMA_VERSION = 2;
|
||||
const MAX_BUFFER_BYTES = 256 * 1024;
|
||||
const MAX_EVENTS_PER_REQUEST = 100;
|
||||
const DEFAULT_FLUSH_TIMEOUT_MS = 1500;
|
||||
@@ -55,18 +58,13 @@ export function bucketDuration(ms: number): '<10s' | '10-60s' | '1-5m' | '5m+' {
|
||||
return '5m+';
|
||||
}
|
||||
|
||||
/** Collapse a backend identifier (e.g. `node-sqlite`) to the schema's enum. */
|
||||
export function backendKind(backend: string): 'native' | 'wasm' {
|
||||
return backend.toLowerCase().includes('wasm') ? 'wasm' : 'native';
|
||||
}
|
||||
|
||||
/**
|
||||
* Shared "a full index completed" event (CLI init/index + installer local
|
||||
* init): language names and coarse buckets only — never paths, file names,
|
||||
* or exact counts. Structurally typed so callers don't need engine imports.
|
||||
*/
|
||||
export function recordIndexEvent(
|
||||
cg: { getStats(): { filesByLanguage: Record<string, number> }; getBackend(): string },
|
||||
cg: { getStats(): { filesByLanguage: Record<string, number> } },
|
||||
result: { filesIndexed: number; durationMs: number },
|
||||
): void {
|
||||
try {
|
||||
@@ -77,7 +75,6 @@ export function recordIndexEvent(
|
||||
languages,
|
||||
file_count_bucket: bucketFileCount(result.filesIndexed),
|
||||
duration_bucket: bucketDuration(result.durationMs),
|
||||
sqlite_backend: backendKind(cg.getBackend()),
|
||||
});
|
||||
} catch {
|
||||
/* telemetry must never break indexing */
|
||||
|
||||
Reference in New Issue
Block a user