Large-codebase indexing died at the end of "Resolving refs" two ways: watchdog kills of healthy work (24k-file Java on Windows, #1212 — third iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale, where v1.3.0 could not complete at any watchdog setting). Root causes: ~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no yield points, several materialized whole-graph snapshots (kotlin expect/actual opened with getAllNodes() — 2M nodes in one array; the C fn-pointer pass retained every C file's contents twice plus every function node), and the post-index WAL checkpoint ran minutes of synchronous IO on the main thread, killing even a successful index at the finish line. The pipeline tail now follows the same discipline as the rest: never hold O(graph) in the heap, yield everywhere. - All synthesis passes stream node-kind scans (cursors, not arrays) and yield on time-budgeted checkpoints; language gates skip passes whose filters a project's file languages provably can't satisfy. - kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded, units stream one file at a time, and the all-functions array + write-only id map are gone; spring reads each .java once, not twice. - runMaintenance moved to a worker thread (own SQLite connection); per-file store commits chunk with yields behind a serialized flush chain (preserving #1015 file-order determinism); resolver warm-up streams the DISTINCT name set; resolution batch-tail and merged-edge inserts run in bounded sub-transactions. - Daemon: fixed a socket-handoff race that could leave a fresh MCP session permanently silent (client-hello tail unshifted into a flowing stream with zero listeners — the long-standing #662 test flake was this real bug); first tool call no longer queues behind the query pool's cold start (pool.ready gate). Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully indexes in 27m8s on a 2-core/6GB container at default heap + default watchdog; llvm-project (180k files) completes under 1GB RSS including kill-and-sync recovery; synthesized-edge and full-graph parity are byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky daemon test passed 25/25 under load. Env-gated diagnostics kept: CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop tracing. Design record: docs/design/main-thread-stall-followup.md. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Fable 5
Claude Fable 5
parent
58b6bf5c60
commit
a3f90089e8
@@ -9,6 +9,12 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixes
|
||||
|
||||
- Indexing very large codebases no longer dies at the end of the "Resolving refs" step. Two failure modes are fixed: on multi-million-symbol projects (e.g. the Linux kernel, ~95,000 files) the final analysis phase ran out of memory and crashed the process outright, and on large projects on slower machines (reported on a 24,000-file Java project on Windows) the same phase could stall long enough that the safety watchdog killed a healthy, still-progressing index at ~98% (#1212). The whole phase now streams its work instead of holding whole-graph snapshots in memory, keeps the process responsive throughout, and skips analysis passes for languages a project doesn't contain — which also makes the tail of indexing noticeably faster on single-language repos. The resulting graph is identical, and a genuinely wedged process is still detected and killed.
|
||||
- Indexing and `codegraph sync` stay responsive through their heaviest internal steps on huge projects: the post-index database maintenance (which on a multi-gigabyte index could stall the process for minutes and get a fully successful index killed by the safety watchdog at the finish line) now runs on a background thread, storing a giant generated file no longer freezes the process mid-extraction, and the reference-resolution bookkeeping between progress updates is broken into small responsive steps. The resulting graph is byte-for-byte identical.
|
||||
- Fixed a race that could leave a freshly-attached MCP session permanently silent: when a client's first messages arrived glued together during the daemon's connection handshake (roughly one attach in five on a busy machine), the daemon could drop them and stop reading that connection entirely — every tool call from that session then hung with no reply. The handshake now hands the connection over losslessly, and the fix is validated by hammering the previously-flaky attach test 25× under load.
|
||||
- The first tool call after the shared daemon starts no longer waits behind the query workers' cold start (which can take many seconds on a busy machine) — it's served directly until the first worker is warm, so a fresh session answers immediately.
|
||||
|
||||
## [1.3.0] - 2026-07-07
|
||||
|
||||
|
||||
@@ -233,21 +233,22 @@ describe('runMaintenance', () => {
|
||||
if (fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('runs without throwing on a fresh database', () => {
|
||||
expect(() => db.runMaintenance()).not.toThrow();
|
||||
it('runs without throwing on a fresh database', async () => {
|
||||
await expect(db.runMaintenance()).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it('runs without throwing after writes', () => {
|
||||
it('runs without throwing after writes', async () => {
|
||||
const q = new QueryBuilder(db.getDb());
|
||||
q.insertNodes([makeNode('n1'), makeNode('n2')]);
|
||||
expect(() => db.runMaintenance()).not.toThrow();
|
||||
await expect(db.runMaintenance()).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it('swallows failures rather than propagating (best-effort)', () => {
|
||||
it('swallows failures rather than propagating (best-effort)', async () => {
|
||||
// Close the DB so the underlying handle would normally throw on any
|
||||
// exec(). runMaintenance must still not propagate.
|
||||
// exec(). runMaintenance (worker on its own connection, or the in-line
|
||||
// fallback) must still not propagate.
|
||||
db.close();
|
||||
expect(() => db.runMaintenance()).not.toThrow();
|
||||
await expect(db.runMaintenance()).resolves.toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -120,6 +120,7 @@ function waitFor<T>(
|
||||
predicate: () => T | undefined | null | false,
|
||||
timeoutMs: number,
|
||||
pollMs = 25,
|
||||
label = '',
|
||||
): Promise<T> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const started = Date.now();
|
||||
@@ -127,7 +128,12 @@ function waitFor<T>(
|
||||
let v: T | undefined | null | false;
|
||||
try { v = predicate(); } catch (e) { return reject(e); }
|
||||
if (v) return resolve(v as T);
|
||||
if (Date.now() - started > timeoutMs) return reject(new Error(`Timed out after ${timeoutMs}ms`));
|
||||
if (Date.now() - started > timeoutMs) {
|
||||
// Name the wait: an async stack loses the await site, so an unlabeled
|
||||
// timeout can't tell WHICH step flaked (the #662 test's recurring
|
||||
// timeout was undiagnosable for exactly this reason).
|
||||
return reject(new Error(`Timed out after ${timeoutMs}ms${label ? ` waiting for: ${label}` : ''}`));
|
||||
}
|
||||
setTimeout(tick, pollMs);
|
||||
};
|
||||
tick();
|
||||
@@ -419,14 +425,25 @@ describe('Shared MCP daemon (issue #411)', () => {
|
||||
const server = spawnServer(tempDir, env);
|
||||
servers.push(server);
|
||||
sendInitialize(server.child, `file://${tempDir}`, 1);
|
||||
await waitFor(() => findResponse(server.stdout, 1), 10000);
|
||||
await waitFor(() => server.stderr.some((l) => l.includes('Attached to shared daemon')), 8000);
|
||||
await waitFor(() => (readLockPid(realRoot) ?? 0) > 0, 8000);
|
||||
await waitFor(() => findResponse(server.stdout, 1), 20000, 25, 'initialize response');
|
||||
await waitFor(() => server.stderr.some((l) => l.includes('Attached to shared daemon')), 8000, 25, 'daemon attach log');
|
||||
await waitFor(() => (readLockPid(realRoot) ?? 0) > 0, 8000, 25, 'daemon pidfile');
|
||||
const daemonPid = readLockPid(realRoot)!;
|
||||
|
||||
// A warm call goes through the daemon.
|
||||
sendMessage(server.child, { jsonrpc: '2.0', id: 2, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } });
|
||||
await waitFor(() => findResponse(server.stdout, 2), 10000);
|
||||
try {
|
||||
await waitFor(() => findResponse(server.stdout, 2), 30000, 25, 'warm tools/call via daemon');
|
||||
} catch (e) {
|
||||
// This is the wait that historically flaked — surface WHERE the request
|
||||
// died: proxy side (stderr) or daemon side (daemon.log).
|
||||
let daemonLog = '<no daemon.log>';
|
||||
try { daemonLog = fs.readFileSync(path.join(realRoot, '.codegraph', 'daemon.log'), 'utf8').split('\n').slice(-25).join('\n'); } catch { /* absent */ }
|
||||
throw new Error(
|
||||
`${(e as Error).message}\ndaemonAlive=${isAlive(daemonPid)} proxyAlive=${isAlive(server.child.pid!)}\n` +
|
||||
`--- proxy stderr tail ---\n${server.stderr.slice(-15).join('')}\n--- daemon.log tail ---\n${daemonLog}`
|
||||
);
|
||||
}
|
||||
|
||||
// Kill the daemon out from under the live proxy.
|
||||
process.kill(daemonPid, 'SIGTERM');
|
||||
@@ -434,7 +451,7 @@ describe('Shared MCP daemon (issue #411)', () => {
|
||||
|
||||
// The proxy must still be alive and still answer — served in-process now.
|
||||
expect(isAlive(server.child.pid!)).toBe(true);
|
||||
await waitFor(() => server.stderr.some((l) => l.includes('serving this session in-process')), 8000);
|
||||
await waitFor(() => server.stderr.some((l) => l.includes('serving this session in-process')), 8000, 25, 'in-process failover log');
|
||||
sendMessage(server.child, { jsonrpc: '2.0', id: 3, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } });
|
||||
const resp = await waitFor(() => findResponse(server.stdout, 3), 15000);
|
||||
expect(resp.result !== undefined || resp.error !== undefined).toBe(true);
|
||||
|
||||
@@ -171,4 +171,28 @@ describe('QueryPool', () => {
|
||||
expect(res.isError).toBe(true);
|
||||
expect(pool.healthy).toBe(false);
|
||||
});
|
||||
|
||||
it('is not `ready` until a worker completes its cold start (#662 first-call stall)', async () => {
|
||||
// A worker cold start is seconds (tens under load); a call queued behind it
|
||||
// waits for the 45s busy backstop with nothing served. The ToolHandler must
|
||||
// be able to see "no warm worker yet" and dispatch in-process instead — so
|
||||
// `ready` is false before the first 'ready' handshake and true after.
|
||||
// (FakeWorker posts 'ready' on a macrotask — the synchronous check below
|
||||
// observes the cold-start window.)
|
||||
const pool = new QueryPool({ root: '/x', size: 1, createWorker: () => new FakeWorker((m) => ({ result: ok(`r:${m.toolName}`) })) });
|
||||
expect(pool.ready).toBe(false); // eager worker spawned but not yet warm
|
||||
await sleep(5); // let the ready handshake land
|
||||
expect(pool.ready).toBe(true);
|
||||
const res = await pool.run('codegraph_status', {});
|
||||
expect(res.content[0].text).toBe('r:codegraph_status');
|
||||
await pool.destroy();
|
||||
expect(pool.ready).toBe(false); // destroyed pool must not be selected
|
||||
});
|
||||
|
||||
it('a failed cold start (ready ok:false) does not mark the pool ready', async () => {
|
||||
const pool = new QueryPool({ root: '/x', size: 1, createWorker: () => new FakeWorker(() => ({ hang: true }), /* readyOk */ false) });
|
||||
await sleep(5);
|
||||
expect(pool.ready).toBe(false); // hard open failure — keep serving in-process
|
||||
await pool.destroy();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
/**
|
||||
* Synthesis-tail scaling regressions (#1212).
|
||||
*
|
||||
* On a 2M-node graph (Linux kernel) the dynamic-edge synthesis tail OOM'd
|
||||
* Node's default heap and/or starved the #850 liveness watchdog: the kotlin
|
||||
* expect/actual pass opened with `getAllNodes()` (hydrating the entire node
|
||||
* table into one array), and most passes ran start-to-finish with no yield
|
||||
* points. The fix streams every whole-kind scan, filters the kotlin pass
|
||||
* SQL-side, and language-gates passes off the files table.
|
||||
*
|
||||
* These tests pin the query-level building blocks and the end-to-end kotlin
|
||||
* bridge so the memory fix can't silently change what gets synthesized.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import * as os from 'node:os';
|
||||
import { CodeGraph } from '../src';
|
||||
|
||||
describe('synthesis-tail scaling (#1212)', () => {
|
||||
let dir: string;
|
||||
beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'synth-scaling-')); });
|
||||
afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); });
|
||||
|
||||
it('kotlin expect/actual still bridges through the streamed decorator query', async () => {
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'Platform.kt'),
|
||||
`package com.example.shared
|
||||
|
||||
expect fun platformName(): String
|
||||
`
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'Platform.jvm.kt'),
|
||||
`package com.example.shared
|
||||
|
||||
actual fun platformName(): String = "JVM"
|
||||
`
|
||||
);
|
||||
const cg = await CodeGraph.init(dir);
|
||||
await cg.indexAll();
|
||||
|
||||
const db = (cg as any).db.db;
|
||||
const edges = db
|
||||
.prepare(
|
||||
`SELECT e.source, e.target FROM edges e
|
||||
WHERE json_extract(e.metadata,'$.synthesizedBy') = 'kotlin-expect-actual'`
|
||||
)
|
||||
.all();
|
||||
expect(edges.length).toBeGreaterThanOrEqual(1);
|
||||
cg.close();
|
||||
});
|
||||
|
||||
it('iterateNodesByLanguageWithDecorator matches getAllNodes().filter exactly', async () => {
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'A.kt'),
|
||||
`package p
|
||||
|
||||
actual fun realActual(): Int = 1
|
||||
`
|
||||
);
|
||||
// TypeScript decorator whose name CONTAINS "actual" — the SQL LIKE
|
||||
// pre-filter must not surface it as a kotlin actual.
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'b.ts'),
|
||||
`function actual(target: object): void {}
|
||||
class C {
|
||||
m(): number { return 1; }
|
||||
}
|
||||
`
|
||||
);
|
||||
const cg = await CodeGraph.init(dir);
|
||||
await cg.indexAll();
|
||||
|
||||
const queries = (cg as unknown as { queries: import('../src/db/queries').QueryBuilder }).queries;
|
||||
const streamed = [...queries.iterateNodesByLanguageWithDecorator('kotlin', 'actual')]
|
||||
.filter((n) => n.decorators?.includes('actual'))
|
||||
.map((n) => n.id)
|
||||
.sort();
|
||||
const reference = queries
|
||||
.getAllNodes()
|
||||
.filter((n) => n.language === 'kotlin' && !!n.decorators?.includes('actual'))
|
||||
.map((n) => n.id)
|
||||
.sort();
|
||||
expect(streamed).toEqual(reference);
|
||||
expect(reference.length).toBeGreaterThanOrEqual(1); // the fixture really has one
|
||||
cg.close();
|
||||
});
|
||||
|
||||
it('getDistinctFileLanguages reports exactly the languages present', async () => {
|
||||
fs.writeFileSync(path.join(dir, 'x.ts'), 'export const a = 1;\n');
|
||||
fs.writeFileSync(path.join(dir, 'y.py'), 'def f():\n return 1\n');
|
||||
const cg = await CodeGraph.init(dir);
|
||||
await cg.indexAll();
|
||||
|
||||
const queries = (cg as unknown as { queries: import('../src/db/queries').QueryBuilder }).queries;
|
||||
const langs = queries.getDistinctFileLanguages();
|
||||
expect(langs.has('typescript')).toBe(true);
|
||||
expect(langs.has('python')).toBe(true);
|
||||
expect(langs.has('kotlin')).toBe(false);
|
||||
cg.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,188 @@
|
||||
# Main-thread stall budget — extraction & resolution follow-up
|
||||
|
||||
**Status: IMPLEMENTED** (same branch as the #1212 tail fix — attribution runs
|
||||
promoted "suspects" to proven culprits fast enough to justify shipping
|
||||
together). What landed, per suspect:
|
||||
|
||||
- **Post-index maintenance — the proven killer, not on the original suspect
|
||||
list.** The first full kernel `init` on the FIXED tail completed every
|
||||
synthesis pass (cFnPtr alone ran 433s at default heap, yielding throughout)
|
||||
and was then SIGKILLed by the default-window watchdog at
|
||||
`db.runMaintenance()`: `PRAGMA optimize` + `wal_checkpoint(PASSIVE)` over a
|
||||
4.2GB DB with a 593MB WAL is minutes of synchronous IO on 2 cores.
|
||||
`runMaintenance` now runs on a worker thread with its own connection
|
||||
(checkpointing from a second connection is standard; `PRAGMA optimize`
|
||||
persists stats in sqlite_stat tables), with a bounded in-line fallback that
|
||||
skips the checkpoint (close() checkpoints after the CLI disarms the
|
||||
watchdog).
|
||||
- **Per-file store commits:** `storeExtractionResult` chunks its node/edge/ref
|
||||
inserts (2,000 rows) with time-budgeted yields between; the ordered-commit
|
||||
pump serializes async stores on a promise chain (preserving the #1015
|
||||
file-order determinism invariant) and its backpressure now also waits on the
|
||||
commit chain so the parse buffer stays bounded.
|
||||
- **Resolver warm-up:** `warmCachesYielding` streams the DISTINCT name set
|
||||
with yields (the sync `warmCaches` stays for non-async callers). The 28.2s
|
||||
`sync` stall dropped to ≤4s total across the whole sync.
|
||||
- **Resolution batch-tail:** edge inserts and keyed deletes run in 1,000-row
|
||||
sub-transactions with yields between (crash semantics unchanged — the batch
|
||||
was already several transactions, and #1187's sweep re-resolves leftovers).
|
||||
- **Scan:** attributed (phase timings now in the code, `[phase-timing]` on
|
||||
`CODEGRAPH_SYNTH_TIMINGS`) — it is the synchronous git enumeration
|
||||
(`getGitVisibleFiles`/`collectGitFiles`), NOT a hash loop. See "Accepted
|
||||
residuals" below for why it was left synchronous.
|
||||
|
||||
**Verification:** full-graph parity (every node id + edge, sorted dump diff)
|
||||
byte-identical on fresh redis and vim indexes, baseline vs fixed; full test
|
||||
suite green; kernel `sync` worst stall 28.2s → ~4s; ES synthesis tail worst
|
||||
stall ≤2.7s.
|
||||
|
||||
**Acceptance gate PASSED:** fresh full kernel `init` (70,129 indexed files,
|
||||
2,048,673 nodes / 6,402,391 edges) completed in **27m 8s** on the 2-core/6GB
|
||||
container at Node's default heap with the default 60s watchdog — `EXIT 0`,
|
||||
identical node/edge counts to the pre-fix partial runs, maintenance 48.8s
|
||||
off-thread with the WAL fully checkpointed (0 bytes). v1.3.0 could not finish
|
||||
this repo at all (OOM at default heap; watchdog kill at the maintenance step
|
||||
even with the tail fixed). Post-run, the one genuine synchronous span the run
|
||||
exposed — the merged synthesized-edge insert (~275k rows, 20.2s in one
|
||||
transaction) — was chunked (2k rows + yield) like the rest; redis parity
|
||||
re-verified byte-identical after.
|
||||
|
||||
## Accepted residuals (measured, documented, deliberately not fixed)
|
||||
|
||||
- **Git enumeration (scan): 2.2–10.5s** single sync span on ~95k-file repos.
|
||||
Fixing it means async-ifying `collectGitFiles`' recursive gitlink/submodule
|
||||
logic (#1038/#1065) or forking sync/async variants — high regression risk
|
||||
for a CPU-bound span ~6× under the watchdog window even on a 2-core
|
||||
container (its cost does not get the Windows/Defender per-file-IO
|
||||
multiplier; it scales with CPU only).
|
||||
- **End-of-sync aggregates: ~2.7s** (count recompute / vocab backfill on a
|
||||
4.2GB DB).
|
||||
- **Warm-up first chunk: ~2.6s** — the DISTINCT name scan's initial sort
|
||||
chunk before the first cursor row arrives; the rest of the scan yields.
|
||||
- **Worker-contention timer lag on tiny containers** — with 2 cpuset cores,
|
||||
the off-thread checkpoint (and the parse pool early in the run) can delay
|
||||
main-loop timers 15–20s even though the main thread executes nothing. The
|
||||
stall monitor and the watchdog heartbeat both measure timer latency, so on
|
||||
a ~1-core box a long checkpoint could still starve heartbeats; if that ever
|
||||
reproduces, the mitigations are a niced worker or heartbeat-side allowance,
|
||||
not more yields.
|
||||
|
||||
If any of these ever shows up in a real watchdog kill, the async-refactor
|
||||
shape for the scan is: thread a `MaybeYield` through `collectGitFiles`'
|
||||
per-line loop and make `getGitVisibleFiles` async, keeping `scanDirectory`
|
||||
(sync) on the walk fallback only.
|
||||
|
||||
---
|
||||
|
||||
*Original plan below, kept for the record.*
|
||||
|
||||
## Context
|
||||
|
||||
The #850 liveness watchdog SIGKILLs the indexer when its event loop stalls past
|
||||
the window (default 60s). #1091 → #1122/#1137 → #1212 each moved the fix deeper:
|
||||
per-batch yields, per-ref yields, then (with #1212) yields + streamed queries +
|
||||
language gates across the entire dynamic-edge synthesis tail, which eliminated
|
||||
the 14–57s single-pass stalls and the two whole-graph OOMs.
|
||||
|
||||
While validating #1212 with an event-loop stall monitor over *full* `init` runs
|
||||
(Linux kernel, 70k indexed files / 2.05M nodes, 2-core 6GB container; and
|
||||
llvm-project, 180k tracked files, macOS), the phases **before** the synthesis
|
||||
tail showed recurring single stalls that nothing currently yields through:
|
||||
|
||||
| Run | Phase | Observed single stalls |
|
||||
|---|---|---|
|
||||
| kernel (2 cores) | initial scan (t+14s, t+22s) | 5.1s, 10.5s |
|
||||
| kernel (2 cores) | extraction (t+980–1080s) | 3.0–3.3s |
|
||||
| kernel (2 cores) | extraction→resolution boundary (t+1354s) | 8.5s |
|
||||
| llvm (mac, fast) | extraction / early resolution (t+1000–1320s) | 5–14s, recurring |
|
||||
| kernel (2 cores) | `codegraph sync` on the same DB (110 files) | **28.2s** (single stall) |
|
||||
|
||||
None of these approaches 60s on the tested hardware, and none are regressions —
|
||||
they pre-date #1212. But the #1212 pattern (Windows NTFS + Defender, small VMs)
|
||||
multiplies per-file and per-transaction costs several-fold, and 14s × a few-fold
|
||||
is a watchdog kill. These are the spans that will produce the *fourth* iteration
|
||||
of this bug class if left unmeasured.
|
||||
|
||||
## Suspects (with code locations)
|
||||
|
||||
1. **Per-file store commits on the main thread** —
|
||||
`ExtractionOrchestrator.storeExtractionResult` (`src/extraction/index.ts:2065`)
|
||||
runs one synchronous transaction per file (`insertNodes` + `insertEdges` +
|
||||
unresolved-ref batch + FTS triggers). A giant generated file (llvm has
|
||||
many multi-MB generated `.inc`/`.cpp`) inserts tens of thousands of nodes in
|
||||
one unyielding span. The parse pool (#1015) moved *parsing* off-thread; the
|
||||
*commit* is still a single main-thread block per file.
|
||||
2. **Resolver cache warm-up** — `warmCaches` (`src/resolution/index.ts:319`)
|
||||
calls `getAllNodeNames()` (`src/db/queries.ts:1879`, `SELECT DISTINCT name`
|
||||
over the whole node table) plus `getAllFilePaths()` synchronously. On the
|
||||
kernel's 2M-row table the DISTINCT alone is seconds; it is the prime suspect
|
||||
for the 8.5s boundary stall and the 28.2s `sync` stall (sync also enters
|
||||
resolution via the orphan sweep, #1191).
|
||||
3. **Resolution batch-tail DB ops** — between the per-ref yields,
|
||||
`resolveAndPersistBatched` (`src/resolution/index.ts`) runs per-5000-ref
|
||||
synchronous spans: `insertEdges(batch)`,
|
||||
`deleteSpecificResolvedReferences` × 2 (a 5000-statement transaction), and
|
||||
`getUnresolvedReferencesCount()`. On a multi-GB DB each is a solid block.
|
||||
4. **Initial scan** (kernel t+14/22s) — file enumeration + content hashing
|
||||
before extraction starts. Unattributed; measure before assuming.
|
||||
|
||||
## Diagnosis plan (before any fix)
|
||||
|
||||
Extend the env-gated timing that located #1212 (`CODEGRAPH_SYNTH_TIMINGS`) to
|
||||
the suspects — or add a sibling `CODEGRAPH_PHASE_TIMINGS` — so each suspect
|
||||
logs spans >250ms with a label:
|
||||
|
||||
- wrap `storeExtractionResult` (log file path + node count when slow — this
|
||||
also identifies the offending generated files),
|
||||
- wrap `warmCaches` (split `getAllNodeNames` vs `getAllFilePaths`),
|
||||
- wrap the three batch-tail ops in `resolveAndPersistBatched`,
|
||||
- wrap the scan phase.
|
||||
|
||||
Re-run the stall monitor + timings on the two existing indexes (assets below).
|
||||
Attribution first: the fix for each suspect is different, and #1180 showed the
|
||||
first guess is often wrong.
|
||||
|
||||
## Fix sketches (per suspect, once confirmed)
|
||||
|
||||
1. **Chunked per-file commits:** split a file's node/edge/ref inserts into
|
||||
bounded sub-transactions (e.g. 2–5k rows) with `maybeYield()` between chunks.
|
||||
**Invariant to preserve:** files must still commit in scan order, whole-file
|
||||
at a time from the resolver's perspective (#1015 — resolution disambiguates
|
||||
same-named candidates by insertion order; chunking *within* one file keeps
|
||||
the order stable). The existing index-completeness marker (`index_state`)
|
||||
already covers a mid-file kill.
|
||||
2. **Yielding warm-up:** stream `SELECT DISTINCT name` with a cursor
|
||||
(`stmt.iterate()`), building the Set with a periodic `maybeYield()` — an
|
||||
async `warmCachesYielding()` used from the async entry points
|
||||
(`resolveAndPersistBatched`, the sync path), leaving the sync `warmCaches()`
|
||||
for callers that can't await. Memory is unchanged (the Set already exists).
|
||||
3. **Chunked batch-tail ops:** split the keyed-delete transaction and the edge
|
||||
insert into sub-transactions with yields between, same pattern as (1).
|
||||
`getUnresolvedReferencesCount` is an indexed aggregate; leave it unless
|
||||
timing says otherwise.
|
||||
4. **Scan:** measure first; likely chunk the hash loop with yields.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Instrumented full `init` on the kernel index (2-core/6GB container) and
|
||||
llvm-project shows **no single event-loop stall > ~2s** in any phase.
|
||||
- `codegraph sync` on the kernel DB shows the same bound (kills the 28.2s span).
|
||||
- Graph parity: byte-identical node/edge sets on a re-index of at least
|
||||
elasticsearch + redis (the #1212 parity harness in the session scratchpad
|
||||
automates the synthesized-edge half; extraction parity = compare
|
||||
`getNodeAndEdgeCount` + a sorted node-id dump).
|
||||
- No end-to-end throughput regression beyond noise (< ~5%) on the same runs —
|
||||
chunked transactions can slow bulk inserts; measure, don't assume.
|
||||
|
||||
## Repro assets (from the #1212 investigation, 2026-07-08)
|
||||
|
||||
- Docker container `cg1212` (2 cores / 6GB, node:22-bookworm) with the Linux
|
||||
kernel cloned at `/work/linux` and its 4.2GB index.
|
||||
- llvm-project (180,074 files) + elasticsearch (45k) + redis + vim clones with
|
||||
indexes in the session scratchpad.
|
||||
- `stall-monitor.cjs` (preload; logs event-loop gaps >1s with timestamps),
|
||||
`synth-only.mjs` / `synth-watchdog.mjs` (drive resolution+synthesis directly
|
||||
against an existing index — ~2 min iteration instead of a 40-min re-index),
|
||||
`parity.mjs` (synthesized-edge set differ).
|
||||
- The #1091 methodology note applies: a real CLI run at a lowered
|
||||
`CODEGRAPH_WATCHDOG_TIMEOUT_MS` is the authoritative kill/no-kill test.
|
||||
+52
-11
@@ -198,8 +198,8 @@ export class DatabaseConnection {
|
||||
}
|
||||
|
||||
/**
|
||||
* Lightweight, non-blocking maintenance to run after bulk writes
|
||||
* (indexAll, sync). Two operations:
|
||||
* Lightweight maintenance to run after bulk writes (indexAll, sync).
|
||||
* Two operations:
|
||||
*
|
||||
* - `PRAGMA optimize` — incremental ANALYZE; SQLite only re-analyzes
|
||||
* tables whose row counts changed materially since the last
|
||||
@@ -211,19 +211,60 @@ export class DatabaseConnection {
|
||||
* unboundedly between automatic checkpoints (auto-fires at 1000
|
||||
* pages by default; large indexAll runs blow past that).
|
||||
*
|
||||
* Both operations are silently swallowed on failure — they're a
|
||||
* best-effort optimization, never load-bearing for correctness.
|
||||
* Runs on a WORKER THREAD with its own connection: on a multi-GB index
|
||||
* these pragmas are minutes of synchronous IO (a 95k-file kernel index
|
||||
* left a 593MB WAL whose checkpoint alone blew the #850 watchdog's 60s
|
||||
* window and got a COMPLETED index SIGKILLed at the finish line). WAL
|
||||
* checkpointing from a second connection is standard SQLite; `PRAGMA
|
||||
* optimize` persists its statistics in sqlite_stat tables, so the main
|
||||
* connection benefits the same. The main thread just awaits a message,
|
||||
* so the event loop — and the watchdog heartbeat — keep turning.
|
||||
*
|
||||
* Everything is silently swallowed on failure — best-effort
|
||||
* optimization, never load-bearing for correctness. If worker threads
|
||||
* are unavailable, falls back to a bounded in-line `PRAGMA optimize`
|
||||
* and SKIPS the checkpoint (the final close() checkpoints after the
|
||||
* CLI has already disarmed its watchdog).
|
||||
*/
|
||||
runMaintenance(): void {
|
||||
try {
|
||||
this.db.exec('PRAGMA optimize');
|
||||
} catch {
|
||||
// ignore
|
||||
async runMaintenance(): Promise<void> {
|
||||
// In-memory / test databases: nothing worth a worker round-trip.
|
||||
if (!this.dbPath || this.dbPath === ':memory:') {
|
||||
try { this.db.exec('PRAGMA optimize'); } catch { /* ignore */ }
|
||||
try { this.db.exec('PRAGMA wal_checkpoint(PASSIVE)'); } catch { /* ignore */ }
|
||||
return;
|
||||
}
|
||||
try {
|
||||
this.db.exec('PRAGMA wal_checkpoint(PASSIVE)');
|
||||
const { Worker } = await import('node:worker_threads');
|
||||
const workerSource = `
|
||||
const { workerData, parentPort } = require('node:worker_threads');
|
||||
try {
|
||||
const { DatabaseSync } = require('node:sqlite');
|
||||
const db = new DatabaseSync(workerData.dbPath);
|
||||
try { db.exec('PRAGMA analysis_limit=1000'); } catch {}
|
||||
try { db.exec('PRAGMA optimize'); } catch {}
|
||||
try { db.exec('PRAGMA wal_checkpoint(PASSIVE)'); } catch {}
|
||||
try { db.close(); } catch {}
|
||||
} catch {}
|
||||
parentPort.postMessage('done');
|
||||
`;
|
||||
await new Promise<void>((resolve) => {
|
||||
let settled = false;
|
||||
const finish = (): void => {
|
||||
if (!settled) { settled = true; resolve(); }
|
||||
};
|
||||
try {
|
||||
const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath } });
|
||||
worker.once('message', () => { void worker.terminate(); finish(); });
|
||||
worker.once('error', () => { void worker.terminate(); finish(); });
|
||||
worker.once('exit', finish);
|
||||
} catch {
|
||||
finish();
|
||||
}
|
||||
});
|
||||
} catch {
|
||||
// ignore (e.g., not in WAL mode)
|
||||
// Worker threads unavailable — bounded in-line fallback, no checkpoint.
|
||||
try { this.db.exec('PRAGMA analysis_limit=1000'); } catch { /* ignore */ }
|
||||
try { this.db.exec('PRAGMA optimize'); } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -880,6 +880,37 @@ export class QueryBuilder {
|
||||
return rows.map(rowToNode);
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream nodes of one language whose `decorators` JSON array contains
|
||||
* `decorator`. The LIKE on the JSON text is a cheap index-free pre-filter
|
||||
* (a decorator name can appear as a substring of another), so callers must
|
||||
* still exact-check `node.decorators.includes(decorator)`. Exists so the
|
||||
* kotlin expect/actual synthesizer never materializes the whole node table
|
||||
* the way `getAllNodes().filter(...)` did — that array alone OOM'd Node's
|
||||
* default heap on a 2M-node graph (#1212).
|
||||
*/
|
||||
*iterateNodesByLanguageWithDecorator(language: Language, decorator: string): IterableIterator<Node> {
|
||||
// Fresh statement per call — an iterator holds an open cursor (see
|
||||
// iterateNodesByKind).
|
||||
const stmt = this.db.prepare(
|
||||
"SELECT * FROM nodes WHERE language = ? AND decorators LIKE '%' || ? || '%'"
|
||||
);
|
||||
for (const row of stmt.iterate(language, `"${decorator}"`)) {
|
||||
yield rowToNode(row as NodeRow);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Distinct languages present in the files table. One indexed aggregate —
|
||||
* lets the dynamic-edge synthesizers skip passes for languages the project
|
||||
* doesn't contain at all (a Kotlin pass has no work on a pure-C repo), so
|
||||
* their cost is zero rather than a full-graph scan that finds nothing (#1212).
|
||||
*/
|
||||
getDistinctFileLanguages(): Set<string> {
|
||||
const rows = this.db.prepare('SELECT DISTINCT language FROM files').all() as Array<{ language: string }>;
|
||||
return new Set(rows.map((r) => r.language));
|
||||
}
|
||||
|
||||
/**
|
||||
* Get nodes by exact name match (uses idx_nodes_name index)
|
||||
*/
|
||||
@@ -1853,6 +1884,19 @@ export class QueryBuilder {
|
||||
return rows.map((r) => r.name);
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream the distinct node names one row at a time — the incremental
|
||||
* counterpart to {@link getAllNodeNames} for callers that need to yield
|
||||
* to the event loop mid-scan (resolver cache warm-up on multi-million-node
|
||||
* indexes). Fresh statement per call: the iterator holds an open cursor.
|
||||
*/
|
||||
*iterateNodeNames(): IterableIterator<string> {
|
||||
const stmt = this.db.prepare('SELECT DISTINCT name FROM nodes');
|
||||
for (const row of stmt.iterate()) {
|
||||
yield (row as { name: string }).name;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get unresolved references scoped to specific file paths.
|
||||
* Uses the idx_unresolved_file_path index for efficient lookup.
|
||||
|
||||
+78
-31
@@ -28,6 +28,7 @@ import { validatePathWithinRoot, normalizePath } from '../utils';
|
||||
import ignore, { Ignore } from 'ignore';
|
||||
import { detectFrameworks } from '../resolution/frameworks';
|
||||
import type { ResolutionContext } from '../resolution/types';
|
||||
import { createYielder, type MaybeYield } from '../resolution/cooperative-yield';
|
||||
|
||||
/**
|
||||
* Number of files to read in parallel during indexing.
|
||||
@@ -1479,6 +1480,10 @@ export class ExtractionOrchestrator {
|
||||
total: 0,
|
||||
});
|
||||
|
||||
// Phase attribution to stderr (same opt-in as the synthesis timings):
|
||||
// early-run 5-10s single stalls were observed on 95k-file repos but never
|
||||
// attributed — these labels settle scan vs framework-detect vs grammars.
|
||||
const tScan = Date.now();
|
||||
const files = await scanDirectoryAsync(this.rootDir, (current, file) => {
|
||||
onProgress?.({
|
||||
phase: 'scanning',
|
||||
@@ -1487,6 +1492,7 @@ export class ExtractionOrchestrator {
|
||||
currentFile: file,
|
||||
});
|
||||
});
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] scan: ${Date.now() - tScan}ms (${files.length} files)`);
|
||||
|
||||
// Detect frameworks once per indexAll run using the scanned file list.
|
||||
// Names are passed to each parse call so framework-specific extractors
|
||||
@@ -1494,7 +1500,9 @@ export class ExtractionOrchestrator {
|
||||
// Framework detection is reset each run so adding e.g. requirements.txt
|
||||
// between runs is picked up without restarting the process.
|
||||
this.detectedFrameworkNames = null;
|
||||
const tFw = Date.now();
|
||||
const frameworkNames = this.ensureDetectedFrameworks(files);
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] framework-detect: ${Date.now() - tFw}ms`);
|
||||
|
||||
if (signal?.aborted) {
|
||||
return {
|
||||
@@ -1586,13 +1594,19 @@ export class ExtractionOrchestrator {
|
||||
let nextToStore = 0; // cursor: next sequence to commit
|
||||
let aborted = false;
|
||||
|
||||
const storeResult = (filePath: string, content: string, stats: fs.Stats, result: ExtractionResult): void => {
|
||||
// Yielder for the in-order commit path: a single giant generated file's
|
||||
// store is otherwise one unyielding multi-second transaction span on the
|
||||
// main thread (5–14s single stalls measured on llvm-project), starving
|
||||
// the #850 watchdog heartbeat on slow hardware.
|
||||
const commitYield = createYielder();
|
||||
|
||||
const storeResult = async (filePath: string, content: string, stats: fs.Stats, result: ExtractionResult): Promise<void> => {
|
||||
processed++;
|
||||
|
||||
// Store in database on main thread (SQLite is not thread-safe)
|
||||
if (result.nodes.length > 0 || result.errors.length === 0) {
|
||||
const language = detectLanguage(filePath, content, overrides);
|
||||
this.storeExtractionResult(filePath, content, language, stats, result);
|
||||
await this.storeExtractionResult(filePath, content, language, stats, result, commitYield);
|
||||
}
|
||||
|
||||
if (result.errors.length > 0) {
|
||||
@@ -1637,17 +1651,31 @@ export class ExtractionOrchestrator {
|
||||
|
||||
// Commit buffered parses to the DB in file order, advancing the cursor over
|
||||
// contiguous completed results. Runs after each parse settles (and once more
|
||||
// after the drain). storeResult / recordParseFailure run here single-threaded,
|
||||
// so shared counters and SQLite writes never race despite parallel parsing.
|
||||
const flushOrdered = (): void => {
|
||||
if (aborted) return;
|
||||
while (completed.has(nextToStore)) {
|
||||
const item = completed.get(nextToStore)!;
|
||||
completed.delete(nextToStore);
|
||||
nextToStore++;
|
||||
if (item.ok) storeResult(item.filePath, item.content, item.stats, item.result);
|
||||
else recordParseFailure(item.filePath, item.err);
|
||||
}
|
||||
// after the drain). storeResult is now async (it yields between chunked
|
||||
// inserts), so commits are SERIALIZED on a promise chain — concurrent parse
|
||||
// completions append to the chain instead of interleaving mid-store, which
|
||||
// preserves both the file-order commit invariant (#1015: resolution
|
||||
// disambiguates same-named candidates by insertion order) and the
|
||||
// single-writer discipline for SQLite. Errors are recorded and re-thrown
|
||||
// at the drain, matching the old synchronous propagation.
|
||||
let flushChain: Promise<void> = Promise.resolve();
|
||||
let flushError: unknown = null;
|
||||
const flushOrdered = (): Promise<void> => {
|
||||
flushChain = flushChain.then(async () => {
|
||||
if (aborted || flushError) return;
|
||||
try {
|
||||
while (completed.has(nextToStore)) {
|
||||
const item = completed.get(nextToStore)!;
|
||||
completed.delete(nextToStore);
|
||||
nextToStore++;
|
||||
if (item.ok) await storeResult(item.filePath, item.content, item.stats, item.result);
|
||||
else recordParseFailure(item.filePath, item.err);
|
||||
}
|
||||
} catch (err) {
|
||||
flushError = err;
|
||||
}
|
||||
});
|
||||
return flushChain;
|
||||
};
|
||||
|
||||
// Dispatch one file's parse (parses run concurrently across the pool), tagged
|
||||
@@ -1670,10 +1698,13 @@ export class ExtractionOrchestrator {
|
||||
// buffered), not just in-flight: a slow file sitting at the commit cursor
|
||||
// lets later parses finish and buffer, which would otherwise grow without
|
||||
// bound. Wait for parses to settle (each may advance the cursor) until the
|
||||
// window has room. `inFlight.size > 0` guards against an empty race — the
|
||||
// cursor file is always still in flight when the window is full.
|
||||
while (nextSeq - nextToStore >= windowSize && inFlight.size > 0) {
|
||||
await Promise.race(inFlight);
|
||||
// window has room. When nothing is in flight but the window is still full,
|
||||
// the async commit chain is what's behind — await it so the cursor
|
||||
// advances (buffered items hold whole file contents, so this bound is
|
||||
// load-bearing for memory).
|
||||
while (nextSeq - nextToStore >= windowSize) {
|
||||
if (inFlight.size > 0) await Promise.race(inFlight);
|
||||
else await flushOrdered();
|
||||
}
|
||||
};
|
||||
|
||||
@@ -1751,7 +1782,8 @@ export class ExtractionOrchestrator {
|
||||
// then commit any results the cursor hasn't reached yet.
|
||||
if (!aborted) {
|
||||
await Promise.all(inFlight);
|
||||
flushOrdered();
|
||||
await flushOrdered();
|
||||
if (flushError) throw flushError;
|
||||
}
|
||||
|
||||
if (signal?.aborted || aborted) {
|
||||
@@ -1823,7 +1855,7 @@ export class ExtractionOrchestrator {
|
||||
if (result.nodes.length > 0 || result.errors.length === 0) {
|
||||
const language = detectLanguage(filePath, content, overrides);
|
||||
const stats = await fsp.stat(path.join(this.rootDir, filePath));
|
||||
this.storeExtractionResult(filePath, content, language, stats, result);
|
||||
await this.storeExtractionResult(filePath, content, language, stats, result, commitYield);
|
||||
|
||||
const idx = errors.indexOf(errEntry);
|
||||
if (idx >= 0) errors.splice(idx, 1);
|
||||
@@ -1873,7 +1905,7 @@ export class ExtractionOrchestrator {
|
||||
if (result.nodes.length > 0 || result.errors.length === 0) {
|
||||
const language = detectLanguage(filePath, fullContent, overrides);
|
||||
const stats = await fsp.stat(path.join(this.rootDir, filePath));
|
||||
this.storeExtractionResult(filePath, fullContent, language, stats, result);
|
||||
await this.storeExtractionResult(filePath, fullContent, language, stats, result, commitYield);
|
||||
|
||||
const idx = errors.indexOf(errEntry);
|
||||
if (idx >= 0) errors.splice(idx, 1);
|
||||
@@ -2053,7 +2085,7 @@ export class ExtractionOrchestrator {
|
||||
|
||||
// Store in database
|
||||
if (result.nodes.length > 0 || result.errors.length === 0) {
|
||||
this.storeExtractionResult(relativePath, content, language, stats, result);
|
||||
await this.storeExtractionResult(relativePath, content, language, stats, result, createYielder());
|
||||
}
|
||||
|
||||
return result;
|
||||
@@ -2062,13 +2094,21 @@ export class ExtractionOrchestrator {
|
||||
/**
|
||||
* Store extraction result in database
|
||||
*/
|
||||
private storeExtractionResult(
|
||||
private async storeExtractionResult(
|
||||
filePath: string,
|
||||
content: string,
|
||||
language: Language,
|
||||
stats: fs.Stats,
|
||||
result: ExtractionResult
|
||||
): void {
|
||||
result: ExtractionResult,
|
||||
onYield?: MaybeYield
|
||||
): Promise<void> {
|
||||
// Bulk inserts run in bounded sub-transactions with a yield between, so a
|
||||
// giant generated file (tens of thousands of symbols) can't block the
|
||||
// event loop — and the #850 watchdog heartbeat — for the whole store.
|
||||
// The file was NEVER one atomic transaction (each insert call has its
|
||||
// own), and the files-table record still lands last, so crash recovery
|
||||
// is unchanged: a partially-stored file has no record and re-indexes.
|
||||
const STORE_CHUNK = 2000;
|
||||
const contentHash = hashContent(content);
|
||||
|
||||
// Check if file already exists and hasn't changed
|
||||
@@ -2107,9 +2147,10 @@ export class ExtractionOrchestrator {
|
||||
// be silently skipped by insertNode() (see issue #42).
|
||||
const validNodes = result.nodes.filter((n) => n.id && n.kind && n.name && n.filePath && n.language);
|
||||
|
||||
// Insert nodes
|
||||
if (validNodes.length > 0) {
|
||||
this.queries.insertNodes(validNodes);
|
||||
// Insert nodes (chunked — see STORE_CHUNK above)
|
||||
for (let i = 0; i < validNodes.length; i += STORE_CHUNK) {
|
||||
this.queries.insertNodes(validNodes.slice(i, i + STORE_CHUNK));
|
||||
await onYield?.();
|
||||
}
|
||||
|
||||
// Filter edges to only reference nodes that were actually inserted
|
||||
@@ -2118,8 +2159,9 @@ export class ExtractionOrchestrator {
|
||||
const validEdges = result.edges.filter(
|
||||
(e) => insertedIds.has(e.source) && insertedIds.has(e.target)
|
||||
);
|
||||
if (validEdges.length > 0) {
|
||||
this.queries.insertEdges(validEdges);
|
||||
for (let i = 0; i < validEdges.length; i += STORE_CHUNK) {
|
||||
this.queries.insertEdges(validEdges.slice(i, i + STORE_CHUNK));
|
||||
await onYield?.();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2159,8 +2201,9 @@ export class ExtractionOrchestrator {
|
||||
filePath: ref.filePath ?? filePath,
|
||||
language: ref.language ?? language,
|
||||
}));
|
||||
if (refsWithContext.length > 0) {
|
||||
this.queries.insertUnresolvedRefsBatch(refsWithContext);
|
||||
for (let i = 0; i < refsWithContext.length; i += STORE_CHUNK) {
|
||||
this.queries.insertUnresolvedRefsBatch(refsWithContext.slice(i, i + STORE_CHUNK));
|
||||
await onYield?.();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2212,11 +2255,15 @@ export class ExtractionOrchestrator {
|
||||
// whether or not the project uses git, and crucially also catches committed
|
||||
// changes from `git pull`/`checkout`/`merge`/`rebase` — which `git status`
|
||||
// cannot see, because the working tree is clean afterward.
|
||||
const tSyncScan = Date.now();
|
||||
const currentFiles = await scanDirectoryAsync(this.rootDir);
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-scan: ${Date.now() - tSyncScan}ms (${currentFiles.length} files)`);
|
||||
filesChecked = currentFiles.length;
|
||||
const currentSet = new Set(currentFiles);
|
||||
|
||||
const tTracked = Date.now();
|
||||
const trackedFiles = this.queries.getAllFiles();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-tracked-load: ${Date.now() - tTracked}ms (${trackedFiles.length} tracked)`);
|
||||
const trackedMap = new Map<string, FileRecord>();
|
||||
for (const f of trackedFiles) {
|
||||
trackedMap.set(f.path, f);
|
||||
|
||||
+16
-3
@@ -485,23 +485,33 @@ export class CodeGraph {
|
||||
// receiver conforms to (protocol-extension / inherited / default-
|
||||
// interface). Needs the implements/extends edges the main pass just
|
||||
// built, so it runs after resolution (#750).
|
||||
const tChained = Date.now();
|
||||
await this.resolver.resolveChainedCallsViaConformance();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[synth-timing] chainedConformance: ${Date.now() - tChained}ms`);
|
||||
// Same lifecycle for `this.<member>` callback registrations whose
|
||||
// member is inherited from a supertype (#808).
|
||||
const tDeferred = Date.now();
|
||||
await this.resolver.resolveDeferredThisMemberRefs();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[synth-timing] deferredThisMember: ${Date.now() - tDeferred}ms`);
|
||||
}
|
||||
|
||||
// Refresh planner stats + checkpoint the WAL after bulk writes.
|
||||
// Cheap and non-blocking; never load-bearing for correctness.
|
||||
// Off-thread (worker connection): on a multi-GB index this is minutes
|
||||
// of IO, and inline it starved the #850 watchdog AFTER a fully
|
||||
// successful index. Never load-bearing for correctness.
|
||||
if (result.success && result.filesIndexed > 0) {
|
||||
this.db.runMaintenance();
|
||||
const tMaint = Date.now();
|
||||
await this.db.runMaintenance();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] maintenance: ${Date.now() - tMaint}ms`);
|
||||
}
|
||||
|
||||
// The orchestrator only sees extraction-phase counts; resolution and
|
||||
// synthesizer edges (often >50% of the graph on JVM repos) come later.
|
||||
// Recompute against the DB so the CLI summary reports the true totals.
|
||||
if (result.success && result.filesIndexed > 0) {
|
||||
const tCount = Date.now();
|
||||
const after = this.queries.getNodeAndEdgeCount();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] count-recompute: ${Date.now() - tCount}ms`);
|
||||
result.nodesCreated = after.nodes - before.nodes;
|
||||
result.edgesCreated = after.edges - before.edges;
|
||||
}
|
||||
@@ -612,7 +622,9 @@ export class CodeGraph {
|
||||
if (filesChanged) {
|
||||
if (result.changedFilePaths) {
|
||||
// Scope resolution to changed files (git fast path — bounded set)
|
||||
const tRefLoad = Date.now();
|
||||
const unresolvedRefs = this.queries.getUnresolvedReferencesByFiles(result.changedFilePaths);
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-ref-load: ${Date.now() - tRefLoad}ms (${unresolvedRefs.length} refs)`);
|
||||
|
||||
options.onProgress?.({
|
||||
phase: 'resolving',
|
||||
@@ -687,8 +699,9 @@ export class CodeGraph {
|
||||
}
|
||||
|
||||
// Refresh planner stats + checkpoint the WAL after bulk writes.
|
||||
// Off-thread — see indexAll's call site.
|
||||
if (filesChanged || result.filesRemoved > 0 || orphanCount > 0) {
|
||||
this.db.runMaintenance();
|
||||
await this.db.runMaintenance();
|
||||
}
|
||||
|
||||
// Heal the segment vocabulary on indexes built before the table
|
||||
|
||||
+20
-1
@@ -803,10 +803,24 @@ function readClientHello(
|
||||
) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
// PAUSE before detaching: removing the last 'data' listener does NOT
|
||||
// stop a flowing stream, so bytes arriving (or unshifted) in the gap
|
||||
// between this handler and the session transport attaching were emitted
|
||||
// to zero listeners and silently DISCARDED — and the listener swap left
|
||||
// the socket's flow state wedged, never delivering to the new listener.
|
||||
// A proxy whose client-hello arrived glued to the initialize hit this
|
||||
// ~1-in-5 under load: the daemon answered nothing for the whole session
|
||||
// (the #662 test flake, and real dead sessions behind it). Paused, the
|
||||
// unshifted tail and any new bytes buffer; SocketTransport.start()
|
||||
// resumes explicitly.
|
||||
try { socket.pause(); } catch { /* stream already gone */ }
|
||||
socket.removeListener('data', onData);
|
||||
socket.removeListener('error', onEnd);
|
||||
socket.removeListener('close', onEnd);
|
||||
clearTimeout(timer);
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) {
|
||||
process.stderr.write(`[mcp-debug] clientHello finish pid=${String(peers.pid)} putBack=${putBack ? putBack.length : 0} flowing=${String(socket.readableFlowing)}\n`);
|
||||
}
|
||||
if (putBack && putBack.length > 0 && !socket.destroyed) {
|
||||
try { socket.unshift(putBack); } catch { /* stream already gone */ }
|
||||
}
|
||||
@@ -836,7 +850,12 @@ function readClientHello(
|
||||
}
|
||||
};
|
||||
const onEnd = () => finish({ pid: null, hostPid: null });
|
||||
const timer = setTimeout(() => finish({ pid: null, hostPid: null }), CLIENT_HELLO_TIMEOUT_MS);
|
||||
// On timeout, hand back whatever partial bytes accumulated — discarding
|
||||
// them would tear the first message the transport parses.
|
||||
const timer = setTimeout(() => {
|
||||
const partial = chunks.length === 0 ? undefined : (chunks.length === 1 ? chunks[0] : Buffer.concat(chunks, total));
|
||||
finish({ pid: null, hostPid: null }, partial);
|
||||
}, CLIENT_HELLO_TIMEOUT_MS);
|
||||
timer.unref?.();
|
||||
socket.on('data', onData);
|
||||
socket.on('error', onEnd);
|
||||
|
||||
+8
-1
@@ -279,10 +279,12 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise<
|
||||
const routeToDaemon = (line: string): void => {
|
||||
if (daemonStatus === 'ready' && daemonSocket) {
|
||||
trackInflight(line);
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] proxy->daemon ${line.slice(0, 80)}\n`);
|
||||
try { daemonSocket.write(line.endsWith('\n') ? line : line + '\n'); } catch { /* close path */ }
|
||||
} else if (daemonStatus === 'failed') {
|
||||
void handleLocally(line);
|
||||
} else {
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] proxy-buffer(${daemonStatus}) ${line.slice(0, 80)}\n`);
|
||||
pending.push(line);
|
||||
}
|
||||
};
|
||||
@@ -364,6 +366,7 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise<
|
||||
if (!line.trim()) continue;
|
||||
let resp: JsonRpc | null = null;
|
||||
try { resp = JSON.parse(line) as JsonRpc; } catch { /* not JSON — relay verbatim */ }
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] daemon->proxy ${line.slice(0, 80)}\n`);
|
||||
if (resp && resp.id !== undefined && ('result' in resp || 'error' in resp)) {
|
||||
inflight.delete(resp.id); // answered — no longer in flight
|
||||
// Suppress the daemon's reply to the initialize we forwarded to prime it
|
||||
@@ -392,7 +395,11 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise<
|
||||
};
|
||||
socket.on('close', onDaemonLost);
|
||||
socket.on('error', onDaemonLost);
|
||||
for (const line of pending) { trackInflight(line); try { socket.write(line + '\n'); } catch { /* ignore */ } }
|
||||
for (const line of pending) {
|
||||
trackInflight(line);
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] proxy-flush ${line.slice(0, 80)}\n`);
|
||||
try { socket.write(line + '\n'); } catch { /* ignore */ }
|
||||
}
|
||||
pending.length = 0;
|
||||
} else if (!shuttingDown) {
|
||||
daemonStatus = 'failed';
|
||||
|
||||
@@ -183,6 +183,23 @@ export class QueryPool {
|
||||
return !this.destroyed && this.totalCrashes < CRASH_BUDGET;
|
||||
}
|
||||
|
||||
/**
|
||||
* True once at least one worker has completed its cold start (posted the
|
||||
* 'ready' handshake). Until then the ToolHandler serves calls IN-PROCESS:
|
||||
* a worker cold start is a full module load + DB open — seconds normally,
|
||||
* tens of seconds on a loaded machine — and a call queued behind it gets
|
||||
* nothing until the 45s busy backstop. The daemon's very first tool call
|
||||
* hitting that window was the recurring #662 test flake (and a real
|
||||
* first-call stall for agents). The pool exists for CONCURRENT load, which
|
||||
* by definition arrives after warm-up; the pre-pool in-process path is
|
||||
* strictly better while nothing is warm. Stays true for the pool's
|
||||
* lifetime — later crash-respawn gaps are covered by retry + backstop.
|
||||
*/
|
||||
get ready(): boolean {
|
||||
return this.everReady && !this.destroyed;
|
||||
}
|
||||
private everReady = false;
|
||||
|
||||
private spawnOne(): void {
|
||||
if (this.destroyed || this.workers.size >= this.maxSize) return;
|
||||
let w: PoolWorker;
|
||||
@@ -204,6 +221,7 @@ export class QueryPool {
|
||||
if (m.type === 'ready') {
|
||||
this.pendingWorkers.delete(w);
|
||||
if (m.ok === false) this.totalCrashes++; // hard open failure
|
||||
else this.everReady = true;
|
||||
this.idle.push(w);
|
||||
this.drain();
|
||||
return;
|
||||
|
||||
@@ -260,9 +260,12 @@ export class MCPSession {
|
||||
return;
|
||||
}
|
||||
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] toolsCall ${toolName} id=${String(request.id)} pre-init\n`);
|
||||
await this.retryInitIfNeeded();
|
||||
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] toolsCall ${toolName} id=${String(request.id)} dispatch\n`);
|
||||
const result = await this.engine.getToolHandler().execute(toolName, toolArgs);
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] toolsCall ${toolName} id=${String(request.id)} done\n`);
|
||||
this.transport.sendResult(request.id, result);
|
||||
// After the reply is on the wire — telemetry must never delay a tool
|
||||
// response (in-memory increment only; see src/telemetry).
|
||||
|
||||
+14
-9
@@ -1399,15 +1399,20 @@ export class ToolHandler {
|
||||
}
|
||||
|
||||
// Read tools: off-load the CPU-heavy dispatch to the worker pool when one
|
||||
// is attached and healthy (daemon mode), so the daemon's single event loop
|
||||
// stays free for the MCP transport under concurrent load — otherwise N
|
||||
// concurrent explores serialize AND starve the transport until the whole
|
||||
// batch drains (clients then time out). With no pool (direct mode) or a
|
||||
// degraded one, dispatch runs in-process exactly as before. Either way the
|
||||
// result flows through the cross-cutting notices — worktree-index mismatch
|
||||
// (#155) and per-file staleness (#403) — which need the watched MAIN
|
||||
// instance and so are always applied here, never in the worker.
|
||||
const result = (this.queryPool && this.queryPool.healthy)
|
||||
// is attached, healthy, AND has finished its first cold start (daemon
|
||||
// mode), so the daemon's single event loop stays free for the MCP
|
||||
// transport under concurrent load — otherwise N concurrent explores
|
||||
// serialize AND starve the transport until the whole batch drains
|
||||
// (clients then time out). Before the first worker is warm, calls run
|
||||
// in-process: a call queued behind a cold start sat invisible until the
|
||||
// 45s busy backstop — the daemon's first tool call stalling for however
|
||||
// long a worker spawn takes on a loaded machine (the #662 flake). With
|
||||
// no pool (direct mode) or a degraded one, dispatch runs in-process
|
||||
// exactly as before. Either way the result flows through the
|
||||
// cross-cutting notices — worktree-index mismatch (#155) and per-file
|
||||
// staleness (#403) — which need the watched MAIN instance and so are
|
||||
// always applied here, never in the worker.
|
||||
const result = (this.queryPool && this.queryPool.healthy && this.queryPool.ready)
|
||||
? await this.queryPool.run(toolName, args)
|
||||
: await this.executeReadTool(toolName, args);
|
||||
const withWorktree = this.withWorktreeNotice(result, args.projectPath as string | undefined);
|
||||
|
||||
@@ -193,7 +193,15 @@ abstract class LineBasedJsonRpcTransport implements JsonRpcTransport {
|
||||
|
||||
if (this.messageHandler) {
|
||||
try {
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) {
|
||||
const m = parsed as { method?: string; id?: unknown };
|
||||
process.stderr.write(`[mcp-debug] recv method=${m.method} id=${String(m.id)}\n`);
|
||||
}
|
||||
await this.messageHandler(parsed as JsonRpcRequest | JsonRpcNotification);
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) {
|
||||
const m = parsed as { method?: string; id?: unknown };
|
||||
process.stderr.write(`[mcp-debug] handled method=${m.method} id=${String(m.id)}\n`);
|
||||
}
|
||||
} catch (err) {
|
||||
const message = parsed as JsonRpcRequest;
|
||||
if ('id' in message) {
|
||||
@@ -353,7 +361,11 @@ export class SocketTransport extends LineBasedJsonRpcTransport {
|
||||
this.messageHandler = handler;
|
||||
|
||||
this.socket.setEncoding('utf8');
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) {
|
||||
process.stderr.write(`[mcp-debug] transport attached flowing=${String(this.socket.readableFlowing)} buffered=${this.socket.readableLength}\n`);
|
||||
}
|
||||
this.socket.on('data', (chunk: string) => {
|
||||
if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] transport data ${chunk.length}b\n`);
|
||||
this.buffer += chunk;
|
||||
let idx;
|
||||
// Drain every complete line; tail-fragment stays in the buffer for the
|
||||
@@ -374,6 +386,11 @@ export class SocketTransport extends LineBasedJsonRpcTransport {
|
||||
process.stderr.write(`[CodeGraph daemon] socket error: ${err.message}\n`);
|
||||
this.handleSocketClose();
|
||||
});
|
||||
// The daemon's hello reader hands the socket over PAUSED (so the unshifted
|
||||
// tail can't be emitted to zero listeners and lost — the #662 wedge).
|
||||
// Attaching 'data' does not resume an explicitly-paused stream; do it here.
|
||||
// Harmless when the socket was never paused.
|
||||
this.socket.resume();
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
|
||||
@@ -52,6 +52,8 @@ import * as path from 'node:path';
|
||||
import type { Edge, Node } from '../types';
|
||||
import type { QueryBuilder } from '../db/queries';
|
||||
import type { ResolutionContext } from './types';
|
||||
import type { MaybeYield } from './cooperative-yield';
|
||||
import { LRUCache } from './lru-cache';
|
||||
import { stripCommentsForRegex } from './strip-comments';
|
||||
|
||||
const C_CPP_EXT = /\.(c|h|cc|cpp|cxx|hpp|hh|hxx|cppm|ipp|inl|tcc)$/i;
|
||||
@@ -306,22 +308,31 @@ const INCLUDE_RE = /#[ \t]*include[ \t]+"([^"\n]+)"/g;
|
||||
/** Included files worth scanning for registration tables (e.g. a generated `.def`). */
|
||||
const INCLUDABLE_EXT = /\.(def|inc|h|hh|hpp|hxx|c|cc|cpp|cxx|ipp|tcc|tbl)$/i;
|
||||
|
||||
export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
|
||||
export async function cFnPointerDispatchEdges(_queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
const files = ctx.getAllFiles().filter((f) => C_CPP_EXT.test(f));
|
||||
if (files.length === 0) return [];
|
||||
|
||||
// Cache raw + stripped source per file (read once, reused across passes).
|
||||
// Raw is needed for `#include "…"` directives — strip blanks string contents.
|
||||
const rawCache = new Map<string, string | null>();
|
||||
// Cache raw + stripped source per file, LRU-BOUNDED. The old unbounded Maps
|
||||
// retained every C/C++ file's raw AND stripped text for the whole pass —
|
||||
// multiple GB on the Linux kernel, one of the two OOM culprits in #1212.
|
||||
// Every sweep below iterates in `files` order, and node-kind scans return
|
||||
// rows in file-commit order, so access is near-sequential and a small LRU
|
||||
// hits; a miss just re-reads + re-strips.
|
||||
const rawCache = new LRUCache<string, string | null>(128);
|
||||
const raw = (file: string): string | null => {
|
||||
if (rawCache.has(file)) return rawCache.get(file)!;
|
||||
const r = ctx.readFile(file);
|
||||
rawCache.set(file, r);
|
||||
return r;
|
||||
};
|
||||
const srcCache = new Map<string, string>();
|
||||
const srcCache = new LRUCache<string, string>(128);
|
||||
const src = (file: string): string | null => {
|
||||
if (srcCache.has(file)) return srcCache.get(file)!;
|
||||
// A cached '' (empty or unreadable file) returns '' where the miss path
|
||||
// returns null for unreadable — every caller falsy-checks, so the two are
|
||||
// interchangeable.
|
||||
const hit = srcCache.get(file);
|
||||
if (hit !== undefined) return hit;
|
||||
const r = raw(file);
|
||||
const s = r == null ? '' : stripCommentsForRegex(r, 'c');
|
||||
srcCache.set(file, s);
|
||||
@@ -347,6 +358,7 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
const fnPtrTypedefs = new Set<string>();
|
||||
const fnTypeTypedefs = new Set<string>();
|
||||
for (const file of files) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
const s = src(file);
|
||||
if (!s || !s.includes('typedef')) continue;
|
||||
FNPTR_TYPEDEF_RE.lastIndex = 0;
|
||||
@@ -426,9 +438,10 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
if (fields.some((f) => f.isFnPtr)) structLayout.set(name, fields);
|
||||
};
|
||||
|
||||
for (const st of ctx.getNodesByKind('struct')) {
|
||||
for (const st of (ctx.iterateNodesByKind?.('struct') ?? ctx.getNodesByKind('struct'))) {
|
||||
if ((++scannedFiles & 255) === 0) await onYield();
|
||||
if (!C_CPP_EXT.test(st.filePath)) continue;
|
||||
const s = srcCache.get(st.filePath) ?? src(st.filePath);
|
||||
const s = src(st.filePath);
|
||||
if (!s) continue;
|
||||
const body = sliceLines(s, st.startLine, st.endLine);
|
||||
const open = body.indexOf('{');
|
||||
@@ -444,11 +457,9 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
const fnPtrFieldOf = (struct: string, field: string): boolean =>
|
||||
!!structLayout.get(struct)?.some((f) => f.name === field && f.isFnPtr);
|
||||
|
||||
// C/C++ function + method nodes, materialized once (bounded by C/C++ files).
|
||||
const cFns: Node[] = [];
|
||||
for (const fn of iterateFns(queries)) {
|
||||
if (C_CPP_EXT.test(fn.filePath)) cFns.push(fn);
|
||||
}
|
||||
// C/C++ function + method nodes are STREAMED per sweep (see passes D/E) —
|
||||
// the old materialized `cFns` array held every function node on the repo
|
||||
// (O(nodes) memory, part of the #1212 kernel OOM).
|
||||
|
||||
// ---- function-name → node resolution (prefer a function in the same file) ----
|
||||
const resolveFn = (name: string, preferFile?: string): Node | null => {
|
||||
@@ -463,13 +474,13 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
};
|
||||
|
||||
// ---- Pass C: registrations — Map<"struct.field", Set<funcNodeId>> ----
|
||||
// Ids only — retaining the full Node per registration (the old `idToNode`)
|
||||
// was write-only dead weight at O(registrations) memory.
|
||||
const reg = new Map<string, Set<string>>();
|
||||
const idToNode = new Map<string, Node>();
|
||||
const addReg = (struct: string, field: string, fn: Node): void => {
|
||||
const key = `${struct}.${field}`;
|
||||
if (!reg.has(key)) reg.set(key, new Set());
|
||||
reg.get(key)!.add(fn.id);
|
||||
idToNode.set(fn.id, fn);
|
||||
};
|
||||
|
||||
// Bare arrays-of-fn-pointers (no struct): array VARIABLE name → per-file sets
|
||||
@@ -483,7 +494,6 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
let e = entries.find((x) => x.file === file);
|
||||
if (!e) { e = { file, ids: new Set() }; entries.push(e); }
|
||||
e.ids.add(fn.id);
|
||||
idToNode.set(fn.id, fn);
|
||||
};
|
||||
|
||||
// A struct value `{ … }` (one element) — register its function entries to the
|
||||
@@ -560,25 +570,26 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
};
|
||||
|
||||
// Per-file macro + include parsing (any file, indexed or not), cached.
|
||||
const fnMacroCache = new Map<string, Map<string, MacroDef>>();
|
||||
// Derived per-file caches, LRU-bounded like the content caches (#1212).
|
||||
const fnMacroCache = new LRUCache<string, Map<string, MacroDef>>(256);
|
||||
const fileFnMacros = (file: string): Map<string, MacroDef> => {
|
||||
let m = fnMacroCache.get(file);
|
||||
if (!m) { m = parseFunctionMacros(src(file) ?? ''); fnMacroCache.set(file, m); }
|
||||
return m;
|
||||
};
|
||||
const objMacroCache = new Map<string, Map<string, string>>();
|
||||
const objMacroCache = new LRUCache<string, Map<string, string>>(256);
|
||||
const fileObjMacros = (file: string): Map<string, string> => {
|
||||
let m = objMacroCache.get(file);
|
||||
if (!m) { m = parseObjectMacros(src(file) ?? ''); objMacroCache.set(file, m); }
|
||||
return m;
|
||||
};
|
||||
const definedCache = new Map<string, Set<string>>();
|
||||
const definedCache = new LRUCache<string, Set<string>>(256);
|
||||
const fileDefinedNames = (file: string): Set<string> => {
|
||||
let d = definedCache.get(file);
|
||||
if (!d) { d = parseDefinedNames(src(file) ?? ''); definedCache.set(file, d); }
|
||||
return d;
|
||||
};
|
||||
const includeCache = new Map<string, string[]>();
|
||||
const includeCache = new LRUCache<string, string[]>(1024);
|
||||
const localIncludesOf = (file: string): string[] => {
|
||||
let out = includeCache.get(file);
|
||||
if (out) return out;
|
||||
@@ -635,39 +646,7 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
objEnv: Map<string, string>;
|
||||
}
|
||||
const indexedSet = new Set(files);
|
||||
const units: Unit[] = [];
|
||||
const seenInclude = new Set<string>();
|
||||
for (const file of files) {
|
||||
const env = new Map<string, MacroDef>();
|
||||
const objEnv = new Map<string, string>();
|
||||
const defined = new Set<string>();
|
||||
buildEnv(file, 2, new Set(), env, objEnv, defined);
|
||||
const s = src(file);
|
||||
if (s) units.push({ text: s, file, env, objEnv });
|
||||
for (const target of localIncludesOf(file)) {
|
||||
if (seenInclude.has(`${file}>${target}`)) continue;
|
||||
const incSrc = src(target);
|
||||
if (!incSrc) continue;
|
||||
if (indexedSet.has(target)) {
|
||||
// Re-scan an indexed header only when this includer unlocks guarded code.
|
||||
const ownDef = fileDefinedNames(target);
|
||||
const adds = [...defined].some((n) => !ownDef.has(n));
|
||||
if (!adds || !/#\s*if/.test(incSrc)) continue;
|
||||
}
|
||||
seenInclude.add(`${file}>${target}`);
|
||||
// The include is pasted into the includer — evaluate its conditionals in
|
||||
// the includer's defined set (a no-op when it has none). Re-parse the
|
||||
// included file's OWN macros from that resolved text so a macro it defines
|
||||
// conditionally (vim's `EXCMD`, whose plain last-wins parse picks the enum
|
||||
// arm) overrides with the ARM THAT IS ACTUALLY ACTIVE here.
|
||||
const text = evalConditionals(incSrc, defined);
|
||||
const incEnv = new Map(env);
|
||||
for (const [k, v] of parseFunctionMacros(text)) incEnv.set(k, v);
|
||||
const incObjEnv = new Map(objEnv);
|
||||
for (const [k, v] of parseObjectMacros(text)) incObjEnv.set(k, v);
|
||||
units.push({ text, file: target, env: incEnv, objEnv: incObjEnv });
|
||||
}
|
||||
}
|
||||
|
||||
// Global variable → struct type, for resolving a dispatch through a file-scope
|
||||
// table by subscript (`cmdnames[i].cmd_func(…)`).
|
||||
@@ -716,9 +695,12 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
// (below) is what separates this from a plain data/struct array.
|
||||
const ARRAY_TABLE_RE =
|
||||
/(?:^|[;{}])\s*(?:(?:static|const|extern|register|volatile)\s+)*(\w+)\s+(\*\s*)?(\w+)\s*\[[^\]]*\]\s*=\s*\{/g;
|
||||
for (const unit of units) {
|
||||
// Process ONE unit's text and discard it. The old shape built every unit up
|
||||
// front (`const units: Unit[]`) — the full text of every C file plus its
|
||||
// expanded includes held simultaneously, gigabytes on the kernel (#1212).
|
||||
const processUnit = (unit: Unit): void => {
|
||||
const s = unit.text;
|
||||
if (!s || !s.includes('{')) continue;
|
||||
if (!s || !s.includes('{')) return;
|
||||
|
||||
INLINE_STRUCT_RE.lastIndex = 0;
|
||||
let im: RegExpExecArray | null;
|
||||
@@ -745,7 +727,7 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
}
|
||||
}
|
||||
|
||||
if (!s.includes('=')) continue;
|
||||
if (!s.includes('=')) return;
|
||||
INIT_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = INIT_RE.exec(s))) {
|
||||
@@ -777,6 +759,41 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
registerArrayValue(am[3]!, s.slice(open + 1, close), unit.file, unit.env);
|
||||
ARRAY_TABLE_RE.lastIndex = close;
|
||||
}
|
||||
};
|
||||
|
||||
// ---- Pass C: registrations — stream each file (and its qualifying local
|
||||
// includes) through processUnit, one at a time.
|
||||
for (const file of files) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
const env = new Map<string, MacroDef>();
|
||||
const objEnv = new Map<string, string>();
|
||||
const defined = new Set<string>();
|
||||
buildEnv(file, 2, new Set(), env, objEnv, defined);
|
||||
const s = src(file);
|
||||
if (s) processUnit({ text: s, file, env, objEnv });
|
||||
for (const target of localIncludesOf(file)) {
|
||||
if (seenInclude.has(`${file}>${target}`)) continue;
|
||||
const incSrc = src(target);
|
||||
if (!incSrc) continue;
|
||||
if (indexedSet.has(target)) {
|
||||
// Re-scan an indexed header only when this includer unlocks guarded code.
|
||||
const ownDef = fileDefinedNames(target);
|
||||
const adds = [...defined].some((n) => !ownDef.has(n));
|
||||
if (!adds || !/#\s*if/.test(incSrc)) continue;
|
||||
}
|
||||
seenInclude.add(`${file}>${target}`);
|
||||
// The include is pasted into the includer — evaluate its conditionals in
|
||||
// the includer's defined set (a no-op when it has none). Re-parse the
|
||||
// included file's OWN macros from that resolved text so a macro it defines
|
||||
// conditionally (vim's `EXCMD`, whose plain last-wins parse picks the enum
|
||||
// arm) overrides with the ARM THAT IS ACTUALLY ACTIVE here.
|
||||
const text = evalConditionals(incSrc, defined);
|
||||
const incEnv = new Map(env);
|
||||
for (const [k, v] of parseFunctionMacros(text)) incEnv.set(k, v);
|
||||
const incObjEnv = new Map(objEnv);
|
||||
for (const [k, v] of parseObjectMacros(text)) incObjEnv.set(k, v);
|
||||
processUnit({ text, file: target, env: incEnv, objEnv: incObjEnv });
|
||||
}
|
||||
}
|
||||
|
||||
// ---- receiver-type resolution within a function's source ----
|
||||
@@ -828,19 +845,23 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
// a fixpoint so a hook slot inherits a registry field's handlers.
|
||||
const FIELD_ASSIGN_RE = /(\w+)\s*(?:->|\.)\s*(\w+)\s*=\s*(\w+)\s*(?:->|\.)\s*(\w+)/g;
|
||||
const propagations: { to: string; from: string }[] = [];
|
||||
for (const fn of cFns) {
|
||||
const s = srcCache.get(fn.filePath);
|
||||
if (!s) continue;
|
||||
const body = sliceLines(s, fn.startLine, fn.endLine);
|
||||
if (!body.includes('=')) continue;
|
||||
FIELD_ASSIGN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = FIELD_ASSIGN_RE.exec(body))) {
|
||||
const [, lrecv, lfield, rrecv, rfield] = m;
|
||||
const lt = recvTypeIn(body, lrecv!);
|
||||
const rt = recvTypeIn(body, rrecv!);
|
||||
if (lt && rt && fnPtrFieldOf(lt, lfield!) && fnPtrFieldOf(rt, rfield!)) {
|
||||
propagations.push({ to: `${lt}.${lfield}`, from: `${rt}.${rfield}` });
|
||||
for (const file of files) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
const s = src(file);
|
||||
if (!s || !s.includes('=')) continue;
|
||||
for (const fn of ctx.getNodesInFile(file)) {
|
||||
if (!FN_KINDS.has(fn.kind)) continue;
|
||||
const body = sliceLines(s, fn.startLine, fn.endLine);
|
||||
if (!body.includes('=')) continue;
|
||||
FIELD_ASSIGN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = FIELD_ASSIGN_RE.exec(body))) {
|
||||
const [, lrecv, lfield, rrecv, rfield] = m;
|
||||
const lt = recvTypeIn(body, lrecv!);
|
||||
const rt = recvTypeIn(body, rrecv!);
|
||||
if (lt && rt && fnPtrFieldOf(lt, lfield!) && fnPtrFieldOf(rt, rfield!)) {
|
||||
propagations.push({ to: `${lt}.${lfield}`, from: `${rt}.${rfield}` });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -875,9 +896,12 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
const ARRAY_DISPATCH_RE = /(?:\(\s*\*\s*)?\b(\w+)\s*\[[^\][]*\]\s*\)?\s*\(/g;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const fn of cFns) {
|
||||
const s = srcCache.get(fn.filePath);
|
||||
for (const file of files) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
const s = src(file);
|
||||
if (!s) continue;
|
||||
for (const fn of ctx.getNodesInFile(file)) {
|
||||
if (!FN_KINDS.has(fn.kind)) continue;
|
||||
const body = sliceLines(s, fn.startLine, fn.endLine);
|
||||
DISPATCH_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
@@ -956,12 +980,7 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return edges;
|
||||
}
|
||||
|
||||
/** C/C++ function + method nodes, streamed (memory-safe on symbol-dense repos). */
|
||||
function* iterateFns(queries: QueryBuilder): IterableIterator<Node> {
|
||||
yield* queries.iterateNodesByKind('function');
|
||||
yield* queries.iterateNodesByKind('method');
|
||||
}
|
||||
|
||||
@@ -280,11 +280,13 @@ async function closureCollectionEdges(queries: QueryBuilder, ctx: ResolutionCont
|
||||
|
||||
/** Phase 2: string-keyed EventEmitter channels (on('e', fn) ↔ emit('e')). */
|
||||
async function eventEmitterEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
const emitsByEvent = new Map<string, Set<string>>(); // event → dispatcher node ids
|
||||
const handlersByEvent = new Map<string, Map<string, string>>(); // event → handler id → registration site (file:line)
|
||||
|
||||
let scanned = 0;
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if ((++scanned & 255) === 0) await onYield(); // #1091: yield mid-scan on huge graphs
|
||||
const content = ctx.readFile(file);
|
||||
if (!content) continue;
|
||||
@@ -348,10 +350,12 @@ async function eventEmitterEdges(ctx: ResolutionContext, onYield: MaybeYield): P
|
||||
* `this.setState`). Over-approximation (all setState methods reach render) is
|
||||
* accepted — it's reachability-correct, like the callback channels.
|
||||
*/
|
||||
function reactRenderEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
|
||||
async function reactRenderEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const cls of queries.getNodesByKind('class')) {
|
||||
for (const cls of queries.iterateNodesByKind('class')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
const children = queries.getOutgoingEdges(cls.id, ['contains'])
|
||||
.map((e) => queries.getNodeById(e.target))
|
||||
.filter((n): n is Node => !!n && n.kind === 'method');
|
||||
@@ -387,10 +391,12 @@ function reactRenderEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[]
|
||||
* body calls `setState(` → `build`. The setState gate + `.dart` file keep this to
|
||||
* Flutter State classes. Over-approximation accepted (reachability-correct).
|
||||
*/
|
||||
function flutterBuildEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
|
||||
async function flutterBuildEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const cls of queries.getNodesByKind('class')) {
|
||||
for (const cls of queries.iterateNodesByKind('class')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
const children = queries.getOutgoingEdges(cls.id, ['contains'])
|
||||
.map((e) => queries.getNodeById(e.target))
|
||||
.filter((n): n is Node => !!n && n.kind === 'method');
|
||||
@@ -447,10 +453,12 @@ const ARKUI_ARRAY_MUTATORS = 'push|pop|shift|unshift|splice|sort|reverse|fill';
|
||||
* with no reactive properties, gets nothing (this is the precision line the
|
||||
* all-sibling-methods design would erase).
|
||||
*/
|
||||
function arkuiStateBuildEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
|
||||
async function arkuiStateBuildEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const struct of queries.getNodesByKind('struct')) {
|
||||
for (const struct of queries.iterateNodesByKind('struct')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (struct.language !== 'arkts') continue;
|
||||
const children = queries.getOutgoingEdges(struct.id, ['contains'])
|
||||
.map((e) => queries.getNodeById(e.target))
|
||||
@@ -515,7 +523,8 @@ const ARKUI_EMITTER_FANOUT_CAP = 8;
|
||||
* handling — their bodies' calls already attribute to the registering method,
|
||||
* so targeting that method keeps the chain connected.
|
||||
*/
|
||||
function arkuiEmitterEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function arkuiEmitterEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
interface Site { nodeId: string; file: string; line: number }
|
||||
// bucket key -> emit sites / handler sites
|
||||
const emits = new Map<string, Site[]>();
|
||||
@@ -533,6 +542,7 @@ function arkuiEmitterEdges(ctx: ResolutionContext): Edge[] {
|
||||
};
|
||||
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!file.endsWith('.ets')) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || !content.includes('emitter.')) continue;
|
||||
@@ -619,7 +629,7 @@ const ARKUI_ROUTER_RE = /\brouter\s*\.\s*(?:pushUrl|replaceUrl)\s*\(\s*\{[^)]{0,
|
||||
* anything still ambiguous is dropped rather than guessed. Only `@Entry`
|
||||
* structs qualify as targets — the decorator is what makes a file a page.
|
||||
*/
|
||||
function arkuiRouterEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function arkuiRouterEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
@@ -635,7 +645,9 @@ function arkuiRouterEdges(ctx: ResolutionContext): Edge[] {
|
||||
return '';
|
||||
};
|
||||
|
||||
let scannedFiles = 0;
|
||||
for (const file of allFiles) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!file.endsWith('.ets')) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || !content.includes('router.')) continue;
|
||||
@@ -690,7 +702,8 @@ function arkuiRouterEdges(ctx: ResolutionContext): Edge[] {
|
||||
* implementation(s). Over-approximation accepted (reachability-correct); capped
|
||||
* per class and gated to C++ to avoid touching other languages' dispatch.
|
||||
*/
|
||||
function cppOverrideEdges(queries: QueryBuilder): Edge[] {
|
||||
async function cppOverrideEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
const methodsOf = (classId: string): Node[] =>
|
||||
@@ -698,7 +711,8 @@ function cppOverrideEdges(queries: QueryBuilder): Edge[] {
|
||||
.getOutgoingEdges(classId, ['contains'])
|
||||
.map((e) => queries.getNodeById(e.target))
|
||||
.filter((n): n is Node => !!n && n.kind === 'method');
|
||||
for (const cls of queries.getNodesByKind('class')) {
|
||||
for (const cls of queries.iterateNodesByKind('class')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
const subMethods = methodsOf(cls.id).filter((n) => n.language === 'cpp');
|
||||
if (subMethods.length === 0) continue;
|
||||
for (const ext of queries.getOutgoingEdges(cls.id, ['extends'])) {
|
||||
@@ -762,7 +776,8 @@ const IFACE_OVERRIDE_LANGS = new Set([
|
||||
* with the other dispatch synthesizers; capped per interface. Empty interfaces
|
||||
* (`any`) are skipped so they don't match every struct.
|
||||
*/
|
||||
function goImplementsEdges(queries: QueryBuilder): Edge[] {
|
||||
async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
@@ -775,11 +790,21 @@ function goImplementsEdges(queries: QueryBuilder): Edge[] {
|
||||
.map((n) => n.name),
|
||||
);
|
||||
|
||||
const goStructs = queries.getNodesByKind('struct').filter((s) => s.language === 'go');
|
||||
// Materializes GO structs only (the pass is language-gated by the caller),
|
||||
// never the whole struct kind — that array is O(nodes) on struct-heavy
|
||||
// repos like the Linux kernel (#1212).
|
||||
const goStructs: Node[] = [];
|
||||
for (const s of queries.iterateNodesByKind('struct')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (s.language === 'go') goStructs.push(s);
|
||||
}
|
||||
const structMethods = new Map<string, Set<string>>();
|
||||
for (const s of goStructs) structMethods.set(s.id, methodNameSet(s.id));
|
||||
|
||||
for (const iface of queries.getNodesByKind('interface')) {
|
||||
for (const iface of queries.iterateNodesByKind('interface')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (iface.language !== 'go') continue;
|
||||
const want = methodNameSet(iface.id);
|
||||
if (want.size === 0) continue; // empty interface (`any`) — would match everything
|
||||
@@ -831,7 +856,8 @@ function goImplementsEdges(queries: QueryBuilder): Edge[] {
|
||||
* matching the same-file edges extraction already emits). Skips methods that
|
||||
* already have a type parent (the same-file case). (#583, cross-file half)
|
||||
*/
|
||||
function goCrossFileMethodContainsEdges(queries: QueryBuilder): Edge[] {
|
||||
async function goCrossFileMethodContainsEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
const TYPE_KINDS = new Set<NodeKind>(['struct', 'class', 'interface', 'enum', 'type_alias']);
|
||||
@@ -840,7 +866,10 @@ function goCrossFileMethodContainsEdges(queries: QueryBuilder): Edge[] {
|
||||
return i >= 0 ? p.slice(0, i) : '';
|
||||
};
|
||||
|
||||
for (const method of queries.getNodesByKind('method')) {
|
||||
for (const method of queries.iterateNodesByKind('method')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (method.language !== 'go') continue;
|
||||
// The receiver type is encoded in the method's qualifiedName as `Recv::name`
|
||||
// (extraction sets `${receiverType}::${name}` for receiver methods).
|
||||
@@ -908,13 +937,18 @@ function kmpKindsCompatible(a: string, b: string): boolean {
|
||||
return a === b || (KMP_TYPE_KINDS.has(a) && KMP_TYPE_KINDS.has(b));
|
||||
}
|
||||
|
||||
function kotlinExpectActualEdges(queries: QueryBuilder): Edge[] {
|
||||
async function kotlinExpectActualEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
const actuals = queries
|
||||
.getAllNodes()
|
||||
.filter((n) => n.language === 'kotlin' && !!n.decorators?.includes('actual'));
|
||||
for (const act of actuals) {
|
||||
// SQL-side language+decorator pre-filter, streamed. The old
|
||||
// `getAllNodes().filter(...)` hydrated the ENTIRE node table into one array
|
||||
// just to find kotlin `actual` declarations — on a 2M-node graph that alone
|
||||
// exceeded Node's default heap and killed the index (#1212). The LIKE
|
||||
// pre-filter can over-match (substring), so the exact decorator check stays.
|
||||
for (const act of queries.iterateNodesByLanguageWithDecorator('kotlin', 'actual')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (!act.decorators?.includes('actual')) continue;
|
||||
let added = 0;
|
||||
for (const cand of queries.getNodesByQualifiedNameExact(act.qualifiedName)) {
|
||||
if (added >= MAX_CALLBACKS_PER_CHANNEL) break;
|
||||
@@ -944,7 +978,8 @@ function kotlinExpectActualEdges(queries: QueryBuilder): Edge[] {
|
||||
return edges;
|
||||
}
|
||||
|
||||
function interfaceOverrideEdges(queries: QueryBuilder): Edge[] {
|
||||
async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
const methodsOf = (classId: string): Node[] =>
|
||||
@@ -957,7 +992,8 @@ function interfaceOverrideEdges(queries: QueryBuilder): Edge[] {
|
||||
// types that conform to protocols. Iterate both.
|
||||
const concreteKinds = ['class', 'struct'] as const;
|
||||
for (const kind of concreteKinds) {
|
||||
for (const cls of queries.getNodesByKind(kind)) {
|
||||
for (const cls of queries.iterateNodesByKind(kind)) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
const implMethods = methodsOf(cls.id).filter((n) => IFACE_OVERRIDE_LANGS.has(n.language));
|
||||
if (implMethods.length === 0) continue;
|
||||
for (const sup of queries.getOutgoingEdges(cls.id, ['implements', 'extends'])) {
|
||||
@@ -1023,7 +1059,8 @@ function interfaceOverrideEdges(queries: QueryBuilder): Edge[] {
|
||||
* Provenance: `heuristic`, `synthesizedBy: 'go-grpc-stub-impl'`. The
|
||||
* stub's source line is the wiring site shown in the trace trail.
|
||||
*/
|
||||
function goGrpcStubImplEdges(queries: QueryBuilder): Edge[] {
|
||||
async function goGrpcStubImplEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
@@ -1037,7 +1074,8 @@ function goGrpcStubImplEdges(queries: QueryBuilder): Edge[] {
|
||||
const methodNamesByStruct = new Map<string, Set<string>>();
|
||||
const methodNodesByStruct = new Map<string, Node[]>();
|
||||
const goStructs: Node[] = [];
|
||||
for (const s of queries.getNodesByKind('struct')) {
|
||||
for (const s of queries.iterateNodesByKind('struct')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (s.language !== 'go') continue;
|
||||
goStructs.push(s);
|
||||
const ms = queries
|
||||
@@ -1117,11 +1155,13 @@ function goGrpcStubImplEdges(queries: QueryBuilder): Edge[] {
|
||||
* (or nothing) and are dropped.
|
||||
*/
|
||||
async function reactJsxChildEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
const PARENT_KINDS = new Set(['method', 'function', 'component']);
|
||||
let scanned = 0;
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if ((++scanned & 255) === 0) await onYield(); // #1091: yield mid-scan on huge graphs
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || (!content.includes('</') && !content.includes('/>'))) continue; // JSX-file gate
|
||||
@@ -1167,7 +1207,9 @@ async function reactJsxChildEdges(ctx: ResolutionContext, onYield: MaybeYield):
|
||||
* component, handler→function/method) keeps precision; inline arrows / `$emit`
|
||||
* skipped.
|
||||
*/
|
||||
function vueTemplateEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function vueTemplateEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
const COMPONENT_KINDS = new Set(['component', 'function', 'class']);
|
||||
@@ -1181,11 +1223,13 @@ function vueTemplateEdges(ctx: ResolutionContext): Edge[] {
|
||||
// misses it (flat components match by basename and don't need this). Map each
|
||||
// nested component's Nuxt name → node so those template usages resolve.
|
||||
const nuxtComponents = new Map<string, Node>();
|
||||
for (const c of ctx.getNodesByKind('component')) {
|
||||
for (const c of (ctx.iterateNodesByKind?.('component') ?? ctx.getNodesByKind('component'))) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
const nn = nuxtComponentName(c.filePath);
|
||||
if (nn && !nuxtComponents.has(nn)) nuxtComponents.set(nn, c);
|
||||
}
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!file.endsWith('.vue')) continue;
|
||||
const content = ctx.readFile(file);
|
||||
const tpl = content && content.match(/<template[^>]*>([\s\S]*)<\/template>/i)?.[1];
|
||||
@@ -1310,7 +1354,8 @@ const RN_JVM_EMIT_RE = /\.emit\s*\(\s*"([^"]+)"\s*,/g;
|
||||
// is followed by `… ) {`) never matches. Multi-line tolerant. (java/kotlin/swift)
|
||||
const RN_NATIVE_SENDEVENT_RE = /\bsendEvent\s*\([^;{}]*?"([^"]+)"/g;
|
||||
|
||||
function rnEventEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
// Native dispatchers (source = the native method whose body sends the
|
||||
// event) and JS handlers (target = the function/method registered as
|
||||
// the listener) keyed by event name.
|
||||
@@ -1318,6 +1363,7 @@ function rnEventEdges(ctx: ResolutionContext): Edge[] {
|
||||
const jsHandlersByEvent = new Map<string, Map<string, string>>();
|
||||
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
const content = ctx.readFile(file);
|
||||
if (!content) continue;
|
||||
|
||||
@@ -1500,11 +1546,13 @@ const FABRIC_NATIVE_SUFFIXES = ['', 'View', 'ViewManager', 'ComponentView', 'Man
|
||||
* caller. The Expo method nodes are id-prefixed `expo-module:` and qualified
|
||||
* `<file>::<module>.<method>` by the framework extractor.
|
||||
*/
|
||||
function expoCrossPlatformEdges(queries: QueryBuilder): Edge[] {
|
||||
async function expoCrossPlatformEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
const byKey = new Map<string, Node[]>();
|
||||
for (const m of queries.getNodesByKind('method')) {
|
||||
for (const m of queries.iterateNodesByKind('method')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (!m.id.startsWith('expo-module:')) continue;
|
||||
const key = m.qualifiedName.split('::').pop(); // `<module>.<method>`
|
||||
if (!key) continue;
|
||||
@@ -1547,7 +1595,8 @@ function expoCrossPlatformEdges(queries: QueryBuilder): Edge[] {
|
||||
* `getFreeDiskStorage`) — that's the JS-visible name, and how the iOS selector
|
||||
* lines up with the bare Android method name.
|
||||
*/
|
||||
function rnCrossPlatformEdges(queries: QueryBuilder): Edge[] {
|
||||
async function rnCrossPlatformEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
const NATIVE = new Set(['java', 'kotlin', 'objc', 'cpp']);
|
||||
@@ -1570,6 +1619,7 @@ function rnCrossPlatformEdges(queries: QueryBuilder): Edge[] {
|
||||
// below only runs for genuine cross-platform candidates.
|
||||
const byName = new Map<string, Node[]>();
|
||||
for (const m of queries.iterateNodesByKind('method')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (!NATIVE.has(m.language)) continue;
|
||||
const key = norm(m.name);
|
||||
const arr = byName.get(key);
|
||||
@@ -1613,18 +1663,25 @@ function rnCrossPlatformEdges(queries: QueryBuilder): Edge[] {
|
||||
return edges;
|
||||
}
|
||||
|
||||
function fabricNativeImplEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function fabricNativeImplEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
// The Fabric extractor IDs are prefixed `fabric-component:` so we can
|
||||
// filter to just those without iterating all `component` nodes.
|
||||
const components = ctx.getNodesByKind('component').filter((n) => n.id.startsWith('fabric-component:'));
|
||||
// filter to just those while streaming — never materializing the whole
|
||||
// `component` kind (#1212).
|
||||
const components: Node[] = [];
|
||||
for (const n of (ctx.iterateNodesByKind?.('component') ?? ctx.getNodesByKind('component'))) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (n.id.startsWith('fabric-component:')) components.push(n);
|
||||
}
|
||||
if (components.length === 0) return edges;
|
||||
|
||||
// Pre-index native classes by name for O(1) lookup.
|
||||
const nativeClassesByName = new Map<string, Node[]>();
|
||||
for (const n of ctx.getNodesByKind('class')) {
|
||||
for (const n of (ctx.iterateNodesByKind?.('class') ?? ctx.getNodesByKind('class'))) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (n.language !== 'objc' && n.language !== 'kotlin' && n.language !== 'java' && n.language !== 'cpp') continue;
|
||||
const arr = nativeClassesByName.get(n.name);
|
||||
if (arr) arr.push(n);
|
||||
@@ -1675,12 +1732,14 @@ function fabricNativeImplEdges(ctx: ResolutionContext): Edge[] {
|
||||
* same simple name) are dropped. We need-not bridge by package because Java
|
||||
* mapper interfaces are typically uniquely named within a project.
|
||||
*/
|
||||
function mybatisJavaXmlEdges(queries: QueryBuilder): Edge[] {
|
||||
async function mybatisJavaXmlEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
// Index Java methods by `<ClassName>::<methodName>` for O(1) lookup.
|
||||
const javaIndex = new Map<string, Node[]>();
|
||||
for (const m of queries.iterateNodesByKind('method')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (m.language !== 'java' && m.language !== 'kotlin') continue;
|
||||
const parts = m.qualifiedName.split('::');
|
||||
const last = parts[parts.length - 1];
|
||||
@@ -1692,6 +1751,7 @@ function mybatisJavaXmlEdges(queries: QueryBuilder): Edge[] {
|
||||
}
|
||||
|
||||
for (const xml of queries.iterateNodesByKind('method')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (xml.language !== 'xml') continue;
|
||||
// Qualified name: `<namespace>::<id>`. Extract the simple class name.
|
||||
const colonIdx = xml.qualifiedName.lastIndexOf('::');
|
||||
@@ -1785,10 +1845,13 @@ function goHandlerIdent(expr: string): string | null {
|
||||
return m ? m[1]! : null;
|
||||
}
|
||||
|
||||
function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
|
||||
async function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
let scannedFiles = 0;
|
||||
// 1. Find the chain dispatcher(s): a Go method that invokes a `handlers` slice by index.
|
||||
const dispatchers: Node[] = [];
|
||||
for (const n of queries.iterateNodesByKind('method')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (n.language !== 'go') continue;
|
||||
const content = ctx.readFile(n.filePath);
|
||||
const src = content && sliceLines(content, n.startLine, n.endLine);
|
||||
@@ -1801,6 +1864,7 @@ function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionContext):
|
||||
// closures are dropped by goHandlerIdent; the rest are HandlerFuncs.
|
||||
const registered = new Map<string, string>(); // name → registeredAt (file:line)
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!file.endsWith('.go')) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || (!content.includes('.Use(') && !/\.(?:GET|POST|PUT|PATCH|DELETE|OPTIONS|HEAD|Any|Handle)\(/.test(content))) continue;
|
||||
@@ -1852,10 +1916,12 @@ function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionContext):
|
||||
* clause. Link the unit → its form so a `.dfm`/`.fmx` used only as a form
|
||||
* definition isn't orphaned, and editing the form surfaces its code-behind unit.
|
||||
*/
|
||||
function pascalFormEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function pascalFormEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
const edges: Edge[] = [];
|
||||
const allFiles = new Set(ctx.getAllFiles());
|
||||
for (const file of allFiles) {
|
||||
if ((++scannedFiles & 255) === 0) await onYield();
|
||||
if (!/\.(dfm|fmx)$/i.test(file)) continue;
|
||||
const pasFile = file.replace(/\.(dfm|fmx)$/i, '.pas');
|
||||
if (!allFiles.has(pasFile)) continue;
|
||||
@@ -1889,12 +1955,14 @@ function pascalFormEdges(ctx: ResolutionContext): Edge[] {
|
||||
* a loader's data shows the page it feeds) and the page's dependencies include
|
||||
* its loader.
|
||||
*/
|
||||
function svelteKitLoadEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function svelteKitLoadEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
const edges: Edge[] = [];
|
||||
const allFiles = new Set(ctx.getAllFiles());
|
||||
const HOOKS = new Set(['load', 'actions']);
|
||||
const HOOK_KINDS = new Set(['function', 'method', 'constant', 'variable']);
|
||||
for (const file of allFiles) {
|
||||
if ((++scannedFiles & 255) === 0) await onYield();
|
||||
const m = file.match(/(.*\/)(\+(?:page|layout))\.svelte$/);
|
||||
if (!m) continue;
|
||||
const dir = m[1]!;
|
||||
@@ -1941,10 +2009,12 @@ const THUNK_DECL_RE = /create(?:Async)?Thunk/;
|
||||
const THUNK_DISPATCH_RE = /\bdispatch\s*\(\s*([A-Za-z_]\w*)\s*[(),]/g;
|
||||
const THUNK_FANOUT_CAP = 24;
|
||||
|
||||
function reduxThunkEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
|
||||
async function reduxThunkEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const node of queries.iterateNodesByKind('constant')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
// Cheap gate: the initializer (captured in `signature`) must be a create(Async)Thunk call —
|
||||
// avoids reading every constant's body on a large repo.
|
||||
if (!node.signature || !THUNK_DECL_RE.test(node.signature)) continue;
|
||||
@@ -2069,10 +2139,12 @@ function resolveRegistryHandler(ctx: ResolutionContext, name: string, chained: s
|
||||
}
|
||||
|
||||
async function objectRegistryEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
let scanned = 0;
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if ((++scanned & 255) === 0) await onYield(); // #1091: yield mid-scan on huge graphs
|
||||
if (!REGISTRY_JS_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
@@ -2174,10 +2246,12 @@ function rtkEndpointNameFromHook(hook: string): string | null {
|
||||
return mid.charAt(0).toLowerCase() + mid.slice(1);
|
||||
}
|
||||
|
||||
function rtkQueryEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
|
||||
async function rtkQueryEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const hook of queries.iterateNodesByKind('function')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
// Only our extracted generated-hook bindings (sentinel) — not a real hook fn.
|
||||
if (hook.signature !== RTK_GENERATED_HOOK_SIGNATURE) continue;
|
||||
const endpointName = rtkEndpointNameFromHook(hook.name);
|
||||
@@ -2219,10 +2293,12 @@ const PINIA_BIND_RE = /\bconst\s+(\w+)\s*=\s*(?:await\s+)?(\w+)\s*\(/g;
|
||||
const PINIA_CALL_RE = /(\w+)\s*\.\s*(\w+)\s*\(/g;
|
||||
const PINIA_FANOUT_CAP = 80;
|
||||
|
||||
function piniaStoreEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function piniaStoreEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
// 1. Map each `const useXStore = defineStore(...)` factory → its store file.
|
||||
const factoryFile = new Map<string, string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!PINIA_CONSUMER_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || !content.includes('defineStore')) continue;
|
||||
@@ -2235,6 +2311,7 @@ function piniaStoreEdges(ctx: ResolutionContext): Edge[] {
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!PINIA_CONSUMER_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || !content.includes('Store')) continue;
|
||||
@@ -2304,7 +2381,8 @@ function pathHasSegment(filePath: string, seg: string): boolean {
|
||||
return new RegExp('[\\\\/]' + seg.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + '[\\\\/.]').test(filePath);
|
||||
}
|
||||
|
||||
function vuexDispatchEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function vuexDispatchEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
const storeFileCache = new Map<string, boolean>();
|
||||
const isStoreFile = (file: string): boolean => {
|
||||
let v = storeFileCache.get(file);
|
||||
@@ -2339,6 +2417,7 @@ function vuexDispatchEdges(ctx: ResolutionContext): Edge[] {
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!PINIA_CONSUMER_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || (!content.includes('dispatch(') && !content.includes('commit('))) continue;
|
||||
@@ -2395,7 +2474,8 @@ const CELERY_PY_EXT = /\.py$/;
|
||||
const CELERY_FANOUT_CAP = 80;
|
||||
const CELERY_DECORATOR_LOOKBACK = 12; // max lines above a `def` to scan for its decorators
|
||||
|
||||
function celeryDispatchEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function celeryDispatchEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
// Memoize the decorator check per task-candidate node: it reads the file and scans a few
|
||||
// lines above the def. Only called on names that are actually `.delay`/`.apply_async`
|
||||
// receivers, so the candidate set stays small.
|
||||
@@ -2434,6 +2514,7 @@ function celeryDispatchEdges(ctx: ResolutionContext): Edge[] {
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!CELERY_PY_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || (!content.includes('.delay(') && !content.includes('.apply_async('))) continue;
|
||||
@@ -2506,13 +2587,20 @@ function springFirstParamType(sig: string | undefined): string | null {
|
||||
return /^[A-Z][A-Za-z0-9_]*$/.test(type) ? type : null;
|
||||
}
|
||||
|
||||
function springEventEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function springEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
// Pass 1 — event-type → listener methods, scanning only event-relevant files.
|
||||
// This is the ONLY full read sweep: publisher files are recorded here so
|
||||
// pass 2 re-reads just those instead of every .java file again (#1212 —
|
||||
// the double full-repo read was one of the tail's longest unyielded spans).
|
||||
const listeners = new Map<string, Node[]>();
|
||||
const publisherFiles: string[] = [];
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!SPRING_JAVA_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content) continue;
|
||||
if (content.includes('.publishEvent(')) publisherFiles.push(file);
|
||||
const hasAnno = content.includes('@EventListener') || content.includes('@TransactionalEventListener');
|
||||
const hasAppListener = SPRING_APP_LISTENER_RE.test(content);
|
||||
if (!hasAnno && !hasAppListener) continue;
|
||||
@@ -2543,11 +2631,12 @@ function springEventEdges(ctx: ResolutionContext): Edge[] {
|
||||
}
|
||||
if (!listeners.size) return [];
|
||||
|
||||
// Pass 2 — link each publishEvent(new XEvent(...)) site → every listener of XEvent.
|
||||
// Pass 2 — link each publishEvent(new XEvent(...)) site → every listener of
|
||||
// XEvent. Only the publisher files recorded in pass 1 are (re-)read.
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if (!SPRING_JAVA_EXT.test(file)) continue;
|
||||
for (const file of publisherFiles) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || !content.includes('.publishEvent(')) continue;
|
||||
const safe = stripCommentsForRegex(content, 'java');
|
||||
@@ -2626,10 +2715,12 @@ function resolveMediatrArgType(arg: string, lines: string[], methodStart: number
|
||||
return declType;
|
||||
}
|
||||
|
||||
function mediatrDispatchEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function mediatrDispatchEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
// Pass 1 — request/notification type → the Handle method of each handler class.
|
||||
const handlers = new Map<string, Node[]>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!MEDIATR_CS_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || (!content.includes('IRequestHandler<') && !content.includes('INotificationHandler<'))) continue;
|
||||
@@ -2657,6 +2748,7 @@ function mediatrDispatchEdges(ctx: ResolutionContext): Edge[] {
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!MEDIATR_CS_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || (!content.includes('.Send(') && !content.includes('.Publish('))) continue;
|
||||
@@ -2714,7 +2806,8 @@ const SIDEKIQ_WORKER_RE = /\binclude\s+Sidekiq::(?:Job|Worker)\b/;
|
||||
const SIDEKIQ_RB_EXT = /\.rb$/;
|
||||
const SIDEKIQ_FANOUT_CAP = 80;
|
||||
|
||||
function sidekiqDispatchEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function sidekiqDispatchEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
// class node id → its instance `perform` method (null if the class isn't a Sidekiq worker),
|
||||
// memoized. Reads the class body for the mixin; only consulted for actual dispatch receivers.
|
||||
const performCache = new Map<string, Node | null>();
|
||||
@@ -2753,6 +2846,7 @@ function sidekiqDispatchEdges(ctx: ResolutionContext): Edge[] {
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!SIDEKIQ_RB_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || !/\.perform_(?:async|in|at)\b/.test(content)) continue;
|
||||
@@ -2917,6 +3011,7 @@ function nixLeadingPlainSegments(name: string): string[] {
|
||||
}
|
||||
|
||||
async function nixOptionPathEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
type Rec = { id: string; filePath: string; startLine: number; endLine: number; segs: string[] };
|
||||
|
||||
// One streaming pass over nix bindings (variables + the odd function-valued
|
||||
@@ -2925,6 +3020,7 @@ async function nixOptionPathEdges(queries: QueryBuilder, onYield: MaybeYield): P
|
||||
let scanned = 0;
|
||||
for (const kind of ['variable', 'function'] as NodeKind[]) {
|
||||
for (const node of queries.iterateNodesByKind(kind)) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if ((++scanned & 0x3fff) === 0 && onYield) await onYield();
|
||||
if (node.language !== 'nix') continue;
|
||||
const segs = nixLeadingPlainSegments(node.name);
|
||||
@@ -3030,9 +3126,16 @@ async function nixOptionPathEdges(queries: QueryBuilder, onYield: MaybeYield): P
|
||||
return edges;
|
||||
}
|
||||
|
||||
function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
|
||||
// Cheap language gate: no Erlang modules → no cost beyond one kind query.
|
||||
const erlangModules = queries.getNodesByKind('namespace').filter((n) => n.language === 'erlang');
|
||||
async function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
let scanned255 = 0;
|
||||
// Cheap language gate: no Erlang modules → no cost beyond one streamed
|
||||
// kind scan (never a materialized array of every namespace — #1212).
|
||||
const erlangModules: Node[] = [];
|
||||
for (const n of queries.iterateNodesByKind('namespace')) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (n.language === 'erlang') erlangModules.push(n);
|
||||
}
|
||||
if (erlangModules.length === 0) return [];
|
||||
|
||||
// Pass 1 — scan every Erlang file with `-callback` decls: behaviour module →
|
||||
@@ -3045,6 +3148,7 @@ function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: ResolutionCont
|
||||
const declaringBehaviours = new Map<string, Node[]>(); // `fn/arity` → behaviour namespaces
|
||||
const callbackNames = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!ERLANG_EXT.test(file)) continue;
|
||||
const behaviour = moduleByFile.get(file);
|
||||
if (!behaviour) continue; // a .hrl or module-less file can't be a behaviour
|
||||
@@ -3095,6 +3199,7 @@ function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: ResolutionCont
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!ERLANG_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || !/[A-Z][A-Za-z0-9_@]*:[a-z]/.test(content)) continue;
|
||||
@@ -3186,7 +3291,8 @@ function phpArrayBody(src: string, openIdx: number): string | null {
|
||||
return null;
|
||||
}
|
||||
|
||||
function laravelEventEdges(ctx: ResolutionContext): Edge[] {
|
||||
async function laravelEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scannedFiles = 0;
|
||||
// event short name → its listener `handle` methods (deduped by node id).
|
||||
const listeners = new Map<string, Map<string, Node>>();
|
||||
const add = (event: string, handle: Node) => {
|
||||
@@ -3204,6 +3310,7 @@ function laravelEventEdges(ctx: ResolutionContext): Edge[] {
|
||||
|
||||
// Pass 1 — build the event→handle map from both registration mechanisms.
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!LARAVEL_PHP_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content) continue;
|
||||
@@ -3246,6 +3353,7 @@ function laravelEventEdges(ctx: ResolutionContext): Edge[] {
|
||||
const edges: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const file of ctx.getAllFiles()) {
|
||||
if ((++scannedFiles & 15) === 0) await onYield();
|
||||
if (!LARAVEL_PHP_EXT.test(file)) continue;
|
||||
const content = ctx.readFile(file);
|
||||
if (!content || !content.includes('event(')) continue;
|
||||
@@ -3301,59 +3409,90 @@ export async function synthesizeCallbackEdges(queries: QueryBuilder, ctx: Resolu
|
||||
// watchdog still catches that. See ./cooperative-yield.
|
||||
const yieldToLoop = createYielder();
|
||||
|
||||
// Per-pass wall-clock timing to stderr, opt-in via CODEGRAPH_SYNTH_TIMINGS
|
||||
// (=1: passes over 250ms; =all: every pass). This is the diagnostic that
|
||||
// located both the #1091/#1122 watchdog stalls and the #1212 OOM — keep it.
|
||||
const markT = { t: Date.now() };
|
||||
const __mark = (label: string): void => {
|
||||
const now = Date.now();
|
||||
const dt = now - markT.t;
|
||||
markT.t = now;
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS && (dt > 250 || process.env.CODEGRAPH_SYNTH_TIMINGS === 'all')) {
|
||||
console.error(`[synth-timing] ${label}: ${dt}ms`);
|
||||
}
|
||||
};
|
||||
|
||||
// Language gating: one indexed DISTINCT over the files table lets a pass
|
||||
// whose own filters reference a specific language/extension be skipped
|
||||
// outright when the project has no such files — its result is provably
|
||||
// empty, so skipping is behavior-identical and the cost drops to zero
|
||||
// (the Kotlin pass was the OOM culprit on the pure-C Linux kernel, #1212).
|
||||
// Passes without an explicit language filter always run.
|
||||
const langs = queries.getDistinctFileLanguages();
|
||||
const has = (...ls: string[]): boolean => ls.some((l) => langs.has(l));
|
||||
const JS_FAMILY = ['typescript', 'javascript', 'tsx', 'jsx'];
|
||||
const NONE: Edge[] = [];
|
||||
|
||||
// Cross-file Go method→type `contains` edges must be synthesized AND persisted
|
||||
// FIRST: a method declared in a different file from its receiver type is
|
||||
// otherwise orphaned from the struct, and goImplementsEdges (next) derives a
|
||||
// struct's method set from its `contains` edges — so without this it would
|
||||
// under-count the interfaces a cross-file struct satisfies. (#583)
|
||||
const goMethodContains = goCrossFileMethodContainsEdges(queries);
|
||||
if (goMethodContains.length > 0) queries.insertEdges(goMethodContains);
|
||||
await yieldToLoop();
|
||||
const goMethodContains = has('go') ? await goCrossFileMethodContainsEdges(queries, yieldToLoop) : NONE;
|
||||
for (let i = 0; i < goMethodContains.length; i += 2000) {
|
||||
queries.insertEdges(goMethodContains.slice(i, i + 2000));
|
||||
await yieldToLoop();
|
||||
}
|
||||
await yieldToLoop(); __mark('goMethodContains');
|
||||
|
||||
// Go implicit `implements` edges must be synthesized AND persisted next: the
|
||||
// interface-dispatch bridge below reads `implements` edges from the DB, and
|
||||
// Go has none statically. (Other languages already have static implements
|
||||
// edges from extraction, so they don't need this pre-pass.)
|
||||
const goImpl = goImplementsEdges(queries);
|
||||
if (goImpl.length > 0) queries.insertEdges(goImpl);
|
||||
await yieldToLoop();
|
||||
const goImpl = has('go') ? await goImplementsEdges(queries, yieldToLoop) : NONE;
|
||||
for (let i = 0; i < goImpl.length; i += 2000) {
|
||||
queries.insertEdges(goImpl.slice(i, i + 2000));
|
||||
await yieldToLoop();
|
||||
}
|
||||
await yieldToLoop(); __mark('goImplements');
|
||||
|
||||
const fieldEdges = await fieldChannelEdges(queries, ctx, yieldToLoop); await yieldToLoop();
|
||||
const closureCollEdges = await closureCollectionEdges(queries, ctx, yieldToLoop); await yieldToLoop();
|
||||
const emitterEdges = await eventEmitterEdges(ctx, yieldToLoop); await yieldToLoop();
|
||||
const renderEdges = reactRenderEdges(queries, ctx); await yieldToLoop();
|
||||
const jsxEdges = await reactJsxChildEdges(ctx, yieldToLoop); await yieldToLoop();
|
||||
const vueEdges = vueTemplateEdges(ctx); await yieldToLoop();
|
||||
const svelteKitEdges = svelteKitLoadEdges(ctx); await yieldToLoop();
|
||||
const pascalEdges = pascalFormEdges(ctx); await yieldToLoop();
|
||||
const flutterEdges = flutterBuildEdges(queries, ctx); await yieldToLoop();
|
||||
const arkuiStateEdges = arkuiStateBuildEdges(queries, ctx); await yieldToLoop();
|
||||
const arkuiEmitter = arkuiEmitterEdges(ctx); await yieldToLoop();
|
||||
const arkuiRoutes = arkuiRouterEdges(ctx); await yieldToLoop();
|
||||
const cppEdges = cppOverrideEdges(queries); await yieldToLoop();
|
||||
const ifaceEdges = interfaceOverrideEdges(queries); await yieldToLoop();
|
||||
const kotlinExpectActual = kotlinExpectActualEdges(queries); await yieldToLoop();
|
||||
const goGrpcEdges = goGrpcStubImplEdges(queries); await yieldToLoop();
|
||||
const rnEventEdgesList = rnEventEdges(ctx); await yieldToLoop();
|
||||
const fabricNativeEdges = fabricNativeImplEdges(ctx); await yieldToLoop();
|
||||
const expoXPlatEdges = expoCrossPlatformEdges(queries); await yieldToLoop();
|
||||
const rnXPlatEdges = rnCrossPlatformEdges(queries); await yieldToLoop();
|
||||
const mybatisEdges = mybatisJavaXmlEdges(queries); await yieldToLoop();
|
||||
const ginEdges = ginMiddlewareChainEdges(queries, ctx); await yieldToLoop();
|
||||
const thunkEdges = reduxThunkEdges(queries, ctx); await yieldToLoop();
|
||||
const registryEdges = await objectRegistryEdges(ctx, yieldToLoop); await yieldToLoop();
|
||||
const rtkEdges = rtkQueryEdges(queries, ctx); await yieldToLoop();
|
||||
const piniaEdges = piniaStoreEdges(ctx); await yieldToLoop();
|
||||
const vuexEdges = vuexDispatchEdges(ctx); await yieldToLoop();
|
||||
const celeryEdges = celeryDispatchEdges(ctx); await yieldToLoop();
|
||||
const springEdges = springEventEdges(ctx); await yieldToLoop();
|
||||
const mediatrEdges = mediatrDispatchEdges(ctx); await yieldToLoop();
|
||||
const sidekiqEdges = sidekiqDispatchEdges(ctx); await yieldToLoop();
|
||||
const erlangBehaviourEdges = erlangBehaviourDispatchEdges(queries, ctx); await yieldToLoop();
|
||||
const laravelEdges = laravelEventEdges(ctx); await yieldToLoop();
|
||||
const cFnPtrEdges = cFnPointerDispatchEdges(queries, ctx); await yieldToLoop();
|
||||
const goframeEdges = goframeRouteEdges(ctx); await yieldToLoop();
|
||||
const nixOptionEdges = await nixOptionPathEdges(queries, yieldToLoop); await yieldToLoop();
|
||||
const fieldEdges = await fieldChannelEdges(queries, ctx, yieldToLoop); await yieldToLoop(); __mark('fieldEdges');
|
||||
const closureCollEdges = await closureCollectionEdges(queries, ctx, yieldToLoop); await yieldToLoop(); __mark('closureCollEdges');
|
||||
const emitterEdges = await eventEmitterEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('emitterEdges');
|
||||
const renderEdges = await reactRenderEdges(queries, ctx, yieldToLoop); await yieldToLoop(); __mark('renderEdges');
|
||||
const jsxEdges = await reactJsxChildEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('jsxEdges');
|
||||
const vueEdges = has('vue') ? await vueTemplateEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('vueEdges');
|
||||
const svelteKitEdges = has('svelte') ? await svelteKitLoadEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('svelteKitEdges');
|
||||
const pascalEdges = await pascalFormEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('pascalEdges');
|
||||
const flutterEdges = has('dart') ? await flutterBuildEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('flutterEdges');
|
||||
const arkuiStateEdges = has('arkts') ? await arkuiStateBuildEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('arkuiStateEdges');
|
||||
const arkuiEmitter = has('arkts') ? await arkuiEmitterEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('arkuiEmitter');
|
||||
const arkuiRoutes = has('arkts') ? await arkuiRouterEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('arkuiRoutes');
|
||||
const cppEdges = has('cpp') ? await cppOverrideEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('cppEdges');
|
||||
const ifaceEdges = has('java', 'kotlin', 'csharp', 'swift', 'scala', 'go', 'rust', 'arkts', ...JS_FAMILY)
|
||||
? await interfaceOverrideEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('ifaceEdges');
|
||||
const kotlinExpectActual = has('kotlin') ? await kotlinExpectActualEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('kotlinExpectActual');
|
||||
const goGrpcEdges = has('go') ? await goGrpcStubImplEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('goGrpcEdges');
|
||||
const rnEventEdgesList = has(...JS_FAMILY) ? await rnEventEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('rnEventEdgesList');
|
||||
const fabricNativeEdges = await fabricNativeImplEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('fabricNativeEdges');
|
||||
const expoXPlatEdges = await expoCrossPlatformEdges(queries, yieldToLoop); await yieldToLoop(); __mark('expoXPlatEdges');
|
||||
const rnXPlatEdges = await rnCrossPlatformEdges(queries, yieldToLoop); await yieldToLoop(); __mark('rnXPlatEdges');
|
||||
const mybatisEdges = has('java', 'kotlin') && has('xml') ? await mybatisJavaXmlEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('mybatisEdges');
|
||||
const ginEdges = has('go') ? await ginMiddlewareChainEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('ginEdges');
|
||||
const thunkEdges = has(...JS_FAMILY) ? await reduxThunkEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('thunkEdges');
|
||||
const registryEdges = await objectRegistryEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('registryEdges');
|
||||
const rtkEdges = has(...JS_FAMILY) ? await rtkQueryEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('rtkEdges');
|
||||
const piniaEdges = has('vue', ...JS_FAMILY) ? await piniaStoreEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('piniaEdges');
|
||||
const vuexEdges = has('vue', ...JS_FAMILY) ? await vuexDispatchEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('vuexEdges');
|
||||
const celeryEdges = has('python') ? await celeryDispatchEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('celeryEdges');
|
||||
const springEdges = has('java') ? await springEventEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('springEdges');
|
||||
const mediatrEdges = has('csharp') ? await mediatrDispatchEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('mediatrEdges');
|
||||
const sidekiqEdges = has('ruby') ? await sidekiqDispatchEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('sidekiqEdges');
|
||||
const erlangBehaviourEdges = has('erlang') ? await erlangBehaviourDispatchEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('erlangBehaviourEdges');
|
||||
const laravelEdges = has('php') ? await laravelEventEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('laravelEdges');
|
||||
const cFnPtrEdges = has('c', 'cpp') ? await cFnPointerDispatchEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('cFnPtrEdges');
|
||||
const goframeEdges = has('go') ? await goframeRouteEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('goframeEdges');
|
||||
const nixOptionEdges = has('nix') ? await nixOptionPathEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('nixOptionEdges');
|
||||
|
||||
const merged: Edge[] = [];
|
||||
const seen = new Set<string>();
|
||||
@@ -3400,6 +3539,14 @@ export async function synthesizeCallbackEdges(queries: QueryBuilder, ctx: Resolu
|
||||
seen.add(key);
|
||||
merged.push(e);
|
||||
}
|
||||
if (merged.length > 0) queries.insertEdges(merged);
|
||||
__mark('dedupe-merge');
|
||||
// Chunked insert with yields: on the Linux kernel the merged synthesized
|
||||
// edge set is ~275k rows, and one transaction for all of them was a 20s
|
||||
// unyielded main-thread span (#1212 follow-up) — the last one in the tail.
|
||||
for (let i = 0; i < merged.length; i += 2000) {
|
||||
queries.insertEdges(merged.slice(i, i + 2000));
|
||||
await yieldToLoop();
|
||||
}
|
||||
__mark('insertMergedEdges');
|
||||
return merged.length + goImpl.length + goMethodContains.length;
|
||||
}
|
||||
|
||||
@@ -25,6 +25,7 @@
|
||||
|
||||
import type { Edge, Node } from '../types';
|
||||
import type { ResolutionContext } from './types';
|
||||
import type { MaybeYield } from './cooperative-yield';
|
||||
import { GOFRAME_ROUTE_MARKER } from './frameworks/goframe';
|
||||
|
||||
const FANOUT_CAP = 2000; // backstop only; real apps are 1 route → 1 method.
|
||||
@@ -73,13 +74,15 @@ function selectHandler(candidates: Node[], routeFile: string): Node | null {
|
||||
return sameModule.length === 1 ? sameModule[0]! : null;
|
||||
}
|
||||
|
||||
export function goframeRouteEdges(ctx: ResolutionContext): Edge[] {
|
||||
export async function goframeRouteEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
// Route nodes the goframe extractor created, keyed by their package-qualified
|
||||
// request type (`cash.ListReq`). `wanted` holds every key a handler signature
|
||||
// could match — the qualified form plus its bare type fallback.
|
||||
const routesByReqType = new Map<string, Node[]>();
|
||||
const wanted = new Set<string>();
|
||||
for (const route of ctx.getNodesByKind('route')) {
|
||||
for (const route of (ctx.iterateNodesByKind?.('route') ?? ctx.getNodesByKind('route'))) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (route.language !== 'go') continue;
|
||||
const marker = route.qualifiedName.indexOf(GOFRAME_ROUTE_MARKER);
|
||||
if (marker < 0) continue;
|
||||
@@ -98,7 +101,8 @@ export function goframeRouteEdges(ctx: ResolutionContext): Edge[] {
|
||||
// pointer, indexed by every matching (qualified + bare) form so a route can
|
||||
// match precisely on `pkg.Type` and fall back to the bare `Type`.
|
||||
const handlersByKey = new Map<string, Node[]>();
|
||||
for (const method of ctx.getNodesByKind('method')) {
|
||||
for (const method of (ctx.iterateNodesByKind?.('method') ?? ctx.getNodesByKind('method'))) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (method.language !== 'go' || !method.signature) continue;
|
||||
for (const t of pointerParamTypes(method.signature)) {
|
||||
if (!wanted.has(t)) continue;
|
||||
|
||||
+60
-20
@@ -328,6 +328,30 @@ export class ReferenceResolver {
|
||||
this.cachesWarmed = true;
|
||||
}
|
||||
|
||||
/**
|
||||
* warmCaches for the async resolution entry points: streams the distinct
|
||||
* name set with periodic yields instead of one synchronous `.all()`. On a
|
||||
* multi-million-node index the DISTINCT scan is a solid multi-second block
|
||||
* (measured up to 28s inside `codegraph sync` on the Linux kernel index),
|
||||
* long enough to matter to the #850 watchdog on slower hardware. Same
|
||||
* result, same memory — only the event loop keeps turning.
|
||||
*/
|
||||
async warmCachesYielding(onYield: MaybeYield): Promise<void> {
|
||||
if (this.cachesWarmed) return;
|
||||
|
||||
this.knownFiles = new Set(this.queries.getAllFilePaths());
|
||||
|
||||
const names = new Set<string>();
|
||||
let scanned = 0;
|
||||
for (const name of this.queries.iterateNodeNames()) {
|
||||
names.add(name);
|
||||
if ((++scanned & 8191) === 0) await onYield();
|
||||
}
|
||||
this.knownNames = names;
|
||||
|
||||
this.cachesWarmed = true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear internal caches
|
||||
*/
|
||||
@@ -421,6 +445,12 @@ export class ReferenceResolver {
|
||||
return result;
|
||||
},
|
||||
|
||||
// Streamed, uncached — synthesizers scan-and-filter whole kinds, and
|
||||
// both the materialized array AND the per-kind cache retention are
|
||||
// O(nodes) memory (#1212). Per-ref resolvers keep the cached array
|
||||
// variant above.
|
||||
iterateNodesByKind: (kind: Node['kind']) => this.queries.iterateNodesByKind(kind),
|
||||
|
||||
fileExists: (filePath: string) => {
|
||||
// Check pre-built known files set first (O(1))
|
||||
if (this.knownFiles) {
|
||||
@@ -1113,8 +1143,6 @@ export class ReferenceResolver {
|
||||
onProgress?: (current: number, total: number) => void,
|
||||
batchSize: number = 5000
|
||||
): Promise<ResolutionResult> {
|
||||
this.warmCaches();
|
||||
|
||||
// Resolution runs on the indexer's MAIN thread, and the #850 liveness
|
||||
// watchdog SIGKILLs a process whose event loop stalls past its window (60s
|
||||
// by default). A single dense batch's resolveAll — or the synthesis pass
|
||||
@@ -1123,6 +1151,8 @@ export class ReferenceResolver {
|
||||
// window to fire; see ./cooperative-yield.
|
||||
const maybeYield = createYielder();
|
||||
|
||||
await this.warmCachesYielding(maybeYield);
|
||||
|
||||
const total = this.queries.getUnresolvedReferencesCount();
|
||||
let processed = 0;
|
||||
const aggregateStats = {
|
||||
@@ -1141,32 +1171,42 @@ export class ReferenceResolver {
|
||||
|
||||
const result = await this.resolveBatchYielding(batch, maybeYield);
|
||||
|
||||
// Persist in bounded sub-transactions with yields between: a whole
|
||||
// batch's edge insert / keyed deletes are otherwise one solid
|
||||
// synchronous span each on a multi-GB index, sitting BETWEEN the
|
||||
// per-ref yields — the last unyielded stretch of the resolution loop.
|
||||
// Crash semantics are unchanged (already several transactions): edges
|
||||
// land before their refs are deleted, so a kill mid-way re-resolves
|
||||
// the remainder idempotently on the next run/sweep (#1187).
|
||||
const PERSIST_CHUNK = 1000;
|
||||
|
||||
// Persist edges immediately
|
||||
const edges = this.createEdges(result.resolved);
|
||||
if (edges.length > 0) {
|
||||
this.queries.insertEdges(edges);
|
||||
for (let i = 0; i < edges.length; i += PERSIST_CHUNK) {
|
||||
this.queries.insertEdges(edges.slice(i, i + PERSIST_CHUNK));
|
||||
await maybeYield();
|
||||
}
|
||||
|
||||
// Clean up resolved refs so they don't appear in the next batch
|
||||
if (result.resolved.length > 0) {
|
||||
this.queries.deleteSpecificResolvedReferences(
|
||||
result.resolved.map((r) => ({
|
||||
fromNodeId: r.original.fromNodeId,
|
||||
referenceName: r.original.referenceName,
|
||||
referenceKind: r.original.referenceKind,
|
||||
}))
|
||||
);
|
||||
const resolvedKeys = result.resolved.map((r) => ({
|
||||
fromNodeId: r.original.fromNodeId,
|
||||
referenceName: r.original.referenceName,
|
||||
referenceKind: r.original.referenceKind,
|
||||
}));
|
||||
for (let i = 0; i < resolvedKeys.length; i += PERSIST_CHUNK) {
|
||||
this.queries.deleteSpecificResolvedReferences(resolvedKeys.slice(i, i + PERSIST_CHUNK));
|
||||
await maybeYield();
|
||||
}
|
||||
|
||||
// Delete unresolvable refs from this batch to avoid re-processing them
|
||||
if (result.unresolved.length > 0) {
|
||||
this.queries.deleteSpecificResolvedReferences(
|
||||
result.unresolved.map((r) => ({
|
||||
fromNodeId: r.fromNodeId,
|
||||
referenceName: r.referenceName,
|
||||
referenceKind: r.referenceKind,
|
||||
}))
|
||||
);
|
||||
const unresolvedKeys = result.unresolved.map((r) => ({
|
||||
fromNodeId: r.fromNodeId,
|
||||
referenceName: r.referenceName,
|
||||
referenceKind: r.referenceKind,
|
||||
}));
|
||||
for (let i = 0; i < unresolvedKeys.length; i += PERSIST_CHUNK) {
|
||||
this.queries.deleteSpecificResolvedReferences(unresolvedKeys.slice(i, i + PERSIST_CHUNK));
|
||||
await maybeYield();
|
||||
}
|
||||
|
||||
// Aggregate stats
|
||||
|
||||
@@ -71,6 +71,15 @@ export interface ResolutionContext {
|
||||
getNodesByQualifiedName(qualifiedName: string): Node[];
|
||||
/** Get all nodes of a kind */
|
||||
getNodesByKind(kind: Node['kind']): Node[];
|
||||
/**
|
||||
* Stream nodes of a kind one at a time instead of materializing (and, unlike
|
||||
* `getNodesByKind`, without populating the resolver's per-kind array cache).
|
||||
* For unbounded kinds (`function`, `method`, `struct`) on a symbol-dense
|
||||
* project the full array is gigabytes — the dynamic-edge synthesizers must
|
||||
* use this so their memory stays O(1) in node count (#610, #1212). Optional
|
||||
* so minimal test contexts compile; callers fall back to getNodesByKind.
|
||||
*/
|
||||
iterateNodesByKind?(kind: Node['kind']): IterableIterator<Node>;
|
||||
/** Check if a file exists */
|
||||
fileExists(filePath: string): boolean;
|
||||
/** Read file content */
|
||||
|
||||
Reference in New Issue
Block a user