fix(index): rebuild a poisoned/oversized index by recreating the DB, not row-DELETE (#1067) (#1073)

Follow-up to #1065/#1066. Those stopped a *new* index from scanning an
ignored gitlink corpus, but a project that had already built the multi-GB
graph before upgrading still couldn't recover: `codegraph index` printed
only "Indexing project" and was then SIGKILLed (137) by the #850 watchdog
~60s later, before scanning even started.

Root cause is not the scanner. `index` cleared the old graph with a
synchronous `DELETE FROM nodes/edges/files`. `nodes` carries an FTS5
`AFTER DELETE` trigger, so deleting ~1.6M rows fires ~1.6M FTS
delete-markers — O(rows), and it grows the WAL further before it can
finish. A deterministic probe puts the DELETE-clear at 20.4s on 1.5M
synthetic nodes (WAL 1.16->2.14GB); at the report's denser ~2.6KB/node WAL
that crosses the 60s main-thread watchdog. `open()` was never the wedge.

A full re-index is documented as "same result as a fresh init", so make it
one: discard the database files and re-initialize, instead of opening the
old DB and DELETE-ing every row.

- db: add removeDatabaseFiles(dbPath) — unlinks codegraph.db + its
  -wal/-shm sidecars (O(1) regardless of size; sidecars best-effort).
- index: add CodeGraph.recreate(projectRoot) — discards the files and
  returns a fresh, empty instance. Never opens or migrates the poisoned
  DB. POSIX unlinks an open file fine (a live daemon heals via
  reopenIfReplaced, #925); a Windows file lock becomes an actionable
  "stop the daemon / remove .codegraph" error.
- cli: `codegraph index` now calls recreate() instead of open()+clear();
  both clear() calls dropped. The public clear() API is unchanged.

This also reclaims the disk the bloated db/-wal were holding.

Validated: deterministic probe (DELETE O(rows) vs recreate O(1)); an
end-to-end run through the built binary recovering a real 800K-node /
419MB poisoned DB in 0.3s with no wedge and the correct small graph; new
unit + CLI regression tests; existing #874 index tests still green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

src/bin/codegraph.ts

@@ -632,16 +632,23 @@ program
       }
 
       const { default: CodeGraph } = await loadCodeGraph();
-      const cg = await CodeGraph.open(projectPath);
+      // `index` is a FULL re-index — identical to a fresh `init`. RECREATE the
+      // database from scratch (discard .codegraph/codegraph.db + its WAL) rather
+      // than opening the old graph and DELETE-ing every row. The clear-then-index
+      // approach reported "0 nodes" without the clear (#874); the recreate keeps
+      // that fixed AND avoids the failure mode where, on a large or pre-fix
+      // poisoned index, the per-row FTS delete churn wedged the main thread long
+      // enough to trip the liveness watchdog before scanning even began (#1067).
+      // recreate() hands back a fresh, empty instance — no clear() needed. For
+      // fast incremental updates use `sync`.
+      const cg = await CodeGraph.recreate(projectPath);
 
       // Supervise the indexer: self-terminate if orphaned (parent shim killed)
       // or if the main thread wedges — neither was guarded on this path (#999).
       const supervision = installCommandSupervision('index');
       try {
         if (options.quiet) {
-          // Quiet mode: no UI, just run. `index` is a full re-index, so clear the
-          // existing graph and rebuild from scratch (see the note below — #874).
-          cg.clear();
+          // Quiet mode: no UI, just run against the freshly-recreated graph.
           const result = await cg.indexAll();
           if (!result.success) process.exit(1);
           cg.destroy();
@@ -651,13 +658,6 @@ program
         const clack = await importESM('@clack/prompts');
         clack.intro('Indexing project');
 
-        // `index` is a FULL re-index: clear the existing graph and rebuild it from
-        // scratch so the result is identical to a fresh `init`. Without the clear,
-        // indexAll() skips every unchanged file by its content hash and reports
-        // "0 nodes, 0 edges" against the already-populated graph — which reads as
-        // "index wiped my index" (#874). For fast incremental updates use `sync`.
-        cg.clear();
-
         let result: IndexResult;
 
         if (options.verbose) {

src/db/index.ts

@@ -281,9 +281,48 @@ function statInode(p: string): string | null {
  */
 export const DATABASE_FILENAME = 'codegraph.db';
 
+/**
+ * SQLite's sidecar files in WAL mode — the write-ahead log and its shared-memory
+ * index. They sit beside the main DB file and are removed alongside it when the
+ * database is discarded (see `removeDatabaseFiles`).
+ */
+const WAL_SIDECAR_SUFFIXES = ['-wal', '-shm'] as const;
+
 /**
  * Get the default database path for a project
  */
 export function getDatabasePath(projectRoot: string): string {
   return path.join(getCodeGraphDir(projectRoot), DATABASE_FILENAME);
 }
+
+/**
+ * Delete a database file and its WAL sidecars (`-wal`/`-shm`).
+ *
+ * This is how a FULL re-index discards an existing database — rather than
+ * opening the old graph and DELETE-ing every row. On a large or pre-fix
+ * poisoned index (e.g. an old graph that scanned an ignored gitlink corpus into
+ * ~1.6M nodes with a multi-GB WAL, #1065) the per-row `nodes_fts` delete-trigger
+ * churn blocks the main thread long enough to trip the #850 liveness watchdog
+ * before indexing even starts, so the rebuild could never recover the bad state
+ * (#1067). Unlinking is O(1) regardless of DB size and also reclaims the disk
+ * the bloated WAL would otherwise keep.
+ *
+ * POSIX removes the directory entry even while another process (a daemon/MCP
+ * server) still holds the file open; that holder heals via `reopenIfReplaced`
+ * (#925). On Windows a live holder can make the unlink fail with EBUSY/EPERM —
+ * that is thrown for the caller to surface ("stop the other process and retry").
+ * The `-wal`/`-shm` sidecars are best-effort: SQLite recreates them on the next
+ * open, so a leftover sidecar is harmless.
+ */
+export function removeDatabaseFiles(dbPath: string): void {
+  // The main DB file first — its removal is the operation that must succeed (or
+  // report why it couldn't). force:true treats an already-missing file as done.
+  fs.rmSync(dbPath, { force: true });
+  for (const suffix of WAL_SIDECAR_SUFFIXES) {
+    try {
+      fs.rmSync(dbPath + suffix, { force: true });
+    } catch {
+      // A sidecar still held/locked is harmless — SQLite rebuilds it on open.
+    }
+  }
+}

src/index.ts

@@ -22,7 +22,7 @@ import {
   BuildContextOptions,
   FindRelevantContextOptions,
 } from './types';
-import { DatabaseConnection, getDatabasePath } from './db';
+import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db';
 import { QueryBuilder } from './db/queries';
 import {
   isInitialized,
@@ -319,6 +319,54 @@ export class CodeGraph {
     return instance;
   }
 
+  /**
+   * Rebuild the project's database from scratch and return a fresh, empty
+   * instance — the "same result as a fresh init" semantics that `codegraph
+   * index` documents.
+   *
+   * Unlike `open()` followed by `clear()`, this DISCARDS the existing
+   * `.codegraph/codegraph.db` (and its `-wal`/`-shm` sidecars) before
+   * re-initializing, instead of opening the old database and DELETE-ing every
+   * row. On a large or pre-fix poisoned index — e.g. an old graph that scanned
+   * an ignored gitlink corpus (#1065) into ~1.6M nodes with a multi-GB WAL —
+   * the per-row `nodes_fts` delete-trigger churn blocks the main thread long
+   * enough to trip the #850 liveness watchdog before indexing even starts, so a
+   * full re-index could never recover the bad state (#1067). Discarding the
+   * files is O(1) regardless of size, reclaims the disk, and sidesteps opening
+   * (and running migrations against) the poisoned database entirely.
+   */
+  static async recreate(projectRoot: string): Promise<CodeGraph> {
+    await initGrammars();
+    const resolvedRoot = path.resolve(projectRoot);
+
+    // Check if initialized — recreate REBUILDS an existing project; it is not a
+    // first-time `init`.
+    if (!isInitialized(resolvedRoot)) {
+      throw new Error(`CodeGraph not initialized in ${resolvedRoot}. Run init() first.`);
+    }
+
+    const dbPath = getDatabasePath(resolvedRoot);
+    try {
+      removeDatabaseFiles(dbPath);
+    } catch (err) {
+      // POSIX unlinks an open file fine; this fires mainly on Windows when a
+      // live daemon/MCP server still holds the database. Turn the raw EBUSY into
+      // an actionable instruction instead of a generic failure.
+      const reason = err instanceof Error ? err.message : String(err);
+      throw new Error(
+        `Could not rebuild the index — the database file is in use (${reason}). ` +
+          `Stop any running CodeGraph MCP server/daemon for this project and retry, ` +
+          `or remove the ${getCodeGraphDir(resolvedRoot)} directory and run "codegraph init".`
+      );
+    }
+
+    // Re-create an empty, freshly-schema'd database at the same path.
+    const db = DatabaseConnection.initialize(dbPath);
+    const queries = new QueryBuilder(db.getDb());
+
+    return new CodeGraph(db, queries, resolvedRoot);
+  }
+
   /**
    * Open synchronously (without sync)
    */