Self-contained distribution: bundle Node + node:sqlite, drop better-sqlite3/wasm (closes #238) (#282)
* fix(db): eliminate concurrent-read "database is locked"; add node:sqlite backend (#238) WAL + busy_timeout were already enabled, so the issue's suggested fix was a no-op. The real causes, addressed here: - busy_timeout is now set first (before journal_mode) and lowered 120s -> 5s, so open-time pragmas wait out a lock instead of hanging for two minutes. - getCodeGraph no longer opens a second connection to the default project when a tool passes its own projectPath (the in-process lock amplifier). - The wasm fallback (no WAL) gets a bounded read-retry on SQLITE_BUSY. - New: node:sqlite backend, preferred over wasm, so installs whose native better-sqlite3 build fails land on a real-WAL backend instead of no-WAL wasm. - codegraph status / codegraph_status now report the effective journal mode, so a lock report is triageable (wal vs delete). - CLI hard-blocks Node < 20 to actually enforce the engines floor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(db)!: node:sqlite is the sole backend; drop better-sqlite3 + wasm Now that distribution will bundle a Node 24 runtime, node:sqlite (real SQLite with WAL + FTS5) is always available. Collapse the three-backend adapter to node:sqlite only and remove the machinery the other two needed: - Remove better-sqlite3 (optionalDependency) and node-sqlite3-wasm (dependency). - Remove WasmDatabaseAdapter, the named->positional param translation, the SQLITE_BUSY read-retry, the wasm fallback banner, the backend env override, and the native/node-sqlite/wasm selection chain. - createDatabase now opens node:sqlite directly, with a clear error pointing at the bundled release / Node 22.5+ when the module is absent. - NodeSqliteAdapter.close() is idempotent and pragma() supports { simple }, to match the better-sqlite3 behavior callers relied on. - status (CLI + MCP) reports the single node:sqlite backend; journal-mode diagnostics and the getCodeGraph single-connection fix are retained. - Tests repointed off better-sqlite3 onto node:sqlite. Net -1044 lines. Running from source now requires Node 22.5+. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dist): self-contained bundle prototype (vendored Node + install channels) Phase 3 of the node:sqlite migration: ship a vendored Node runtime so CodeGraph runs with no system Node and no native build (node:sqlite is built in). - scripts/build-bundle.sh: build a per-platform archive (official Node + dist + prod deps + launcher). Same recipe per platform; pins Node v24.16.0. - install.sh: curl|sh installer (no Node required) — detects os/arch, pulls the archive from Releases, symlinks onto PATH; re-run to upgrade, --uninstall to remove. The VPS/SSH path. - scripts/npm-shim.js: thin launcher for the npm channel — resolves the per-platform optionalDependency bundle and execs it, so `npm i -g` keeps working and the real work runs on the bundled Node regardless of the user's. - BUNDLING.md: distribution design + release-pipeline TODO (CI matrix, platform packages, code signing, brew, retiring the Node-version gate). Validated end-to-end: darwin-arm64 and linux-x64 bundles both run init + index + status (Backend: node:sqlite, Journal: wal) + FTS query with NO system Node — linux-x64 verified in a clean ubuntu:24.04 amd64 container. Release archives are gitignored; CI will produce and upload them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dist): add Windows PowerShell installer (install.ps1) The `irm … | iex` one-liner for Windows, mirroring install.sh: detect arch, pull the matching bundle from Releases, extract to %LOCALAPPDATA%\codegraph, add it to user PATH. Re-run to upgrade. (Windows bundle production in build-bundle.sh is still TODO.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dist): release workflow + npm packaging; README/CHANGELOG for bundled distro - .github/workflows/release.yml: manually-triggered (workflow_dispatch) release matrix. Builds a self-contained bundle per platform on its own runner (darwin-arm64/x64, linux-x64/arm64), publishes a GitHub Release with all archives, and publishes the npm thin-installer (shim + per-platform packages). Windows targets are TODO (build-bundle.sh is unix-only). - scripts/pack-npm.sh: assemble the npm packages from built bundles — per-platform packages tagged os/cpu + the main shim package with them as optionalDependencies (esbuild pattern). Proven locally: npm-install the tarballs, run via the shim, resolves the bundle and runs on the bundled Node 24 (node:sqlite / WAL). - README: install section now leads with the no-Node one-liners (curl|sh, irm|iex) then npm/npx; "bundled · none required" badge. - CHANGELOG: standout headline for the self-contained release, plus Added/Changed/ Removed for the install channels, node:sqlite backend, and dropped deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dist): Windows bundles + single-trigger release workflow - build-bundle.sh: add win32-x64 / win32-arm64 targets — download Node's Windows zip, bundle node.exe + a .cmd launcher, output a .zip. Verified structurally (PE32+ node.exe, CRLF .cmd, portable node_modules). Since there are no native addons, any target builds on any OS, so the whole matrix builds on one runner. - pack-npm.sh: handle .zip bundles and win32 packages (os: win32, node.exe). - release.yml: simplified to your spec — manual trigger reads the version from package.json, builds all platform bundles, creates the GitHub Release with notes pulled from CHANGELOG.md, and publishes the npm shim + platform packages. - BUNDLING.md: Windows + build-anywhere notes; release pipeline documented. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
b8aec39abd
commit
ac52fd76c0
@@ -0,0 +1,152 @@
|
||||
/**
|
||||
* Issue #238 — "database is locked" on concurrent MCP tool calls.
|
||||
*
|
||||
* With node:sqlite (real WAL) as the backend, the fixes that remain relevant:
|
||||
* 1. busy_timeout is a bounded few-second wait (not a 2-minute hang) and WAL is
|
||||
* active — so a reader never blocks on a concurrent writer.
|
||||
* 2. The MCP ToolHandler reuses the default instance when a tool passes a
|
||||
* projectPath pointing at the default project, instead of opening a SECOND
|
||||
* connection to the same DB.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import CodeGraph from '../src';
|
||||
import { ToolHandler } from '../src/mcp/tools';
|
||||
import { DatabaseConnection } from '../src/db';
|
||||
|
||||
/** Normalize a PRAGMA read across return shapes (array | object | scalar). */
|
||||
function pragmaValue(raw: unknown, key: string): unknown {
|
||||
const row = Array.isArray(raw) ? raw[0] : raw;
|
||||
if (row !== null && typeof row === 'object') return (row as Record<string, unknown>)[key];
|
||||
return row;
|
||||
}
|
||||
|
||||
describe('issue #238 — connection PRAGMAs (#1)', () => {
|
||||
let dir: string;
|
||||
let conn: DatabaseConnection;
|
||||
|
||||
beforeAll(() => {
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg238-pragma-'));
|
||||
conn = DatabaseConnection.initialize(path.join(dir, 'codegraph.db'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
conn.close();
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('uses a bounded busy_timeout, not the old 2-minute hang', () => {
|
||||
const ms = Number(pragmaValue(conn.getDb().pragma('busy_timeout'), 'timeout'));
|
||||
expect(ms).toBeGreaterThan(0);
|
||||
expect(ms).toBeLessThanOrEqual(30000); // far below the old 120000
|
||||
});
|
||||
|
||||
it('runs in WAL mode — the mode that lets readers proceed during a write', () => {
|
||||
const mode = String(pragmaValue(conn.getDb().pragma('journal_mode'), 'journal_mode')).toLowerCase();
|
||||
expect(mode).toBe('wal');
|
||||
});
|
||||
|
||||
it('getJournalMode() surfaces the effective mode for status triage', () => {
|
||||
expect(conn.getJournalMode()).toBe('wal');
|
||||
});
|
||||
});
|
||||
|
||||
describe('issue #238 — WAL lets a reader proceed during a writer', () => {
|
||||
let dir: string;
|
||||
|
||||
beforeAll(() => {
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg238-wal-'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('a read on a 2nd connection succeeds while a writer holds the lock', () => {
|
||||
const dbPath = path.join(dir, 'codegraph.db');
|
||||
const writer = DatabaseConnection.initialize(dbPath);
|
||||
// The property only holds under WAL; skip if the filesystem couldn't enable it.
|
||||
if (writer.getJournalMode() !== 'wal') {
|
||||
writer.close();
|
||||
return;
|
||||
}
|
||||
const reader = DatabaseConnection.open(dbPath);
|
||||
try {
|
||||
writer.getDb().prepare('BEGIN EXCLUSIVE').run(); // hard write lock, held open
|
||||
const t0 = Date.now();
|
||||
const row = reader.getDb().prepare('SELECT COUNT(*) AS c FROM nodes').get() as { c: number };
|
||||
const waited = Date.now() - t0;
|
||||
expect(row.c).toBe(0);
|
||||
expect(waited).toBeLessThan(1000); // proceeds immediately, no busy wait
|
||||
} finally {
|
||||
try { writer.getDb().prepare('COMMIT').run(); } catch { /* ignore */ }
|
||||
reader.close();
|
||||
writer.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('issue #238 — ToolHandler reuses the default instance (#2)', () => {
|
||||
let dir: string;
|
||||
let cg: CodeGraph;
|
||||
let root: string;
|
||||
let handler: ToolHandler;
|
||||
|
||||
beforeAll(async () => {
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg238-tools-'));
|
||||
fs.writeFileSync(path.join(dir, 'a.ts'), 'export function helper(): number { return 1; }\n');
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'b.ts'),
|
||||
"import { helper } from './a';\nexport function main(): number { return helper(); }\n"
|
||||
);
|
||||
cg = await CodeGraph.init(dir, { index: true });
|
||||
root = cg.getProjectRoot();
|
||||
handler = new ToolHandler(cg);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cg.close();
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('getCodeGraph(defaultRoot) returns the default instance, not a new connection', () => {
|
||||
const openSpy = vi.spyOn(CodeGraph, 'openSync');
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const resolved = (handler as any).getCodeGraph(root);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const nested = (handler as any).getCodeGraph(path.join(root, 'does', 'not', 'exist'));
|
||||
expect(resolved).toBe(cg);
|
||||
expect(nested).toBe(cg); // a sub-path resolves up to the same default project
|
||||
expect(openSpy).not.toHaveBeenCalled(); // no second connection opened
|
||||
} finally {
|
||||
openSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('concurrent read tool calls (mixed projectPath) all succeed without "database is locked"', async () => {
|
||||
const openSpy = vi.spyOn(CodeGraph, 'openSync');
|
||||
try {
|
||||
const calls: Promise<{ content: Array<{ text: string }>; isError?: boolean }>[] = [
|
||||
handler.execute('codegraph_search', { query: 'helper' }),
|
||||
handler.execute('codegraph_search', { query: 'helper', projectPath: root }),
|
||||
handler.execute('codegraph_callers', { symbol: 'helper', projectPath: root }),
|
||||
handler.execute('codegraph_callees', { symbol: 'main' }),
|
||||
handler.execute('codegraph_files', { projectPath: root }),
|
||||
handler.execute('codegraph_status', { projectPath: root }),
|
||||
];
|
||||
const results = await Promise.all(calls);
|
||||
for (const r of results) {
|
||||
expect(r.isError).not.toBe(true);
|
||||
expect(r.content[0]?.text ?? '').not.toMatch(/database is locked/i);
|
||||
}
|
||||
// Passing the default project's own path must not open a second connection.
|
||||
expect(openSpy).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
openSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* node:sqlite backend (issue #238 follow-up).
|
||||
*
|
||||
* node:sqlite (Node's built-in real SQLite) is now the sole backend. This drives
|
||||
* a real index + queries through it, so WAL, FTS5 search, and @named-param
|
||||
* writes are all exercised end-to-end.
|
||||
*
|
||||
* Skipped on Node < 22.5 where node:sqlite doesn't exist.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import CodeGraph from '../src';
|
||||
|
||||
let nodeSqliteAvailable = false;
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||
require('node:sqlite');
|
||||
nodeSqliteAvailable = true;
|
||||
} catch {
|
||||
nodeSqliteAvailable = false;
|
||||
}
|
||||
|
||||
describe.skipIf(!nodeSqliteAvailable)('node:sqlite backend — real index + queries', () => {
|
||||
let dir: string;
|
||||
let cg: CodeGraph;
|
||||
|
||||
beforeAll(async () => {
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-nodesqlite-'));
|
||||
fs.writeFileSync(path.join(dir, 'a.ts'), 'export function helper(): number { return 1; }\n');
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'b.ts'),
|
||||
"import { helper } from './a';\nexport function main(): number { return helper(); }\n"
|
||||
);
|
||||
cg = await CodeGraph.init(dir, { index: true });
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cg?.close();
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('uses the node:sqlite backend', () => {
|
||||
expect(cg.getBackend()).toBe('node-sqlite');
|
||||
});
|
||||
|
||||
it('runs in WAL mode — the whole reason it beats the wasm fallback', () => {
|
||||
expect(cg.getJournalMode()).toBe('wal');
|
||||
});
|
||||
|
||||
it('indexed the project (write path: @named-param INSERTs via node:sqlite)', () => {
|
||||
const stats = cg.getStats();
|
||||
expect(stats.fileCount).toBe(2);
|
||||
expect(stats.nodeCount).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('FTS5 search returns the indexed symbol (read path)', () => {
|
||||
const results = cg.searchNodes('helper');
|
||||
const names = results.map(r => r.node.name);
|
||||
expect(names).toContain('helper');
|
||||
});
|
||||
|
||||
it('graph traversal resolves the cross-file caller', () => {
|
||||
const helper = cg.searchNodes('helper').find(r => r.node.name === 'helper');
|
||||
expect(helper).toBeTruthy();
|
||||
const callers = cg.getCallers(helper!.node.id);
|
||||
expect(callers.map(c => c.node.name)).toContain('main');
|
||||
});
|
||||
});
|
||||
@@ -7,7 +7,7 @@
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { buildNode25BlockBanner } from '../src/bin/node-version-check';
|
||||
import { buildNode25BlockBanner, buildNodeTooOldBanner, MIN_NODE_MAJOR } from '../src/bin/node-version-check';
|
||||
|
||||
describe('buildNode25BlockBanner', () => {
|
||||
it('embeds the reported Node version in the header', () => {
|
||||
@@ -41,3 +41,29 @@ describe('buildNode25BlockBanner', () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildNodeTooOldBanner', () => {
|
||||
it('embeds the reported Node version in the header', () => {
|
||||
expect(buildNodeTooOldBanner('18.20.0')).toContain(
|
||||
'Unsupported Node.js version: 18.20.0'
|
||||
);
|
||||
});
|
||||
|
||||
it('states the supported floor matching MIN_NODE_MAJOR', () => {
|
||||
expect(MIN_NODE_MAJOR).toBe(20);
|
||||
expect(buildNodeTooOldBanner('18.0.0')).toContain(
|
||||
`requires Node.js ${MIN_NODE_MAJOR} or newer`
|
||||
);
|
||||
});
|
||||
|
||||
it('points users to Node 22 LTS via nvm and Homebrew', () => {
|
||||
const banner = buildNodeTooOldBanner('16.0.0');
|
||||
expect(banner).toContain('Node.js 22 LTS');
|
||||
expect(banner).toContain('nvm install 22');
|
||||
expect(banner).toContain('brew install node@22');
|
||||
});
|
||||
|
||||
it('documents the CODEGRAPH_ALLOW_UNSAFE_NODE override', () => {
|
||||
expect(buildNodeTooOldBanner('18.0.0')).toContain('CODEGRAPH_ALLOW_UNSAFE_NODE=1');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -45,11 +45,11 @@ function cleanupTempDir(dir: string): void {
|
||||
}
|
||||
}
|
||||
|
||||
// Check if better-sqlite3 native bindings are available
|
||||
// Check if the node:sqlite backend is available (Node >= 22.5)
|
||||
function hasSqliteBindings(): boolean {
|
||||
try {
|
||||
const Database = require('better-sqlite3');
|
||||
const db = new Database(':memory:');
|
||||
const { DatabaseSync } = require('node:sqlite');
|
||||
const db = new DatabaseSync(':memory:');
|
||||
db.close();
|
||||
return true;
|
||||
} catch {
|
||||
|
||||
@@ -1,59 +1,18 @@
|
||||
/**
|
||||
* SQLite backend visibility tests
|
||||
* SQLite backend reporting.
|
||||
*
|
||||
* Pins the WASM-fallback banner content + the per-instance backend
|
||||
* tracking. Closes the visibility gap behind issue #138.
|
||||
* node:sqlite (Node's built-in real SQLite) is the sole backend. Pin that
|
||||
* DatabaseConnection / CodeGraph report it and come up in WAL.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import {
|
||||
buildWasmFallbackBanner,
|
||||
WASM_FALLBACK_FIX_RECIPE,
|
||||
} from '../src/db/sqlite-adapter';
|
||||
import { DatabaseConnection } from '../src/db';
|
||||
import { CodeGraph } from '../src';
|
||||
|
||||
describe('buildWasmFallbackBanner — fix-recipe content', () => {
|
||||
it('includes the macOS / Linux / cross-platform fix commands', () => {
|
||||
const banner = buildWasmFallbackBanner();
|
||||
expect(banner).toContain('WASM SQLite fallback active');
|
||||
expect(banner).toContain('5-10x slower');
|
||||
expect(banner).toContain('xcode-select --install');
|
||||
expect(banner).toContain('apt install build-essential');
|
||||
expect(banner).toContain('npm rebuild better-sqlite3');
|
||||
expect(banner).toContain('npm install better-sqlite3 --save');
|
||||
expect(banner).toContain('codegraph status');
|
||||
});
|
||||
|
||||
it('appends the native load error when one is provided', () => {
|
||||
const banner = buildWasmFallbackBanner(
|
||||
"Cannot find module 'better-sqlite3'"
|
||||
);
|
||||
expect(banner).toContain(
|
||||
"Native load error: Cannot find module 'better-sqlite3'"
|
||||
);
|
||||
});
|
||||
|
||||
it('omits the load-error block when no error is supplied', () => {
|
||||
const banner = buildWasmFallbackBanner();
|
||||
expect(banner).not.toContain('Native load error:');
|
||||
});
|
||||
});
|
||||
|
||||
describe('WASM_FALLBACK_FIX_RECIPE — single source of truth', () => {
|
||||
it('mentions the three recovery commands', () => {
|
||||
expect(WASM_FALLBACK_FIX_RECIPE).toContain('xcode-select --install');
|
||||
expect(WASM_FALLBACK_FIX_RECIPE).toContain('npm rebuild better-sqlite3');
|
||||
expect(WASM_FALLBACK_FIX_RECIPE).toContain(
|
||||
'npm install better-sqlite3 --save'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('DatabaseConnection — per-instance backend reporting', () => {
|
||||
describe('DatabaseConnection — backend reporting', () => {
|
||||
let dir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
@@ -66,11 +25,10 @@ describe('DatabaseConnection — per-instance backend reporting', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('reports a concrete backend (native or wasm) for an initialized DB', () => {
|
||||
const dbPath = path.join(dir, 'test.db');
|
||||
const conn = DatabaseConnection.initialize(dbPath);
|
||||
const backend = conn.getBackend();
|
||||
expect(['native', 'wasm']).toContain(backend);
|
||||
it('reports the node-sqlite backend in WAL for an initialized DB', () => {
|
||||
const conn = DatabaseConnection.initialize(path.join(dir, 'test.db'));
|
||||
expect(conn.getBackend()).toBe('node-sqlite');
|
||||
expect(conn.getJournalMode()).toBe('wal');
|
||||
conn.close();
|
||||
});
|
||||
|
||||
@@ -78,7 +36,7 @@ describe('DatabaseConnection — per-instance backend reporting', () => {
|
||||
fs.writeFileSync(path.join(dir, 'x.ts'), `export function x(): void {}\n`);
|
||||
const cg = await CodeGraph.init(dir, { index: true });
|
||||
try {
|
||||
expect(['native', 'wasm']).toContain(cg.getBackend());
|
||||
expect(cg.getBackend()).toBe('node-sqlite');
|
||||
} finally {
|
||||
cg.destroy();
|
||||
}
|
||||
|
||||
@@ -25,8 +25,8 @@ beforeAll(async () => {
|
||||
|
||||
function hasSqliteBindings(): boolean {
|
||||
try {
|
||||
const Database = require('better-sqlite3');
|
||||
const db = new Database(':memory:');
|
||||
const { DatabaseSync } = require('node:sqlite');
|
||||
const db = new DatabaseSync(':memory:');
|
||||
db.close();
|
||||
return true;
|
||||
} catch {
|
||||
|
||||
Reference in New Issue
Block a user