perf(resolution): worker connection recycling — WAL-depth writes-under-readers fix, superphase −11.4% at 8c (#1362)
The §7a.6 anomaly probed to its mechanism with five discriminating runs (§7a.7 table): main-thread B-tree writes triple under attached readers because READERS PIN WAL checkpoint progress — the deep WAL taxes every writer page operation (deletes 42.6s pool-off vs 118.8s pool-4 on identical hardware; an aggressive 64MB valve recovers the writes but overpays +129s in full-park folds; the v2 cache resurrection was falsified — long-tail name traffic is uncacheable at any capacity). Fix: workers close and reopen their read-only connections every 8 batches at the double-buffer's worker-idle boundary (ResolverPool.recycleWorkers + QueryBuilder.rebind + a cadence call). Reopens are sub-millisecond, resolver caches survive (only prepared statements re-prepare), and the existing checkpoints advance instead of parking. Failed recycle downgrades to sequential, same as a failed fan-out. Measured (8c pool-4, linux v7.2-rc2, cadence 25 → 8 iterated): resolution superphase 715.0 → 633.6s (−11.4%), envelope best 14.8min, recreate 59.7 → 45.3s. Byte-neutral everywhere: git dumps byte-identical old-vs-new, linux dump sha 6dd1185b reproduced (10,446,478 lines), counts 2,049,153/6,413,518, suite 2517 green. 2c unchanged by construction (no pool → no recycling). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
5955d04c97
commit
971a5a0483
@@ -63,6 +63,7 @@ export class ResolverPool {
|
||||
private nextId = 0;
|
||||
private waiters = new Map<number, { resolve: (r: ChunkResult) => void; reject: (e: Error) => void }>();
|
||||
private synthWaiters = new Map<number, { resolve: (r: SynthPassResult) => void; reject: (e: Error) => void }>();
|
||||
private recycleWaiters = new Map<number, () => void>();
|
||||
private failed: Error | null = null;
|
||||
|
||||
/**
|
||||
@@ -174,6 +175,10 @@ export class ResolverPool {
|
||||
const waiter = this.synthWaiters.get(msg.id);
|
||||
this.synthWaiters.delete(msg.id);
|
||||
waiter?.resolve({ edges: msg.edges ?? [], ms: msg.ms ?? 0 });
|
||||
} else if (msg.type === 'recycled' && msg.id !== undefined) {
|
||||
const waiter = this.recycleWaiters.get(msg.id);
|
||||
this.recycleWaiters.delete(msg.id);
|
||||
waiter?.();
|
||||
} else if (msg.type === 'error') {
|
||||
pw.busy--;
|
||||
const err = new Error(`resolver worker: ${msg.message}`);
|
||||
@@ -211,6 +216,10 @@ export class ResolverPool {
|
||||
this.waiters.clear();
|
||||
for (const [, waiter] of this.synthWaiters) waiter.reject(this.failed);
|
||||
this.synthWaiters.clear();
|
||||
// Pending recycles resolve rather than reject: their per-call timeout
|
||||
// owns rejection, and the recycle caller checks this.failed next round.
|
||||
for (const [, done] of this.recycleWaiters) done();
|
||||
this.recycleWaiters.clear();
|
||||
}
|
||||
|
||||
/** Whether this batch is worth fanning out. */
|
||||
@@ -273,6 +282,43 @@ export class ResolverPool {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Ask every worker to close and reopen its read-only connection, and wait
|
||||
* for all acks. MUST be called only at the pool-idle boundary (all fanned
|
||||
* chunks settled, next batch not yet dispatched) — the workers close their
|
||||
* connections in place. Why: a long-lived reader pins WAL checkpoint
|
||||
* progress, and the deep WAL behind it taxes every main-thread B-tree
|
||||
* page operation (writes-under-readers, plan §7a.6 — deletes 42.6→118.8s
|
||||
* from 0 to 4 attached readers). Releasing the read marks periodically
|
||||
* lets the existing checkpoints advance, keeping the WAL shallow WITHOUT
|
||||
* the full-park folds an aggressive valve pays (+129s measured at 64MB).
|
||||
* A recycle failure fails the pool — the caller's sequential fallback
|
||||
* covers the rest of the run.
|
||||
*/
|
||||
async recycleWorkers(): Promise<void> {
|
||||
if (this.failed) throw this.failed;
|
||||
await Promise.all(
|
||||
this.workers.map(
|
||||
(pw) =>
|
||||
new Promise<void>((resolve, reject) => {
|
||||
const id = this.nextId++;
|
||||
const t = setTimeout(() => {
|
||||
if (this.recycleWaiters.delete(id)) {
|
||||
const err = new Error('resolver worker recycle timed out');
|
||||
this.fail(err);
|
||||
reject(err);
|
||||
}
|
||||
}, 10_000);
|
||||
this.recycleWaiters.set(id, () => {
|
||||
clearTimeout(t);
|
||||
resolve();
|
||||
});
|
||||
pw.worker.postMessage({ type: 'recycle', id });
|
||||
})
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
async destroy(): Promise<void> {
|
||||
await Promise.all(
|
||||
this.workers.map(
|
||||
|
||||
Reference in New Issue
Block a user