fix(explore): damp ambient declaration files on flow queries (CG-28)
A file that declares nothing but types and that nothing in the index depends on — a hand-written ambient `.d.ts` of global shims, vendored typings, module augmentation — cannot answer a flow question: no bodies, no call edges, no behaviour, nothing typed by it. But the identifiers it declares are exactly the generic ones a prose question uses (`Body`, `Message`, `ImageMetadata`, `ReadableStream`), so on term overlap it out-scored the implementation. Measured on the new fixture: rank #1 and 51% of delivered source, with the flow's own entry file pushed out of the response entirely. Measured first, per the issue: the Wrangler `worker-configuration.d.ts` that opened this is already handled by CG-25's banner detection, worth 15-46 points of envelope share across four flow queries. CG-25 credited; only the un-bannered case needed anything. `rankPenalty` now multiplies score and graph mass by 0.5 for such files, taken as the STRONGER of it and the generated penalty rather than multiplied — one property two signals see must not be charged twice. Detection is structural, not by extension, and four conditions deep. Two of them were forced by measurement: requiring every symbol to be type-level takes the corpus flag rate from 1-18% (which swept in Kotlin sealed classes, Rust mod.rs re-exports and django's locale tables) down to 0-4%; requiring that nothing depends on the file separates an ambient shim from a working types module, and without it the rule demoted displacement-ts's pipeline `types.ts` and broke the CG-31 gate. A query that NAMES a declared type is exempt, so a question about a type still reaches its declaration at full weight. Precise tokens only, so "…the file body…" cannot exempt a `Body` interface it never meant to name; this needs its own set because `namedSeedIds` is callable-only and a type never becomes one. Regression evidence in docs/benchmarks/explore-declaration-only-cg28.md: 6-repo envelope sweep byte-identical against a clean baseline build, zero ambient files reach the candidate set on VS Code across five queries, corpus flag rate 0-0.74%, both allocation fixtures PASS, full suite 2,978 green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
co-authored by
Claude Opus 5
Claude Opus 5
parent
463f6e7844
commit
9efae0f8f2
@@ -0,0 +1,207 @@
|
||||
/**
|
||||
* Regression gate for DECLARATION-ONLY files in explore ranking (task CG-28).
|
||||
*
|
||||
* A file that holds nothing but type declarations — an ambient `.d.ts`, vendored
|
||||
* typings, a `types.ts` of pure interfaces — cannot answer a FLOW question: no
|
||||
* bodies, no call edges, no behaviour. But the identifiers it declares are
|
||||
* exactly the generic ones a prose question uses (`Body`, `Message`,
|
||||
* `ImageMetadata`, `ReadableStream`), so on term overlap it out-scored the
|
||||
* implementation and took the envelope. Measured on this fixture before the fix:
|
||||
* rank #1 and 51% of delivered source on a prose flow query.
|
||||
*
|
||||
* CG-25 already covers the file that STARTED this — a Wrangler
|
||||
* `worker-configuration.d.ts`, which announces itself with a generated banner.
|
||||
* `docs/benchmarks/explore-declaration-only-cg28.md` has that measurement; the
|
||||
* banner alone is worth 15–46 points of envelope share. What it does not cover
|
||||
* is a declaration file with no banner at all, which is what this fixture's
|
||||
* `platform-shims.d.ts` is, and what the damping in `rankPenalty` addresses.
|
||||
*
|
||||
* Two claims, and BOTH have to hold — the counter-case is why the penalty is
|
||||
* guarded rather than flat:
|
||||
*
|
||||
* 1. a prose flow query must not let a declaration-only file outrank the
|
||||
* implementation files that answer it;
|
||||
* 2. a query genuinely ABOUT a declared type must still reach the declaration
|
||||
* at full weight.
|
||||
*
|
||||
* The suppression the issue explicitly forbids is also pinned: a damped file is
|
||||
* still a candidate and still named in the response, so one follow-up explore
|
||||
* fetches it.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import CodeGraph from '../src/index';
|
||||
import { ToolHandler } from '../src/mcp/tools';
|
||||
import type { ExploreDiagnosticReport, ExploreDiagnosticFile } from '../src/mcp/explore-diagnostics';
|
||||
|
||||
const FIXTURE_SRC = path.join(__dirname, 'fixtures', 'ambient-decls-ts');
|
||||
|
||||
/** Declaration-only, hand-written, NO generated banner — the surviving gap. */
|
||||
const HANDWRITTEN_DECL = 'types/platform-shims.d.ts';
|
||||
/** Declaration-only WITH a Wrangler banner — the CG-25 control in the same run. */
|
||||
const GENERATED_DECL = 'types/worker-configuration.d.ts';
|
||||
/** Declaration-only but IMPORTED by the storage layer — must never be damped. */
|
||||
const SHARED_TYPES = 'src/storage/types.ts';
|
||||
|
||||
/** Prose, naming no symbol — the query shape that let the original file in. */
|
||||
const FLOW_QUERY =
|
||||
'how does an upload request stream the file body to storage and record image metadata';
|
||||
/** Prose that DOES name a declared type — the counter-case. */
|
||||
const TYPE_QUERY = 'what does the UploadStorage interface declare for putting an object';
|
||||
|
||||
describe('CG-28 — a declaration-only file does not outrank implementation on a flow query', () => {
|
||||
let testDir: string;
|
||||
let cg: CodeGraph;
|
||||
let sidecar: string;
|
||||
|
||||
/** One explore call; returns its diagnostic report plus the response text. */
|
||||
const explore = async (query: string): Promise<{ report: ExploreDiagnosticReport; text: string }> => {
|
||||
fs.rmSync(sidecar, { force: true });
|
||||
const previous = process.env.CODEGRAPH_EXPLORE_DEBUG;
|
||||
process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar;
|
||||
let text: string;
|
||||
try {
|
||||
text = (await new ToolHandler(cg).execute('codegraph_explore', { query })).content?.[0]?.text ?? '';
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.CODEGRAPH_EXPLORE_DEBUG;
|
||||
else process.env.CODEGRAPH_EXPLORE_DEBUG = previous;
|
||||
}
|
||||
const written = fs.readFileSync(sidecar, 'utf-8').trim().split('\n').filter(Boolean);
|
||||
return { report: JSON.parse(written[written.length - 1]!) as ExploreDiagnosticReport, text };
|
||||
};
|
||||
|
||||
const fileOf = (report: ExploreDiagnosticReport, p: string): ExploreDiagnosticFile | undefined =>
|
||||
report.files.find((f) => f.path === p);
|
||||
|
||||
let flow: { report: ExploreDiagnosticReport; text: string };
|
||||
let typed: { report: ExploreDiagnosticReport; text: string };
|
||||
|
||||
beforeAll(async () => {
|
||||
testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cg28-'));
|
||||
fs.cpSync(FIXTURE_SRC, testDir, { recursive: true });
|
||||
fs.rmSync(path.join(testDir, '.codegraph'), { recursive: true, force: true });
|
||||
sidecar = path.join(testDir, 'explore-diag.jsonl');
|
||||
|
||||
cg = CodeGraph.initSync(testDir);
|
||||
await cg.indexAll();
|
||||
|
||||
flow = await explore(FLOW_QUERY);
|
||||
typed = await explore(TYPE_QUERY);
|
||||
}, 120_000);
|
||||
|
||||
afterAll(() => {
|
||||
if (cg) cg.destroy();
|
||||
if (testDir && fs.existsSync(testDir)) fs.rmSync(testDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('fixture shape — if this rots, the gate below means nothing', () => {
|
||||
it('holds two declaration-only files that differ only in the banner', () => {
|
||||
for (const p of [HANDWRITTEN_DECL, GENERATED_DECL]) {
|
||||
const nodes = cg.getNodesInFile(p).filter((n) => n.kind !== 'file' && n.kind !== 'import');
|
||||
expect(nodes.length, `${p} declares nothing`).toBeGreaterThan(10);
|
||||
// Every symbol type-level, nothing with a body — the structural test the
|
||||
// penalty keys on. A `function`/`class` creeping in would silently exempt
|
||||
// the file and make every assertion below vacuous.
|
||||
expect(nodes.every((n) => n.kind === 'interface' || n.kind === 'type_alias'), `${p} has a non-type symbol`).toBe(true);
|
||||
}
|
||||
// Only one of them announces itself, so the CG-25 penalty is the ONLY
|
||||
// difference between the two — that is what makes them comparable.
|
||||
expect(cg.getFile(GENERATED_DECL)?.generated).toBe(true);
|
||||
expect(cg.getFile(HANDWRITTEN_DECL)?.generated).toBeFalsy();
|
||||
});
|
||||
|
||||
it('holds a pure-type module the code IMPORTS, as the safety control', () => {
|
||||
// Identical to the ambient files on kinds and bodies; different only in
|
||||
// that the storage layer is typed by it. This is the shape the penalty
|
||||
// must NOT catch — a `types.ts` the codebase depends on is part of the
|
||||
// structure of any answer about that code.
|
||||
const nodes = cg.getNodesInFile(SHARED_TYPES).filter((n) => n.kind !== 'file' && n.kind !== 'import');
|
||||
expect(nodes.length).toBeGreaterThan(0);
|
||||
expect(nodes.every((n) => n.kind === 'interface' || n.kind === 'type_alias')).toBe(true);
|
||||
expect(cg.getFile(SHARED_TYPES)?.generated).toBeFalsy();
|
||||
});
|
||||
|
||||
it('holds implementation files that DO answer the flow question', () => {
|
||||
for (const p of ['src/routes/upload.ts', 'src/storage/stream.ts', 'src/storage/metadata.ts']) {
|
||||
expect(cg.getNodesInFile(p).some((n) => n.kind === 'function'), `${p} has no functions`).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('the gate — a prose flow query', () => {
|
||||
it('damps the un-bannered declaration file rather than letting it rank free', () => {
|
||||
const rec = fileOf(flow.report, HANDWRITTEN_DECL);
|
||||
expect(rec, 'the declaration file is not even a candidate — fixture drifted').toBeDefined();
|
||||
expect(rec!.ambientDeclaration).toBe(true);
|
||||
expect(rec!.penalty).toBeLessThan(1);
|
||||
});
|
||||
|
||||
it('does not let it outrank the implementation files', () => {
|
||||
const decl = fileOf(flow.report, HANDWRITTEN_DECL)!;
|
||||
const impl = flow.report.files.filter((f) => f.path.startsWith('src/') && f.finalChars > 0);
|
||||
expect(impl.length, 'no implementation file delivered anything').toBeGreaterThanOrEqual(2);
|
||||
// Measured before the fix: the declaration file was rank #1 with score 53
|
||||
// against the best implementation file's 34. The bar is that at least one
|
||||
// implementation file now ranks above it — ordinary budget movement must
|
||||
// not fail the suite, but the inversion coming back must.
|
||||
expect(impl.some((f) => f.rank < decl.rank), 'declaration file still ranks first').toBe(true);
|
||||
});
|
||||
|
||||
it('still names it in the response, so one follow-up call fetches it', () => {
|
||||
// The issue forbids suppression: a damped file must remain reachable.
|
||||
expect(flow.text).toContain(HANDWRITTEN_DECL);
|
||||
});
|
||||
|
||||
it('leaves the implementation files at full weight', () => {
|
||||
for (const f of flow.report.files.filter((x) => x.path.startsWith('src/'))) {
|
||||
expect(f.ambientDeclaration, `${f.path} was misread as an ambient declaration`).toBe(false);
|
||||
expect(f.penalty).toBe(1);
|
||||
}
|
||||
});
|
||||
|
||||
it('does not damp a pure-type module the codebase imports', () => {
|
||||
// The condition that keeps this narrow enough to be safe. Without it the
|
||||
// same rule demotes `displacement-ts`'s pipeline `types.ts` — pure
|
||||
// interfaces, but 13 inbound imports — and breaks the CG-31 gate.
|
||||
const rec = flow.report.files.find((f) => f.path === SHARED_TYPES);
|
||||
if (rec) {
|
||||
expect(rec.ambientDeclaration, `${SHARED_TYPES} was flagged ambient`).toBe(false);
|
||||
expect(rec.penalty).toBe(1);
|
||||
}
|
||||
// Independent of whether this query ranked it: the predicate itself must
|
||||
// separate the two shapes.
|
||||
const isAmbient = cg.ambientDeclarationFilePredicate([SHARED_TYPES, HANDWRITTEN_DECL]);
|
||||
expect(isAmbient(SHARED_TYPES)).toBe(false);
|
||||
expect(isAmbient(HANDWRITTEN_DECL)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the counter-case — a query that NAMES a declared type', () => {
|
||||
it('reaches the declaration at full weight, undamped', () => {
|
||||
const rec = fileOf(typed.report, HANDWRITTEN_DECL);
|
||||
expect(rec, 'the named type\'s file is not a candidate').toBeDefined();
|
||||
expect(rec!.ambientDeclaration).toBe(true);
|
||||
// Detected as declaration-only, but EXEMPT — the query asked for it.
|
||||
expect(rec!.penalty).toBe(1);
|
||||
});
|
||||
|
||||
it('ranks it first and delivers its source', () => {
|
||||
const rec = fileOf(typed.report, HANDWRITTEN_DECL)!;
|
||||
expect(rec.rank).toBe(1);
|
||||
expect(rec.finalChars).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the two penalties do not stack', () => {
|
||||
it('charges a generated declaration file once, at the stronger rate', () => {
|
||||
// A file that is BOTH generated and declaration-only has ONE property two
|
||||
// signals happen to see. Penalising twice (0.3 * 0.5 = 0.15) is how a file
|
||||
// gets cliffed out of answers where it is genuinely relevant.
|
||||
const rec = flow.report.files.find((f) => f.generated && f.ambientDeclaration);
|
||||
if (!rec) return; // not a candidate for this query — nothing to assert
|
||||
expect(rec.penalty).toBeGreaterThanOrEqual(0.3);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"name": "ambient-decls-ts-fixture",
|
||||
"private": true,
|
||||
"version": "0.0.0",
|
||||
"type": "module",
|
||||
"description": "CG-28 fixture — declaration-only files competing with implementation for one explore envelope."
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
export interface BucketObject {
|
||||
key: string;
|
||||
body: ReadableStream<Uint8Array>;
|
||||
size: number;
|
||||
}
|
||||
|
||||
export interface Bucket {
|
||||
put(
|
||||
key: string,
|
||||
value: ReadableStream<Uint8Array>,
|
||||
options?: { httpMetadata?: { contentType?: string } },
|
||||
): Promise<void>;
|
||||
get(key: string): Promise<BucketObject | null>;
|
||||
}
|
||||
|
||||
export interface MetadataStore {
|
||||
put(id: string, value: string): Promise<void>;
|
||||
get(id: string): Promise<string | null>;
|
||||
}
|
||||
|
||||
const objects = new Map<string, BucketObject>();
|
||||
const rows = new Map<string, string>();
|
||||
|
||||
/** The object-storage binding. */
|
||||
export function openBucket(): Bucket {
|
||||
return {
|
||||
async put(key, value) {
|
||||
objects.set(key, { key, body: value, size: 0 });
|
||||
},
|
||||
async get(key) {
|
||||
return objects.get(key) ?? null;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** The metadata key-value binding. */
|
||||
export function openMetadataStore(): MetadataStore {
|
||||
return {
|
||||
async put(id, value) {
|
||||
rows.set(id, value);
|
||||
},
|
||||
async get(id) {
|
||||
return rows.get(id) ?? null;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
export interface UploadMessageBody {
|
||||
key: string;
|
||||
metadataId: string;
|
||||
contentType: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Publish the follow-up message for a stored upload. Batched so a burst of
|
||||
* uploads does not open one producer call per object.
|
||||
*/
|
||||
export async function enqueueUploadMessage(body: UploadMessageBody): Promise<void> {
|
||||
const queue = openUploadQueue();
|
||||
await queue.send(body, { contentType: 'json' });
|
||||
}
|
||||
|
||||
/** Consumer side: process a batch of upload messages. */
|
||||
export async function consumeUploadBatch(messages: UploadMessageBody[]): Promise<number> {
|
||||
let handled = 0;
|
||||
for (const message of messages) {
|
||||
if (!message.key) continue;
|
||||
handled += 1;
|
||||
}
|
||||
return handled;
|
||||
}
|
||||
|
||||
interface UploadQueue {
|
||||
send(body: UploadMessageBody, options: { contentType: string }): Promise<void>;
|
||||
}
|
||||
|
||||
/** The binding lookup, isolated so tests can swap it. */
|
||||
export function openUploadQueue(): UploadQueue {
|
||||
return {
|
||||
async send() {
|
||||
/* binding provided by the runtime */
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
export interface ParsedUpload {
|
||||
ok: true;
|
||||
key: string;
|
||||
body: ReadableStream<Uint8Array>;
|
||||
contentType: string;
|
||||
width: number;
|
||||
height: number;
|
||||
format: string;
|
||||
}
|
||||
|
||||
export interface ParseFailure {
|
||||
ok: false;
|
||||
error: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pull the object key, declared dimensions and the raw body stream off an
|
||||
* upload request. Never buffers the body — the stream is handed straight to
|
||||
* the storage layer.
|
||||
*/
|
||||
export async function parseUploadRequest(
|
||||
request: Request,
|
||||
): Promise<ParsedUpload | ParseFailure> {
|
||||
const url = new URL(request.url);
|
||||
const key = url.searchParams.get('key');
|
||||
if (!key) return { ok: false, error: 'missing key' };
|
||||
if (!request.body) return { ok: false, error: 'missing body' };
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
key,
|
||||
body: request.body as ReadableStream<Uint8Array>,
|
||||
contentType: request.headers.get('content-type') ?? 'application/octet-stream',
|
||||
width: numberParam(url, 'width'),
|
||||
height: numberParam(url, 'height'),
|
||||
format: url.searchParams.get('format') ?? 'jpeg',
|
||||
};
|
||||
}
|
||||
|
||||
function numberParam(url: URL, name: string): number {
|
||||
const raw = url.searchParams.get(name);
|
||||
const parsed = raw ? Number.parseInt(raw, 10) : 0;
|
||||
return Number.isFinite(parsed) ? parsed : 0;
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
import { streamBodyToStorage } from '../storage/stream.js';
|
||||
import { recordImageMetadata } from '../storage/metadata.js';
|
||||
import { enqueueUploadMessage } from '../lib/queue.js';
|
||||
import { parseUploadRequest } from '../lib/request.js';
|
||||
|
||||
export interface UploadResult {
|
||||
key: string;
|
||||
bytes: number;
|
||||
contentType: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Entry point for an upload request: parse it, stream the body into object
|
||||
* storage, record the image metadata, then queue the follow-up work.
|
||||
*/
|
||||
export async function handleUploadRequest(request: Request): Promise<Response> {
|
||||
const parsed = await parseUploadRequest(request);
|
||||
if (!parsed.ok) {
|
||||
return new Response(JSON.stringify({ error: parsed.error }), { status: 400 });
|
||||
}
|
||||
|
||||
const stored = await streamBodyToStorage(parsed.body, parsed.key, parsed.contentType);
|
||||
const metadata = await recordImageMetadata(stored.key, {
|
||||
width: parsed.width,
|
||||
height: parsed.height,
|
||||
format: parsed.format,
|
||||
bytes: stored.bytes,
|
||||
});
|
||||
|
||||
await enqueueUploadMessage({
|
||||
key: stored.key,
|
||||
metadataId: metadata.id,
|
||||
contentType: stored.contentType,
|
||||
});
|
||||
|
||||
return new Response(JSON.stringify(summarizeUpload(stored, metadata.id)), {
|
||||
status: 201,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
});
|
||||
}
|
||||
|
||||
/** Shape the client sees back after a successful upload. */
|
||||
export function summarizeUpload(stored: UploadResult, metadataId: string) {
|
||||
return {
|
||||
key: stored.key,
|
||||
bytes: stored.bytes,
|
||||
contentType: stored.contentType,
|
||||
metadataId,
|
||||
};
|
||||
}
|
||||
|
||||
/** Reject uploads whose declared size exceeds the per-account ceiling. */
|
||||
export function isWithinUploadLimit(bytes: number, limit: number): boolean {
|
||||
if (!Number.isFinite(bytes) || bytes < 0) return false;
|
||||
return bytes <= limit;
|
||||
}
|
||||
|
||||
/** Delete-side counterpart, kept here so the route module is not a one-liner. */
|
||||
export async function handleDeleteRequest(request: Request, key: string): Promise<Response> {
|
||||
const parsed = await parseUploadRequest(request);
|
||||
if (!parsed.ok) {
|
||||
return new Response(JSON.stringify({ error: parsed.error }), { status: 400 });
|
||||
}
|
||||
await enqueueUploadMessage({ key, metadataId: '', contentType: 'application/x-delete' });
|
||||
return new Response(null, { status: 204 });
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import { openMetadataStore } from '../lib/bucket.js';
|
||||
|
||||
export interface ImageMetadataInput {
|
||||
width: number;
|
||||
height: number;
|
||||
format: string;
|
||||
bytes: number;
|
||||
}
|
||||
|
||||
export interface ImageMetadataRecord extends ImageMetadataInput {
|
||||
id: string;
|
||||
key: string;
|
||||
recordedAt: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record the image metadata for a stored object. Writes go to the metadata
|
||||
* store keyed by object key; the returned record carries the id the queue
|
||||
* message references.
|
||||
*/
|
||||
export async function recordImageMetadata(
|
||||
key: string,
|
||||
input: ImageMetadataInput,
|
||||
): Promise<ImageMetadataRecord> {
|
||||
const store = openMetadataStore();
|
||||
const record: ImageMetadataRecord = {
|
||||
...input,
|
||||
id: metadataIdFor(key, input),
|
||||
key,
|
||||
recordedAt: 0,
|
||||
};
|
||||
await store.put(record.id, JSON.stringify(record));
|
||||
return record;
|
||||
}
|
||||
|
||||
/** Deterministic id so a retried upload records the same metadata row. */
|
||||
export function metadataIdFor(key: string, input: ImageMetadataInput): string {
|
||||
return `${key}:${input.format}:${input.width}x${input.height}`;
|
||||
}
|
||||
|
||||
/** Read a metadata record back for the download and listing paths. */
|
||||
export async function loadImageMetadata(id: string): Promise<ImageMetadataRecord | null> {
|
||||
const store = openMetadataStore();
|
||||
const raw = await store.get(id);
|
||||
return raw ? (JSON.parse(raw) as ImageMetadataRecord) : null;
|
||||
}
|
||||
|
||||
/** Normalize a client-declared format string to the canonical set. */
|
||||
export function normalizeFormat(format: string): string {
|
||||
const lowered = format.trim().toLowerCase();
|
||||
if (lowered === 'jpg') return 'jpeg';
|
||||
if (lowered === 'tif') return 'tiff';
|
||||
return lowered;
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
import { openBucket } from '../lib/bucket.js';
|
||||
import type { StorageFailure, UploadTelemetry } from './types.js';
|
||||
|
||||
export interface StoredObject {
|
||||
key: string;
|
||||
bytes: number;
|
||||
contentType: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream a request body into object storage without buffering it in memory.
|
||||
* The body is piped through a counting transform so the byte total is known
|
||||
* by the time the put resolves.
|
||||
*/
|
||||
export async function streamBodyToStorage(
|
||||
body: ReadableStream<Uint8Array>,
|
||||
key: string,
|
||||
contentType: string,
|
||||
): Promise<StoredObject> {
|
||||
const bucket = openBucket();
|
||||
const counter = createByteCounter();
|
||||
const piped = body.pipeThrough(counter.transform, { preventClose: false });
|
||||
|
||||
await bucket.put(key, piped, { httpMetadata: { contentType } });
|
||||
|
||||
return { key, bytes: counter.total(), contentType };
|
||||
}
|
||||
|
||||
/**
|
||||
* A transform stream that counts the bytes flowing through it. Separated from
|
||||
* the pipe above so the byte total can be read after the stream settles.
|
||||
*/
|
||||
export function createByteCounter() {
|
||||
let total = 0;
|
||||
const transform = new TransformStream<Uint8Array, Uint8Array>({
|
||||
transform(chunk, controller) {
|
||||
total += chunk.byteLength;
|
||||
controller.enqueue(chunk);
|
||||
},
|
||||
});
|
||||
return { transform, total: () => total };
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a stored object back out of the bucket as a stream, for the download
|
||||
* path. Mirrors the upload side so both directions live in one module.
|
||||
*/
|
||||
export async function readObjectStream(key: string): Promise<ReadableStream<Uint8Array> | null> {
|
||||
const bucket = openBucket();
|
||||
const object = await bucket.get(key);
|
||||
if (!object) return null;
|
||||
return object.body;
|
||||
}
|
||||
|
||||
/** Timing/retry record for one stored object, handed to the metrics sink. */
|
||||
export function telemetryFor(stored: StoredObject, durationMs: number): UploadTelemetry {
|
||||
return { key: stored.key, bytes: stored.bytes, durationMs, retries: 0 };
|
||||
}
|
||||
|
||||
/** Describe a failed stage so the caller can report it without re-deriving it. */
|
||||
export function storageFailure(
|
||||
key: string,
|
||||
stage: StorageFailure['stage'],
|
||||
message: string,
|
||||
): StorageFailure {
|
||||
return { key, stage, message };
|
||||
}
|
||||
|
||||
/** Cap a stream at `limit` bytes, erroring out rather than storing an overrun. */
|
||||
export function limitStream(
|
||||
source: ReadableStream<Uint8Array>,
|
||||
limit: number,
|
||||
): ReadableStream<Uint8Array> {
|
||||
let seen = 0;
|
||||
const guard = new TransformStream<Uint8Array, Uint8Array>({
|
||||
transform(chunk, controller) {
|
||||
seen += chunk.byteLength;
|
||||
if (seen > limit) {
|
||||
controller.error(new Error(`upload exceeded ${limit} bytes`));
|
||||
return;
|
||||
}
|
||||
controller.enqueue(chunk);
|
||||
},
|
||||
});
|
||||
return source.pipeThrough(guard);
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
/**
|
||||
* Shared shapes for the storage layer. Declaration-only like the ambient files
|
||||
* under `types/` — but the modules that answer a flow question are typed BY it,
|
||||
* so it is part of that answer's structure rather than a global shim.
|
||||
*/
|
||||
|
||||
export interface UploadTelemetry {
|
||||
key: string;
|
||||
bytes: number;
|
||||
durationMs: number;
|
||||
retries: number;
|
||||
}
|
||||
|
||||
export interface StorageFailure {
|
||||
key: string;
|
||||
stage: 'parse' | 'stream' | 'metadata' | 'queue';
|
||||
message: string;
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
// Hand-maintained ambient declarations for the parts of the platform our
|
||||
// runtime exposes but the published typings do not cover yet. Edit freely —
|
||||
// nothing regenerates this file. Kept alongside the app so module augmentation
|
||||
// and the global shims live in one place.
|
||||
|
||||
declare global {
|
||||
interface UploadStorage {
|
||||
put(
|
||||
key: string,
|
||||
body: ReadableStream<Uint8Array>,
|
||||
options?: UploadPutOptions,
|
||||
): Promise<StoredUploadObject>;
|
||||
get(key: string): Promise<StoredUploadObject | null>;
|
||||
head(key: string): Promise<StoredUploadHead | null>;
|
||||
delete(key: string | string[]): Promise<void>;
|
||||
list(options?: UploadListOptions): Promise<UploadListResult>;
|
||||
}
|
||||
|
||||
interface StoredUploadObject {
|
||||
readonly key: string;
|
||||
readonly size: number;
|
||||
readonly etag: string;
|
||||
readonly uploaded: Date;
|
||||
readonly body: ReadableStream<Uint8Array>;
|
||||
readonly contentType: string;
|
||||
readonly metadata?: ImageMetadataShim;
|
||||
arrayBuffer(): Promise<ArrayBuffer>;
|
||||
text(): Promise<string>;
|
||||
json<T>(): Promise<T>;
|
||||
}
|
||||
|
||||
interface StoredUploadHead {
|
||||
readonly key: string;
|
||||
readonly size: number;
|
||||
readonly etag: string;
|
||||
readonly uploaded: Date;
|
||||
readonly contentType: string;
|
||||
}
|
||||
|
||||
interface UploadPutOptions {
|
||||
contentType?: string;
|
||||
cacheControl?: string;
|
||||
customMetadata?: Record<string, string>;
|
||||
checksum?: string;
|
||||
storageClass?: 'standard' | 'infrequent';
|
||||
}
|
||||
|
||||
interface UploadListOptions {
|
||||
prefix?: string;
|
||||
cursor?: string;
|
||||
limit?: number;
|
||||
delimiter?: string;
|
||||
include?: ('metadata' | 'contentType')[];
|
||||
}
|
||||
|
||||
interface UploadListResult {
|
||||
objects: StoredUploadHead[];
|
||||
truncated: boolean;
|
||||
cursor?: string;
|
||||
prefixes: string[];
|
||||
}
|
||||
|
||||
interface ImageMetadataShim {
|
||||
format: string;
|
||||
fileSize: number;
|
||||
width: number;
|
||||
height: number;
|
||||
orientation?: number;
|
||||
colorSpace?: string;
|
||||
}
|
||||
|
||||
interface MetadataRowShim {
|
||||
id: string;
|
||||
key: string;
|
||||
recordedAt: number;
|
||||
format: string;
|
||||
bytes: number;
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
|
||||
interface MetadataStoreShim {
|
||||
put(id: string, value: string, options?: MetadataPutOptions): Promise<void>;
|
||||
get(id: string): Promise<string | null>;
|
||||
getWithMetadata<T>(id: string): Promise<{ value: string | null; metadata: T | null }>;
|
||||
delete(id: string): Promise<void>;
|
||||
list(options?: MetadataListOptions): Promise<MetadataListResult>;
|
||||
}
|
||||
|
||||
interface MetadataPutOptions {
|
||||
expiration?: number;
|
||||
expirationTtl?: number;
|
||||
metadata?: unknown;
|
||||
}
|
||||
|
||||
interface MetadataListOptions {
|
||||
prefix?: string | null;
|
||||
cursor?: string | null;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
interface MetadataListResult {
|
||||
keys: { name: string; expiration?: number }[];
|
||||
list_complete: boolean;
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
interface UploadQueueShim<Body = unknown> {
|
||||
send(body: Body, options?: UploadSendOptions): Promise<void>;
|
||||
sendBatch(bodies: Iterable<UploadSendRequest<Body>>): Promise<void>;
|
||||
}
|
||||
|
||||
interface UploadSendOptions {
|
||||
contentType?: UploadContentType;
|
||||
delaySeconds?: number;
|
||||
}
|
||||
|
||||
type UploadContentType = 'text' | 'bytes' | 'json' | 'v8';
|
||||
|
||||
interface UploadSendRequest<Body = unknown> {
|
||||
body: Body;
|
||||
options?: UploadSendOptions;
|
||||
}
|
||||
|
||||
interface UploadMessageShim<Body = unknown> {
|
||||
readonly id: string;
|
||||
readonly timestamp: Date;
|
||||
readonly body: Body;
|
||||
readonly attempts: number;
|
||||
retry(options?: UploadRetryOptions): void;
|
||||
ack(): void;
|
||||
}
|
||||
|
||||
interface UploadRetryOptions {
|
||||
delaySeconds?: number;
|
||||
}
|
||||
|
||||
interface UploadMessageBatch<Body = unknown> {
|
||||
readonly messages: readonly UploadMessageShim<Body>[];
|
||||
readonly queue: string;
|
||||
retryAll(options?: UploadRetryOptions): void;
|
||||
ackAll(): void;
|
||||
}
|
||||
|
||||
interface StreamPipeOptionsShim {
|
||||
preventClose?: boolean;
|
||||
preventAbort?: boolean;
|
||||
preventCancel?: boolean;
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
interface ByteCounterShim {
|
||||
readonly transform: TransformStream<Uint8Array, Uint8Array>;
|
||||
total(): number;
|
||||
}
|
||||
|
||||
interface StreamLimitShim {
|
||||
readonly limit: number;
|
||||
readonly seen: number;
|
||||
exceeded(): boolean;
|
||||
}
|
||||
|
||||
interface RequestBodyShim {
|
||||
readonly body: ReadableStream<Uint8Array> | null;
|
||||
readonly bodyUsed: boolean;
|
||||
readonly headers: Headers;
|
||||
readonly url: string;
|
||||
arrayBuffer(): Promise<ArrayBuffer>;
|
||||
formData(): Promise<FormData>;
|
||||
blob(): Promise<Blob>;
|
||||
}
|
||||
|
||||
interface ParsedUploadShim {
|
||||
key: string;
|
||||
contentType: string;
|
||||
width: number;
|
||||
height: number;
|
||||
format: string;
|
||||
}
|
||||
|
||||
interface ImageTransformerShim {
|
||||
transform(transform: ImageTransformShim): ImageTransformerShim;
|
||||
output(options: ImageOutputShim): Promise<ImageResultShim>;
|
||||
}
|
||||
|
||||
interface ImageTransformShim {
|
||||
width?: number;
|
||||
height?: number;
|
||||
fit?: 'scale-down' | 'contain' | 'cover' | 'crop' | 'pad';
|
||||
rotate?: number;
|
||||
}
|
||||
|
||||
interface ImageOutputShim {
|
||||
format?: string;
|
||||
quality?: number;
|
||||
background?: string;
|
||||
}
|
||||
|
||||
interface ImageResultShim {
|
||||
contentType(): string;
|
||||
image(): ReadableStream<Uint8Array>;
|
||||
response(): Response;
|
||||
}
|
||||
|
||||
interface UploadEnvShim {
|
||||
UPLOADS: UploadStorage;
|
||||
METADATA: MetadataStoreShim;
|
||||
UPLOAD_QUEUE: UploadQueueShim<unknown>;
|
||||
}
|
||||
}
|
||||
|
||||
export {};
|
||||
@@ -0,0 +1,271 @@
|
||||
// Generated by Wrangler by running `wrangler types` (hash: 4f1c8ad2b90e)
|
||||
// Runtime types generated with workerd@1.20260701.0 2026-07-01 nodejs_compat
|
||||
declare namespace Cloudflare {
|
||||
interface Env {
|
||||
UPLOADS: R2Bucket;
|
||||
METADATA: KVNamespace;
|
||||
UPLOAD_QUEUE: Queue<UploadMessageBody>;
|
||||
IMAGES: ImagesBinding;
|
||||
}
|
||||
}
|
||||
|
||||
interface UploadMessageBody {
|
||||
key: string;
|
||||
metadataId: string;
|
||||
contentType: string;
|
||||
}
|
||||
|
||||
interface R2Bucket {
|
||||
head(key: string): Promise<R2Object | null>;
|
||||
get(key: string, options?: R2GetOptions): Promise<R2ObjectBody | null>;
|
||||
put(
|
||||
key: string,
|
||||
value: ReadableStream | ArrayBuffer | string | null,
|
||||
options?: R2PutOptions,
|
||||
): Promise<R2Object>;
|
||||
delete(keys: string | string[]): Promise<void>;
|
||||
list(options?: R2ListOptions): Promise<R2Objects>;
|
||||
createMultipartUpload(key: string, options?: R2MultipartOptions): Promise<R2MultipartUpload>;
|
||||
resumeMultipartUpload(key: string, uploadId: string): R2MultipartUpload;
|
||||
}
|
||||
|
||||
interface R2Object {
|
||||
readonly key: string;
|
||||
readonly version: string;
|
||||
readonly size: number;
|
||||
readonly etag: string;
|
||||
readonly httpEtag: string;
|
||||
readonly checksums: R2Checksums;
|
||||
readonly uploaded: Date;
|
||||
readonly httpMetadata?: R2HTTPMetadata;
|
||||
readonly customMetadata?: Record<string, string>;
|
||||
readonly range?: R2Range;
|
||||
readonly storageClass: string;
|
||||
writeHttpMetadata(headers: Headers): void;
|
||||
}
|
||||
|
||||
interface R2ObjectBody extends R2Object {
|
||||
get body(): ReadableStream;
|
||||
get bodyUsed(): boolean;
|
||||
arrayBuffer(): Promise<ArrayBuffer>;
|
||||
text(): Promise<string>;
|
||||
json<T>(): Promise<T>;
|
||||
blob(): Promise<Blob>;
|
||||
bytes(): Promise<Uint8Array>;
|
||||
}
|
||||
|
||||
interface R2GetOptions {
|
||||
onlyIf?: R2Conditional | Headers;
|
||||
range?: R2Range;
|
||||
ssecKey?: ArrayBuffer | string;
|
||||
}
|
||||
|
||||
interface R2PutOptions {
|
||||
onlyIf?: R2Conditional | Headers;
|
||||
httpMetadata?: R2HTTPMetadata | Headers;
|
||||
customMetadata?: Record<string, string>;
|
||||
md5?: ArrayBuffer | string;
|
||||
sha1?: ArrayBuffer | string;
|
||||
sha256?: ArrayBuffer | string;
|
||||
storageClass?: string;
|
||||
ssecKey?: ArrayBuffer | string;
|
||||
}
|
||||
|
||||
interface R2ListOptions {
|
||||
limit?: number;
|
||||
prefix?: string;
|
||||
cursor?: string;
|
||||
delimiter?: string;
|
||||
startAfter?: string;
|
||||
include?: ('httpMetadata' | 'customMetadata')[];
|
||||
}
|
||||
|
||||
interface R2Objects {
|
||||
objects: R2Object[];
|
||||
truncated: boolean;
|
||||
cursor?: string;
|
||||
delimitedPrefixes: string[];
|
||||
}
|
||||
|
||||
interface R2MultipartOptions {
|
||||
httpMetadata?: R2HTTPMetadata | Headers;
|
||||
customMetadata?: Record<string, string>;
|
||||
storageClass?: string;
|
||||
}
|
||||
|
||||
interface R2MultipartUpload {
|
||||
readonly key: string;
|
||||
readonly uploadId: string;
|
||||
uploadPart(
|
||||
partNumber: number,
|
||||
value: ReadableStream | ArrayBuffer | string | Blob,
|
||||
): Promise<R2UploadedPart>;
|
||||
abort(): Promise<void>;
|
||||
complete(uploadedParts: R2UploadedPart[]): Promise<R2Object>;
|
||||
}
|
||||
|
||||
interface R2UploadedPart {
|
||||
partNumber: number;
|
||||
etag: string;
|
||||
}
|
||||
|
||||
interface R2HTTPMetadata {
|
||||
contentType?: string;
|
||||
contentLanguage?: string;
|
||||
contentDisposition?: string;
|
||||
contentEncoding?: string;
|
||||
cacheControl?: string;
|
||||
cacheExpiry?: Date;
|
||||
}
|
||||
|
||||
interface R2Checksums {
|
||||
readonly md5?: ArrayBuffer;
|
||||
readonly sha1?: ArrayBuffer;
|
||||
readonly sha256?: ArrayBuffer;
|
||||
toJSON(): R2StringChecksums;
|
||||
}
|
||||
|
||||
interface R2StringChecksums {
|
||||
md5?: string;
|
||||
sha1?: string;
|
||||
sha256?: string;
|
||||
}
|
||||
|
||||
interface R2Conditional {
|
||||
etagMatches?: string;
|
||||
etagDoesNotMatch?: string;
|
||||
uploadedBefore?: Date;
|
||||
uploadedAfter?: Date;
|
||||
secondsGranularity?: boolean;
|
||||
}
|
||||
|
||||
interface R2Range {
|
||||
offset?: number;
|
||||
length?: number;
|
||||
suffix?: number;
|
||||
}
|
||||
|
||||
interface KVNamespace<Key extends string = string> {
|
||||
get(key: Key, options?: Partial<KVNamespaceGetOptions<undefined>>): Promise<string | null>;
|
||||
getWithMetadata<Metadata = unknown>(
|
||||
key: Key,
|
||||
options?: Partial<KVNamespaceGetOptions<undefined>>,
|
||||
): Promise<KVNamespaceGetWithMetadataResult<string, Metadata>>;
|
||||
put(
|
||||
key: Key,
|
||||
value: string | ArrayBuffer | ArrayBufferView | ReadableStream,
|
||||
options?: KVNamespacePutOptions,
|
||||
): Promise<void>;
|
||||
delete(key: Key): Promise<void>;
|
||||
list<Metadata = unknown>(
|
||||
options?: KVNamespaceListOptions,
|
||||
): Promise<KVNamespaceListResult<Metadata, Key>>;
|
||||
}
|
||||
|
||||
interface KVNamespaceGetOptions<Type> {
|
||||
type: Type;
|
||||
cacheTtl?: number;
|
||||
}
|
||||
|
||||
interface KVNamespacePutOptions {
|
||||
expiration?: number;
|
||||
expirationTtl?: number;
|
||||
metadata?: unknown | null;
|
||||
}
|
||||
|
||||
interface KVNamespaceListOptions {
|
||||
limit?: number;
|
||||
prefix?: string | null;
|
||||
cursor?: string | null;
|
||||
}
|
||||
|
||||
interface KVNamespaceListResult<Metadata, Key extends string = string> {
|
||||
keys: KVNamespaceListKey<Metadata, Key>[];
|
||||
list_complete: boolean;
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
interface KVNamespaceListKey<Metadata, Key extends string = string> {
|
||||
name: Key;
|
||||
expiration?: number;
|
||||
metadata?: Metadata;
|
||||
}
|
||||
|
||||
interface KVNamespaceGetWithMetadataResult<Value, Metadata> {
|
||||
value: Value | null;
|
||||
metadata: Metadata | null;
|
||||
cacheStatus: string | null;
|
||||
}
|
||||
|
||||
interface Queue<Body = unknown> {
|
||||
send(message: Body, options?: QueueSendOptions): Promise<void>;
|
||||
sendBatch(messages: Iterable<MessageSendRequest<Body>>): Promise<void>;
|
||||
}
|
||||
|
||||
interface QueueSendOptions {
|
||||
contentType?: QueueContentType;
|
||||
delaySeconds?: number;
|
||||
}
|
||||
|
||||
type QueueContentType = 'text' | 'bytes' | 'json' | 'v8';
|
||||
|
||||
interface MessageSendRequest<Body = unknown> {
|
||||
body: Body;
|
||||
options?: QueueSendOptions;
|
||||
}
|
||||
|
||||
interface Message<Body = unknown> {
|
||||
readonly id: string;
|
||||
readonly timestamp: Date;
|
||||
readonly body: Body;
|
||||
readonly attempts: number;
|
||||
retry(options?: QueueRetryOptions): void;
|
||||
ack(): void;
|
||||
}
|
||||
|
||||
interface QueueRetryOptions {
|
||||
delaySeconds?: number;
|
||||
}
|
||||
|
||||
interface MessageBatch<Body = unknown> {
|
||||
readonly messages: readonly Message<Body>[];
|
||||
readonly queue: string;
|
||||
retryAll(options?: QueueRetryOptions): void;
|
||||
ackAll(): void;
|
||||
}
|
||||
|
||||
interface ImagesBinding {
|
||||
info(stream: ReadableStream<Uint8Array>): Promise<ImageMetadata>;
|
||||
input(stream: ReadableStream<Uint8Array>): ImageTransformer;
|
||||
}
|
||||
|
||||
interface ImageMetadata {
|
||||
format: string;
|
||||
fileSize: number;
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
|
||||
interface ImageTransformer {
|
||||
transform(transform: ImageTransform): ImageTransformer;
|
||||
output(options: ImageOutputOptions): Promise<ImageTransformationResult>;
|
||||
}
|
||||
|
||||
interface ImageTransform {
|
||||
width?: number;
|
||||
height?: number;
|
||||
fit?: 'scale-down' | 'contain' | 'cover' | 'crop' | 'pad';
|
||||
rotate?: number;
|
||||
}
|
||||
|
||||
interface ImageOutputOptions {
|
||||
format?: string;
|
||||
quality?: number;
|
||||
background?: string;
|
||||
}
|
||||
|
||||
interface ImageTransformationResult {
|
||||
contentType(): string;
|
||||
image(): ReadableStream<Uint8Array>;
|
||||
response(): Response;
|
||||
}
|
||||
Reference in New Issue
Block a user