Real-world validation of #1575 on indexes damaged by the released v1.5.0
binary surfaced both of these.
getUnresolvedReferencesByFiles chunked its INPUT under SQLite's parameter
limit but appended each chunk's RESULT rows with a spread — every row
becomes a call argument, so a dense recovery sync (the #1541 self-heal
re-indexing 919 files produced 234,440 rows) exceeded V8's argument limit
and killed resolution mid-sync with "Maximum call stack size exceeded",
leaving the graph 226k edges short until another sync resumed the orphans
(and that sweep resolves measurably worse than the batched path — see the
follow-up issue). The failed-ref retry loader had the identical pattern on
unbounded result rows. Both append with a loop now (#1558).
The #1575 stripped-salvage warning also never rendered: init's summary
prints only index_partial warnings and counts only hard errors, so a run
with salvaged files still read as fully clean — and with no hard errors the
detail wasn't written to errors.log either. Salvage entries now carry code
'salvaged_stripped', the summary prints a visible warning naming the files,
and errors.log is written for salvage-only runs.
Validated on real corpora with full-graph dumps: healthy-path inits stay
byte-identical to the pre-#1575 baseline (cpython Lib, Alamofire, with a
determinism control); a realistically-damaged index (41 wiped + 5 missing
files, damage generated by the released binary) heals in one plain sync to
identical per-file counts and an edge set within the normal incremental
residual; pathological mass damage (52% of the repo) completes without
crashing. New regression test reproduces the RangeError on the old code
with 200k pending refs.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The parse-pool workers return kernel-language extractions as an undecoded
buffer transport (nodes/edges EMPTY, tables in kernelBuffers). indexAll's
main loop decodes them (or hands the buffers to the store worker), but its
two retry passes — plain retry and the comments-stripped last resort —
stored the transport as-is: the storage gate passed via errors.length === 0,
zero nodes were inserted, and the files row was written with node_count = 0
while the original error was spliced out of the summary. Any worker
crash/timeout whose in-flight file was a kernel-routed language permanently
recorded that file as "(0 symbols)" — silently, and immune to later syncs
because the stored hash matches the on-disk bytes (#1541; v1.4.1 predates
the kernel path, which is why it was unaffected).
- Both retry passes now materialize kernel results before the gate, store,
counters, and log lines.
- storeExtractionResult materializes at entry as defense-in-depth, so no
storage path can persist an undecoded transport again.
- Zero-node rows on symbol-bearing languages (only the wipe produces these —
every real extraction stores at least the file node) are dropped during
full-reconcile sync and indexAll so already-affected files re-index
automatically after upgrading. Scoped watcher syncs leave rows outside
their scope untouched.
- The comments-stripped salvage now downgrades the failure to a visible
warning instead of erasing it: the recovered result can be incomplete, and
reporting clean success made a fresh index quietly disagree with a later
per-file re-parse of the same bytes (#1565's init-vs-sync divergence).
Repro (released 1.5.0): CODEGRAPH_PARSE_TIMEOUT_MS=1 codegraph init on any
Python project → "Retry OK: <file> (0 nodes)" and permanent
"(python, 0 symbols)" rows. Fixed build stores real symbols under the same
forcing, and heals rows wiped by prior runs.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Previously, `codegraph_explore` queries explicitly naming files by path (e.g., `src/routes/m/projects/[id]/runs/[runId]/+page.svelte`) were shredded. Bracketed path segments exploded into "named symbol" seeds, and FTS on fragments like `page` or `runs` admitted every sibling file, starving the user's intended target.
This change introduces:
- **Query path pinning:** File paths named in a query are now resolved against the index, "pinned," and stripped from the query. Pinned files are guaranteed inclusion, top ranking, and fair allocation. Unresolvable path-like spans are reported.
- **Segment vocabulary supplement:** Natural language query terms (e.g., "auto-scroll to bottom") can now reach camelCase identifiers (e.g., `pinFeedIfNearBottom`, `feedAtBottom`) by matching against their constituent segments.
- **Variable seeding:** `variable` and `constant` node kinds are now included in identifier seeding, improving recall for `$state`-style variables common in frameworks like Svelte.
Resolves the CHANGELOG conflict — main and this branch each prepended a
bullet to [Unreleased] > Fixes; both are kept. Everything else auto-merged,
including src/mcp/tools.ts, which main reworked heavily for the explore
allocation/displacement work (CG-28/31/36/38) while this branch added the
`union` kind to its container sets.
Verified on the merged tree with the native kernel built: 3070 passed,
9 skipped, 0 failed.
Making unions first-class nodes leaves the third loss in #1515 open:
interfaceOverrideEdges enumerates its concrete side as ['class','struct'],
so a union implementor is skipped even though it now has a real node and a
real `implements` edge. "Who implements this trait" then answers wrongly
rather than incompletely — the struct beside it bridges and the union does
not.
Add 'union' to that tuple, plus a regression test that pins the Rust
trait -> union-impl hop (the struct implementor is the control proving the
synthesizer ran). Verified the test fails on the union assertion alone
before this change.
No EXTRACTION_VERSION bump: main is already at 25 against v1.5.0's 24, so
existing indexes are flagged stale for the next release regardless, and
over-bumping is what turns the re-index hint into noise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Recut of #678 against the current instructions — the original predated the
explore-first rewrite and conflicted in both files it touched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`codegraph_explore` never returned `queueMessage` (L1087) or
`flushQueuedMessages` (L1102) from a 1,414-line file, on a symbol bag or a
prose question, even with that file at rank #1 holding 67% of the envelope —
the agent got a same-stem `QueuedMessage` interface at L70 and had to Read the
file for the functions it had named. Pre-existing at every build including
pre-epic (controlled bisect, index held fixed).
Two independent causes:
1. `buildFlowFromNamedSymbols` returns the Flow prose AND the set of node ids
the agent named — and the latter is the whole guarantee, since it injects a
named def into its file's cluster ranges at importance 9. Its bail-outs
returned EMPTY, zeroing the identity whenever there was nothing to PRINT.
Two sibling closures that never call each other produce no chain, no synth
hop and no boundary, so both defs lost importance 9 and the file rendered
from its head. `identityOnly()` now separates the two, gated on
shape-precise tokens so a prose word that exact-matches a callable cannot
promote itself.
2. The ceiling trim filled in SOURCE order, so an over-ceiling render always
dropped the END of a large file first. The shrink HAD kept both symbols
(1022-1121); the trim cut back to 839. `windowToCeiling` now takes the
spine call site plus every importance>=9 member as focus lines, tries the
full ceiling first, and splits the held-back reserve evenly with
carry-forward — greedy-in-source-order reproduced the bug one level down.
The shrink's loose size estimate is left alone deliberately, and the comment
now says why: making it exact was built and measured WORSE (it stops at the
last member that fits whole and the released bytes carry forward to
lower-ranked files, costing payroll-go's `s.store.Upsert`). `bound()` clamps to
the ceiling anyway, so the slack costs no bytes; it just must not pick the
survivors, which is what the trim now handles.
The measurement gap this closes: every existing probe is aggregate — envelope
share, per-file spend, source totals, file counts — and all are green on a
response that returns 25K from the right file and omits the named function.
`probe-named-symbol.mjs` checks the definition LINE against the response's
rendered lines, per symbol.
Suite envelope byte-identical to main on all six repos; probe-allocation 4/4,
no starvation flags; 180 files / 2,997 tests green. Fixture: 7/7 fail on main,
7/7 pass here, deterministic over 4 runs per arm.
A file's ranked clusters were all-or-nothing past the first one: the top-ranked
cluster was taken (shrunk to fit when it had to be) and every cluster below it
was rendered whole, then either fit the remainder or was dropped entirely. On a
file whose top-ranked cluster is TRIVIAL that discards the answer — django's
`db/models/sql/query.py` kept a 22-line glue cluster and dropped the 624-line
`Query` body, spending 1,923 of a 7,947 reservation; okhttp's
`RealInterceptorChain.kt` did the same behind its import header.
The response stayed full, which is why this was invisible: the unspent
reservation carried forward exactly as designed and a file scoring a fifth as
much took the bytes.
Two sites, the same rule — hold the remainder while it is still worth a section
(CG-26's between-FILES lesson, applied between CLUSTERS):
- selection now shrinks a later cluster into what is left of the file's budget,
by the same whole-member rule the first cluster already used;
- the ceiling trim re-renders the weakest cluster into the room that remains
before dropping it. On excalidraw's `typeChecks.ts` the section-cost estimate
missed by 13 chars and a 1,512-char cluster — the file's highest-SCORING one —
was thrown away to pay for it.
Cluster RANKING is untouched: measured, both real cases lost on `maxImportance`,
not on the density tiebreak the issue suspected, and density-first is what keeps
Alamofire's `Session.swift` from burying its methods under the property list.
Suite (6 repos, clean-rebuilt indexes): all 8 starvation flags cleared,
+1,012 source chars net. django's `sql/query.py` 1,923 -> 10,082 of 7,947,
okhttp's `RealInterceptorChain.kt` 1,474 -> 6,038 of 6,058, gin's
`routergroup.go` 3,273 -> 5,632. okhttp trades its rank-6 file (score 21) for
+7,196 chars in the two files that answer the question.
Ships two fixtures pulling in opposite directions (`starved-cluster-ts` and
`dense-header-ts`), a `spendShareAtLeast` gate in probe-allocation, and
probe-file-spend.mjs — a standing per-file reservation-vs-delivered sweep.
A file that declares nothing but types and that nothing in the index depends
on — a hand-written ambient `.d.ts` of global shims, vendored typings, module
augmentation — cannot answer a flow question: no bodies, no call edges, no
behaviour, nothing typed by it. But the identifiers it declares are exactly the
generic ones a prose question uses (`Body`, `Message`, `ImageMetadata`,
`ReadableStream`), so on term overlap it out-scored the implementation. Measured
on the new fixture: rank #1 and 51% of delivered source, with the flow's own
entry file pushed out of the response entirely.
Measured first, per the issue: the Wrangler `worker-configuration.d.ts` that
opened this is already handled by CG-25's banner detection, worth 15-46 points
of envelope share across four flow queries. CG-25 credited; only the un-bannered
case needed anything.
`rankPenalty` now multiplies score and graph mass by 0.5 for such files, taken
as the STRONGER of it and the generated penalty rather than multiplied — one
property two signals see must not be charged twice. Detection is structural, not
by extension, and four conditions deep. Two of them were forced by measurement:
requiring every symbol to be type-level takes the corpus flag rate from 1-18%
(which swept in Kotlin sealed classes, Rust mod.rs re-exports and django's
locale tables) down to 0-4%; requiring that nothing depends on the file
separates an ambient shim from a working types module, and without it the rule
demoted displacement-ts's pipeline `types.ts` and broke the CG-31 gate.
A query that NAMES a declared type is exempt, so a question about a type still
reaches its declaration at full weight. Precise tokens only, so "…the file
body…" cannot exempt a `Body` interface it never meant to name; this needs its
own set because `namedSeedIds` is callable-only and a type never becomes one.
Regression evidence in docs/benchmarks/explore-declaration-only-cg28.md:
6-repo envelope sweep byte-identical against a clean baseline build, zero
ambient files reach the candidate set on VS Code across five queries, corpus
flag rate 0-0.74%, both allocation fixtures PASS, full suite 2,978 green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A generated Cloudflare Wrangler ambient-types file was not flagged generated, so
it ranked with no penalty and competed with hand-written source on generic token
overlap. The banner shape it uses — "Generated by <tool> by running <command>" —
matched none of the existing content patterns, all of which require DO NOT EDIT,
a standalone @generated, or the "auto(matically) generated by" phrasings.
Precision is held by requiring TWO 'by' clauses: the banner must name a tool and
then say 'by running'. Ordinary prose ("the report is generated by running the
nightly job") has only one and does not match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cloudflare Wrangler's `worker-configuration.d.ts` (~12k lines of ambient
types) carried no banner any GENERATED_CONTENT_PATTERNS entry matched:
every existing marker requires `DO NOT EDIT`, a standalone `@generated`,
`<auto-generated>`, or the literal `automatically/auto-generated by`
phrasings. Wrangler emits a bare `Generated by Wrangler by running
`wrangler types``, so the file ranked with pen 1.00 and won 79.4% of an
explore envelope on generic token overlap alone (CG-24).
The discriminator is the reproduction instruction, not the word
"generated": the banner must name a tool AND then say `by running`, i.e.
two separate "by" clauses. That keeps prose out — "the nightly summary is
generated by running the ETL job" has only one — while catching every
CLI-driven emitter that tells you how to regenerate.
Precision swept over 441,856 files across the whole local source tree: 5
hits, all genuine Wrangler output, no false positives.
Isolated before/after on the CG-24 repro (same query, same index, only
the `files.generated` flag differing):
before pen 1.00 score 115.0 share 79.4% 3 files rendered
after pen 0.30 score 35.4 share 21.1% 4 files rendered
The new pattern stays in the existing table position, below the header
window the detector scans, so the module still does not classify itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A live, auto-synced index did not converge to a clean rebuild of the same
tree — 4.3% of distinct edges wrong in both directions on this repo's own
index, overwhelmingly `calls`, which is what flow queries traverse and what
explore's file ranking weights. Silent: nothing warned, and the symptom read
as "codegraph isn't very good" rather than "this index needs rebuilding."
Two causes, and the fix needed both. Resolution binds a reference to one of
the same-named definitions PROJECT-WIDE, so a definition appearing or
vanishing changes the correct answer for references in files the sync never
touches — and those references resolved successfully once, which deletes
their unresolved_refs row, leaving nothing to revisit them with (#1240's
retry only revisits refs parked as failed). Separately, when nothing
disambiguated the candidates the winner came down to rowid, i.e. the order
files happened to be WRITTEN, which differs between a scan-order full index
and a sync that appends each file as it changes. That second one is why
re-resolution alone could not converge: re-resolving against the identical
graph still picked a different candidate.
So getNodesByName now orders by (file_path, start_line) — a property of the
code, not of the write order — and sync computes a definitionDelta and
re-opens the resolution edges whose answer it may have invalidated,
re-inserting each as the reference that created it for the orphan sweep to
bind against the post-sync graph.
The delta compares `file\0name` pairs per file rather than one name set over
the batch: a commit that adds `collect` to a new file while an unrelated
changed file already defines `collect` cancels out of a batch-wide set, and
that miss was the largest residual class in the first measurement.
Conservative where the failure modes are asymmetric — a wrong deletion is a
permanent edge loss, a missed rebind is only residual drift. Edges without a
refName stamp are never touched (nothing to restore them from), sources the
sync already re-extracted are skipped, and a per-name ceiling declines the
generic names. Edges are deleted before the sweep re-inserts, since
INSERT OR IGNORE against idx_edges_identity would otherwise keep both rows
when a reference rebinds elsewhere.
Replaying real commits of this repo through sync, then diffing against a
rebuild: 16 commits 48 -> 0; 80 commits 1,634 -> 361, with the actively
misleading direction (stale edges the index keeps asserting) 671 -> 2.
Index and sync wall-clock are unchanged; the ORDER BY costs 18% per uncached
name lookup, which never reaches wall-clock because the resolver memoizes it.
The 357-edge residual at 80 commits is one pre-existing class: refs to
generic names (`push`, `join`) parked above #1240's per-name retry ceiling,
which a rebuild resolves into cross-language garbage — a TS test file
"calling" an R method. Converging there would mean manufacturing wrong edges,
so it is left alone. And no drift metric in `codegraph status`: it cannot be
computed without the rebuild it would be recommending, and a proxy would fire
on that residual and train users to ignore it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The invariant this closes: every admitted file receives at least its
reservation before any file draws on carry-forward slack. CG-30 bounded an
oversize cluster member and CG-31 gave the cluster path a displacement guard;
three holes were left, and each one starved a file that had been admitted,
reserved and — in the worst case — rendered.
1. The whole-file arms had no displacement guard. BUY's fit test read
`renderCeiling - totalChars` (everyone's room) while its source-space
sibling refused the same trade, and GRACE was not fit-tested at all.
okhttp's CallServerInterceptor.kt shipped 8,499 chars on a 5,964 funded
ceiling and the rank-6 file below it delivered nothing. Both arms now test
the render they actually produce against `fundedHeadroom`, and a whole
render that does not fit falls through to clustering instead of skipping
the file.
2. Every section was charged a flat 200 chars while a real header runs
300-500. The loop believed it had room it did not have — okhttp allocated
26,601 against a 24,400 ceiling — so the final truncation threw a
fully-rendered section away. Sections are charged their real cost now, the
owed-below arithmetic uses a per-file overhead estimated from the file's own
symbols, and a marginal overrun trims the weakest cluster (or windows the
last one into the room that is left) rather than skipping the file over a
rounding difference.
3. `owedPayableBelow` held all-or-nothing. When the last admitted file's FULL
reservation no longer fit, nothing was held for it: on the precise-query
fixture the rank-5 file took 4,134 chars against a 2,948 reservation while
rank 6 — admitted, reserved 2,539 — was left 4 chars and skipped. It now
holds the remainder while that remainder is still worth a section
(MIN_CHARS).
And the epilogue is budgeted instead of discarded. The flat 600-char margin was
neither the epilogue's size (1,064 gin, 1,788 django, 2,231 excalidraw) nor a
bound on it, so four of six suite repos shipped with no pointer list and no
reminders at all. The loop now reserves the epilogue's FLOOR — the one line
that says an uncovered area exists, plus a pointer for every file whose bytes
were deliberately withheld (CG-12) — and the rest is fitted to the room that
actually remains, in priority order, entry by entry. Sized from the real
strings; no constant was swept against the suite.
Deterministic, same clean-rebuilt indexes, baseline = CG-31 tip:
repo base source new source files ceiling
django 20,791 20,878 6 -> 6 was discarding its epilogue
tokio 21,521 21,607 5 -> 5 was discarding its epilogue
okhttp 19,034 18,870 5 -> 6 +1 file delivered
excalidraw 20,204 19,652 8 -> 8 keeps its pointer list
gin 10,776 10,776 4 -> 4 byte-identical
alamofire 11,662 11,662 2 -> 2 byte-identical
No repo truncates any more and none loses a file. okhttp and excalidraw trade
164 and 552 source chars on their LAST-ranked file for the pointer list naming
what the response could not cover — bytes the CG-31 tip only had because it
over-filled a ceiling it mis-measured and then discarded the epilogue whole.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The '⚠ changed on disk after the last index sync' banner is an honesty claim
about source we DID render — line refs elsewhere in the response may be
shifted — not a note about the response. Drawing the epilogue boundary after
it means the size cut can never be what silences it. Suite numbers unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two corrections found by measuring the first cut of the guard against the
6-repo suite. The first version held back the FULL sum of the reservations
below a file. On django that took 2,319 chars off a file the agent receives
and handed them to a section the hard ceiling then threw away — the guard's
own failure mode, one layer down. tokio lost 1,298 the same way.
1. `owedPayableBelow` — hold back only the prefix of what is owed below that
the response can still PAY, in rank order. A promise the ceiling cannot
reach is not a claim on this file's bytes.
2. The final truncation now spends the EPILOGUE before it spends a rendered
file section. It used to cut at the last section header, dropping that
section AND the trailing notes; dropping the notes alone is almost always
enough. A section is source the agent otherwise has to Read; the epilogue
is a pointer list and two reminders, and the note that replaces it carries
the "explore these names" instruction forward.
Also count `flow.text` in `totalChars`. It is prepended to `lines` to make the
final output, so the render loop always spent against a ceiling it was ~2K
under on symbol-bag queries.
Deterministic, same clean-rebuilt indexes, both builds (baseline = CG-30 tip):
repo base source new source files
django 20,033 20,791 5 trunc -> 6
excalidraw 18,776 20,204 7 trunc -> 8
okhttp 15,628 19,034 4 trunc -> 5
tokio 20,340 21,521 4 trunc -> 5
gin 10,776 10,776 4 -> 4 (byte-identical)
alamofire 11,662 11,662 2 -> 2 (byte-identical)
No repo delivers less; four stop truncating. `funded` in the diagnostic now
reports the render CEILING the guard allows, which is what every render path
is actually bounded by.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Carry-forward slack let a file spend what the files ABOVE it left on the
table. Nothing held back what was promised BELOW it. The whole-file BUY arm
has always refused that trade (`owedBelow`); the cluster path read `headroom`
— what is left before the hard ceiling — instead of what is still owed, so
`fileBudget` and `SPINE_CEILING` could pay a 1.5x overshoot out of another
file's reservation.
`fundedHeadroom` is the same inequality in the units the cluster path spends
in: source PLUS the per-section overhead each unreached file will charge.
Floored at the file's own reservation — a kept promise is not a displacement —
and it is <= `headroom` by construction, so it is the only bound the three
render sites need. The skeleton path's `bodyCap` takes it too.
Measured on `__tests__/fixtures/displacement-ts` (a 4-stage pipeline padded
past 500 files, where the 24K envelope genuinely saturates the 24.4K render
ceiling):
before ingest.ts emitted 9,301 on a 6,289 spendable, then lost the whole
section to the final ceiling — 0 delivered. types.ts and sink.ts
skipped `budget-whole-file`. 3 of 6 admitted files delivered.
after ingest.ts bounded to the 4,913 actually free. 6 of 6 delivered,
envelope 14,908 -> 22,066.
The self-query allocation fixture flips back to PASS with it, on a clean full
rebuild of this repo's index (CG-33). Its `afterCG30` verdict blamed an
over-RESERVED incidental file; the reservation was identical in both arms —
the file was over-SPENDING. Recorded honestly in `afterCG31`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d652c14 was committed while an ab-new-vs-baseline run had the engine checked out
at the BASELINE ref — that harness swaps src/ files mid-run and restores them on
exit — so it captured main's tools.ts and explore-diagnostics.ts and silently
undid 765c06a. Restored from 765c06a, with d652c14's doc tweak re-applied.
The A/B runs started after that commit are void with it (their "changed:" line
lists only explore-diagnostics.ts, i.e. both arms ran the same retrieval code)
and are re-run rather than reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The self-query allocation probe fixture's delivered-share gates now fail. The
cause is not the new bound: allocation is unchanged between arms (parse-run.mjs
32.3% vs 33.9% on main) and tools.ts delivers the same 8,282 chars in both. What
changed is that the incidental file now DELIVERS — on main its whole section was
cut by the hard-ceiling truncation, so the fixture passed on truncation luck.
Every file on this repo obeys the new bound (max 1.40x of spendable).
Recorded as `afterCG30` with that reasoning rather than tuning the bound to
restore the pass. The over-reservation it exposes is epic CG-24's subject.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A `union` declaration produced no symbol at all in any of the four
languages that have one. The type never entered the graph, and neither
did anything attached to it — in Rust every `impl Trait for MyUnion`
lost its edge, and the impl's methods were left with a qualifiedName
pointing at a type the graph did not contain.
`union_specifier` / `union_item` were absent from the extraction layer
entirely: no `<x>Types` list on the TS side, no dispatch branch in
either kernel walker.
They join `structTypes` (kind `struct` — NodeKind has no `union`), which
is the extension point the table-driven extractors already provide. The
body guard in extractStruct is untouched, so a bodiless `union U;` stays
a forward declaration and is still skipped, exactly like `struct U;`.
`resolveTypeAliasKind` accepts `union_specifier` too, so
`typedef union { … } N;` takes the typedef's name the way
`typedef struct { … } N;` already did. Without it the anonymous union
body would mint a second `<anonymous>` node beside the alias.
Both walkers change together so kernel<->wasm parity holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
shrinkCluster keeps an oversize cluster's highest-importance member WHOLE on
purpose — an empty file section sends the agent to Read, the outcome explore
exists to prevent. What it lacked was a bound, and "never empty" quietly meant
"never bounded": on the reporting repo one file emitted 22,376 chars against a
9,181-char reservation (2.44x), past both the per-file budget and the spine
ceiling. That overshoot is what collapses `headroom` for every file below it.
The same rule has a second face. When the top member is bigger than the whole
response ceiling, the file does not overshoot — it is dropped entirely at the
renderCeiling check, so the agent gets nothing for a file it named.
renderCluster now takes a ceiling (1.5x what the file may spend — the same
multiple SPINE_CEILING already draws, and never below the cap, so a cluster
that fits is untouched). Past it the member is WINDOWED on whole lines rather
than emitted whole or dropped: leading window plus, on a flow cluster, a window
on the spine's call site. A partial window shorter than 12 lines is dropped
instead — a sliver in the session record forces the next call's dedup to shred
the block around it or re-send it — unless nothing else was emitted, where the
never-empty floor wins.
Measured on the new fixture, pre-fix vs post-fix:
monthly.ts 12,391 chars on a 3,334 budget (3.7x) → 4,941 (1.48x)
quarterly.ts dropped, no headroom left → 4,004 delivered
Also: the diagnostic now reports `spendable` (reservation + inherited slack)
alongside `reserved`. Every render bound reads the former, so reporting only
the latter makes an ordinary carry-forward read as a file spending over budget
— and it made the overshoot this issue is about unmeasurable. A windowed file
is now flagged `clipped` too, instead of presenting a window as the whole file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A later explore call re-served whatever it re-ranked, so on the #1500 report
the 4th call spent its envelope on the spine the 1st call had already
delivered. CG-17 recorded what was served; this acts on it.
What a withheld span becomes is the whole design: a POINTER, never a silence.
An insufficient-feeling response is what sends an agent to Read, and one or
two of those early in a session teach it to abandon codegraph — so the
replacement names the file, the symbols and the line spans, and says both
that the source came from THIS conversation and that the file has not changed
since.
- Content fingerprint, not the drift flag, gates it. They answer different
questions: two calls inside one drift window served the same current bytes,
while a file edited AND re-synced between calls is never "stale" and yet
the agent's copy is now wrong. An edited file re-emits in full.
- Only a covered run of >= 8 lines is replaced, and a remainder under 160
chars folds into the pointer. Below those the pointer costs more than the
source and the block reads as shredded — a fence holding `228\t` is a
broken-looking response, which is the expensive failure.
- The reclaimed bytes go to files the agent has NOT seen, two ways: a smaller
`sourceSpent` hands slack down CG-21's carry-forward pool, and a fully
back-referenced file gives up its maxFiles slot the way a cliffed one does.
Within a file, the cluster shrink now reads the DEDUPED length, so it never
drops new symbols to make room for source it isn't sending.
- If dedup suppresses everything and nothing new takes its place, the top
suppressed file is spliced back in whole. An all-pointer response is the
shape that reads as "codegraph found nothing"; one re-served file is the
cheaper mistake.
Kill switch: CODEGRAPH_EXPLORE_DEDUP=0.
Explore answers every call as if it were the first: no record of the files
and line ranges it already sent, so a 4th call re-serves the 1st call's
spine and the tier call budget can only be asked for, never enforced.
Track it per MCP session, per resolved project root — files, coalesced line
ranges, bytes, and the call's index in the session. Nothing reads it yet:
the response is byte-identical, which the suite pins against an untracked
call of the same query.
The daemon shares ONE ToolHandler and a pool of worker threads across every
connected client, so the state can live neither on the handler nor in a
worker. It lives on MCPSession and is handed to execute() per call; the
session's view rides DOWN on the args and the call's emission rides BACK on
the ToolResult, both as plain properties so they survive the structured
clone to and from a worker. execute() records the emission on the main
thread and deletes it unconditionally — including for callers that track
nothing, like the CLI — so it can never reach the wire. A view a client
spells itself is discarded rather than trusted.
Ranges are reported by the render loop itself (buildSection now returns the
spans it slices alongside the text), and only files that survive the final
hard-ceiling truncation are recorded. Where a bound forces a choice the
record keeps FEWER ranges than were emitted: under-reporting re-serves
something the agent has, over-reporting withholds source it never saw and
costs a Read.
Every bound caps detail only — callCount keeps counting past eviction, so
CG-19's decay can't reset itself every 8 calls.
Found reviewing the CG-21 fix rather than by a failing test, and it is the
same defect inverted. A whole render that overruns `renderCeiling` is
skipped ENTIRELY (the branch refuses to slice a file mid-method), so a buy
that is approved by the funding pool but refused by the ceiling trades a
clustered section for NO section.
Only reachable on the 24K tiers. The funding line is
`reservedTotal + 0.15 * envelope` — ~27.2K when a medium repo saturates —
while `renderCeiling` is `min(1.5 * envelope, 25000) - 600` = 24.4K, so
funding can approve ~2.8K the ceiling then refuses. At 13K the line is
~14.4K against an 18.9K ceiling and the two cannot cross, which is why the
small-tier fixtures cannot see it.
Failing the test in `buysWhole` drops through to the cluster path, which is
bounded by `headroom` and always renders something. The GRACE arm is left
alone deliberately: a file within a sliver of its reservation that still
does not fit is genuinely at the end of a full response, and that predates
this epic.
Verified inert on the three A/B repos, so the agent A/B measured the same
behaviour: excalidraw byte-identical on 3 queries, client-go byte-identical
on 3 queries, express reproducer unchanged at 15,984. Full suite green
(2,867 passed; one unrelated fs.watch timing flake that passes 30/30 in
isolation).
A file whose proportional reservation lands below its own size stopped
rendering whole, and the fallback cluster render could leave most of that
reservation unspent — the bytes were neither delivered nor redistributed.
Found by CG-15's agent A/B on the express control: `lib/utils.js`, the
top-ranked file, was reserved 3,870 chars and spent 583. The whole-file
grace bound (reservation + a sliver) sat just under the file's 5,293
bytes, so the whole render was declined and three matched symbols became a
stub. The source envelope fell 13,849 -> 9,241 against an UNCHANGED
budget, and the agent Read the file back four times in 1 run of 3.
Two levers, per the task's candidate fixes:
- WHOLE_FILE_BUY_FRACTION: a reservation that already covers 60% of a
file buys the whole file. Funded from ONE shared overshoot pool sized
at 15% of the envelope, spent in rank order. Per-file funding is the
version that fails, and it fails the same way the bug does — the merit
test is a ratio, so several files qualify at once and N independent
overshoots push the last section past the render ceiling. Measured on
the payroll fixture: three files bought whole and `payslip_builder.go`
was dropped entirely. A dropped section is strictly worse than a
clustered one.
- Reservation carry-forward: what a file cannot spend goes to the next
file down, bounded by MAX_SHARE. Tracked as two running totals rather
than a `spent` variable threaded through the render loop's dozen exit
paths, so no path can forget to account, and symmetric — a buy that
overshoots suppresses slack until a later under-spend covers it.
Express reproducer: `lib/utils.js` 583 -> 6,268 whole, envelope 9,241 ->
14,505 on the same 13,000 budget. The `memory-budget.ts` exception CG-14
documented is RESOLVED rather than re-justified: it ships whole again at
5,672 (27.3%) while `src/mcp/tools.ts` rises to 52.6% — so the answer
file wins the envelope AND no previously-unclipped file is clipped, which
is CG-12's own acceptance criterion finally holding.
Two hermetic fixtures added, one per lever, because nothing in the suite
had this shape — which is how it shipped. Both mutation-tested: removing
the buy arm reddens 3, removing the carry-forward reddens 2, and removing
the funding guard reddens 4 (including payroll's dropped
`payslip_builder.go`). Their `fixture shape` blocks are load-bearing: the
gates pass vacuously if a target ever drifts inside the grace bound, so
the window is asserted directly.
Full suite green (2,868 passed); both #1500 regression fixtures pass.
Coverage for the CG-12 allocator, built around "would this go red if the
lever were removed" rather than line coverage — every way this regresses
is silent, ending in an agent falling back to Read.
Unit (`explore-proportional-allocation.test.ts`, 18 -> 38): calibration
pins, envelope safety across every tier and 30 candidate shapes, the
cliff boundary, spine weighting/trim survival, the diffuse control, and
the degenerate inputs — identical scores, a lone file, a runaway top
scorer, zero results, maxFiles 0, a non-finite score.
End-to-end (`explore-allocation-e2e.test.ts`, new): CG-6's second
regression fixture as a deterministic synthetic mirror — a large relevant
file, a small helper that used to win by shipping whole, and an
incidental `explore`/`BUDGET` collision — asserting per-file budget
share, not file presence. Plus degenerate result sets and a survey-style
diffuse control through the real render loop. The live self-query arm
stays in probe-allocation.mjs, where drift is a number to re-baseline
rather than a red suite.
Reverting the render loop to the pre-CG-12 rules reproduces #1500 on the
mirror exactly and takes 5 e2e + 2 payroll gates red:
file score pre-CG-12 CG-12
src/mcp/allocator.ts 77.5 4,843 (39.7%) 9,335 (80.1%)
src/util/budget-math.ts 36.0 6,079 (49.8%) 1,037 ( 8.9%)
Two defects the invariants surfaced, both fixed in tools.ts:
- rounded shares could sum past `pool`, so "reservations fit the
envelope" was approximate rather than exact; both terms now floor
- a non-finite score made every share Infinity/Infinity, handing the
render loop a NaN allowance; `weightOf` now fails safe to 0
Also adds a hard-ceiling gate to the payroll fixture — at 19.3K against
a 19.5K ceiling it is the only fixture that stresses the ~25K inline cap
— and exports EXPLORE_ALLOCATION so invariant tests read the constants
while one test pins the literals.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The explore envelope used to follow FILE SIZE, not relevance. Every admitted
file was capped at the same flat `maxCharsPerFile`, while the whole-file rule
handed anything under `maxCharsPerFile * 3` its entire contents — a 3x swing
decided by how big a file happened to be:
- self-query: `memory-budget.ts` (score 18) shipped whole and took 51.2% of
the response; `src/mcp/tools.ts` (score 41, 4x the graph mass, 3x the term
hits — it holds the allocator itself) was clipped at 3,800 and got 32.9%.
- #1500 Go fixture: two generated CRUD files shipped whole at ~4.5K each AND
consumed two of the tier's four file slots, so `BuildPayslip` — the
hand-written "calculate" half of the question — ranked #6 and never
rendered at all.
`allocateExploreBudget` now reserves each ranked file a share of the envelope
before anything renders, so the render loop spends a reservation instead of
racing for whatever the files above it left:
- weight = score x worth x (spine ? 2 : 1), where `worth` is `rankPenalty`
applied a SECOND time — ranking answers "is this file about the query",
allocation answers "will these bytes teach the agent anything", and
generated CRUD can legitimately rank while its bytes stay boilerplate;
- a relative cliff at 15% of the top weight (capped at SCORE_FLOOR_MAX, so a
god-file can't silence peers the score floor just admitted) gives a file
ZERO source — path, symbols and line numbers only — and crucially frees its
`maxFiles` slot for a file that earns its bytes;
- every admitted file gets MIN_CHARS, then the remainder splits by weight:
the floor keeps a diffuse survey question returning a spread, the remainder
concentrates a precise one;
- the flat per-file cap is retired as the primary guard, leaving a 70%-of-
envelope safety valve.
Two changes were needed to make the reservation bite: an oversize cluster now
shrinks by whole MEMBER symbol ranges (a single-cluster god-file previously
took ~40% more than allotted, and the file below it was dropped for lack of
room), and the arrival-order budget stops are gone — they cut files by the
order they were reached rather than by merit.
Measured: payroll-go answer group 25.6% -> 78.7%, generated 57.4% -> 0%, and
`func (s *Service) BuildPayslip` now delivered; self-query `tools.ts` 18.5% ->
60.6%, past the epic's >50% bar. Controls hold: cobra/gin diffuse survey
queries keep their file spread (3->3, 3->4), express's middleware query is
byte-identical, and gin's flow query moves its top file from the thin `ginS`
singleton wrapper to `routergroup.go`.
One documented exception to "no previously-unclipped file becomes clipped":
`memory-budget.ts` was unclipped-whole at 5,672 and now clusters within its
3.1K reservation. That is the epic's own diagnosis of the bug — it scored 18
against 58 and was taking the larger slice purely for being small.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Explore's per-file relevance awarded +50/+10/+3/+1 by match class and admitted
anything scoring >= 3. Neither half held up: the tier said HOW a symbol reached
us, never whether the match was evidence, and an absolute floor admits noise on
any repo where the top file scores 50+. Three scripts/agent-eval/*.mjs harnesses
took 63% of this repo's own "how does explore allocate its output budget" answer
on nothing but an unused `const explore` and a `const BUDGET`.
Four levers:
- KIND WEIGHT (RELEVANCE_KIND_WEIGHT): callables and types 1.0, members ~0.5,
variable/constant/parameter 0.15-0.35. A weak-kind symbol with no usage edge
anywhere in the graph (`contains` excluded — nesting is not usage) drops to
0.08. Only weak kinds in the top two tiers pay for the DB probe; the subgraph's
own edges answer most cases free. No measurable latency change (210 vs 211
ms/call, n=12 interleaved).
- PERIPHERAL CAP: nodes >=2 hops from any match accumulate into a bucket capped
at 5. Uncapped they added a flat +1 each, so a file grew more relevant by being
bigger — parse-session.mjs reached 22 off one constant plus twelve unrelated
symbols.
- RANK PENALTY: generated files x0.3, low-value x0.5, applied to the score AND
the graph mass. Score alone would not have fixed#1500 — the generated CRUD
carries MORE graph mass than the hand-written use-case, and graph mass outranks
score in the comparator. Self-normalizing, never a hard exclusion.
- RELATIVE FLOOR: clamp(topScore * 0.2, 1, 10). Capped at one full-strength
direct match so concentration elsewhere can never exclude one (without it a
named-seed-heavy file pushed the floor to 21 and dropped a file the agent had
named by class name). Backfills to 3 candidates when it would leave fewer, and
drops the evidence requirement rather than return nothing at all.
excludeLowValueFiles was dead config — declared per tier, read nowhere; the
test/spec exclusion has been unconditional for a while. Removed. The real gap was
the detector: `isLowValue` anchored on a leading `/`, so a repo-ROOT `test/` dir
(express, cobra, most of npm and Go) never matched — express's routing question
spent 59% of its envelope on three test files. Anchored at `^` too, and the
filter now runs before the floor and judges "are there other candidates?" on the
whole gather.
Measured before/after on the same indexes (baseline bd86ad2):
- payroll-go fixture: generated 57.4% -> 23.5%; answer 25.6% -> 61.5%; cycle.go
delivered 0 -> 38.9%. Generated ranks #3/#4, was #1/#2.
- self-query fixture: eval scripts 72% -> 0%; tools.ts ranks #1.
- express "route a request": 59% to test/* -> lib/application.js + lib/response.js
- cobra x3, codegraph "indexing pipeline": byte-identical (control)
Diagnostic gains a per-file penalty multiplier and NodeKind mix, so "why did this
file score X" is legible. Selection stages reordered to match the pipeline.
CG-6's gates flip from it.fails to live regressions except the byte-split ones,
which stay open for CG-12 (allocation still follows file size within the ranked
set).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`isGeneratedFile` was path-only, but Go's own convention is a CONTENT
marker (`// Code generated by <tool>. DO NOT EDIT.`), not a filename one.
A Go monorepo with generated CRUD in ordinarily-named files sitting beside
hand-written use-cases was therefore invisible to every generated-file
down-rank in the codebase — that is #1500.
Measured on kubernetes/client-go (2,453 Go files): the canonical banner
appears in 2,001 of them, the path check flags 0, the new content check
flags exactly those 2,001 — no false positives, no misses.
Design: decide at INDEX time (content is already in memory for parsing),
persist on `files.generated`, read from the DB. Explore never reads file
headers per request.
- `hasGeneratedHeader(content)` recognizes the standard banners — Go's,
protoc's, `@generated`, `<auto-generated>`, Thrift, OpenAPI Generator,
FlatBuffers, bindgen, ANTLR. Precision-first and fenced three ways: an
8KB/60-line header window, a comment-line requirement (leader or open
block comment), and markers tight enough that prose can't trip them. A
generator's own source, holding the banner as a string constant in its
body, is not flagged; neither is this module itself (pinned by test).
- `isGeneratedFile(path)` is unchanged — cheap, sync, still the fallback.
- Schema v9 adds `files.generated` + a PARTIAL index. DDL only, no
backfill: the flag derives from content the migration cannot see, so
rows stay 0 until a re-index and every reader unions the flag with the
path check — an un-migrated index keeps pre-#1500 behavior rather than
regressing. Re-index required; noted in the CHANGELOG.
- `generatedPredicateFor(paths)` gives ranking a bounded probe + O(1)
lookups. Bounded, not cached: no invalidation, so a ranking call can
never serve a verdict the last sync already replaced. Wired into explore
ranking, findSymbolMatches, findAllSymbols, search (MCP + CLI), the
context formatter, and the dominant-file/route-file hygiene filters.
Cost (acceptance bar was no measurable index-time regression): a single
unanchored `/generat/i` test over the header rejects ~every hand-written
file before any line splitting. 4.6 µs/file on client-go (worst case —
82% generated). End-to-end `codegraph init` on client-go, n=3 alternating
arms: 5.73s median with detection vs 5.76s path-only baseline; the arms
cross over between runs, so the difference is inside run-to-run noise.
Scope note: generated status remains a stable TIEBREAK at equal score,
exactly where it was. Making it a strong negative signal is CG-10, which
this unblocks by making the signal correct and available.
Two pre-existing tests hard-coded schema version 8; both now track
CURRENT_SCHEMA_VERSION (or the migration table) so future migrations
don't require editing them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
How codegraph_explore divides its byte envelope among files was
unobservable — you could read a response and guess, but not say "this
file took 16% and that one took 20%." Nothing else in the budget-
allocation epic is measurable without that.
CODEGRAPH_EXPLORE_DEBUG now emits one report per explore call (stderr
table, stderr JSON, or a JSONL sidecar path). Per file: relevance score,
graph mass, term hits, ranking flags, render mode, bytes allocated vs
delivered, both shares, and whether it was clipped — plus why a ranked
candidate never rendered. Totals cover envelope vs maxOutputChars vs the
hard ceiling, the source/meta split, the selection funnel, and the score
floor and relevance-gate thresholds applied.
Allocated and delivered are reported separately on purpose: they diverge
exactly when the 25K ceiling truncates, and conflating them is how a
dropped trailing file goes unnoticed.
Off by default and byte-identical when off — it ships in the product
binary, and a diagnostic that perturbs the response by one byte would
invalidate every A/B taken with it on. ExploreDiagnostics.start() returns
null unless the env var is set, so every call site is a `diag?.` no-op.
Baseline recorded in docs/design/explore-budget-allocation.md: on this
repo, src/mcp/tools.ts gets 15.8% of the envelope while three weakly-
relevant agent-eval scripts take 61% between them — despite tools.ts
carrying 5.4x the score and 2.6x the graph mass of any of them. Small
files ship whole; the large answer file is clipped at maxCharsPerFile.
Rank ordering is correct and buys nothing. The loop also allocated 23,193
chars against an 18,000 budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "no covering tests found" flag only inspected a symbol's direct
callers, so helpers exercised transitively by tests (logDebug runs
1,471x under npm test) were reported untested — wrong for ~40% of
flagged symbols per the issue's measurement.
The check now BFSes up the caller graph (3 hops, 64-lookup budget per
entry) and reports indirect coverage as "tested via callers: <files>".
When nothing is found it claims only what was measured — "no tests
found within 3 caller hops", or the weaker "no test calls this
directly" if the budget ran out — and drops the warning glyph.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Python's class-as-value idioms (return SomeClass, x = SomeClass, registry
dicts, classes passed as arguments) produced no references edges, so
callers/impact on a Django/DRF serializer missed the views that consume it.
Three gates dropped them:
- return_statement was never dispatched by PYTHON_SPEC (kernel mirrored)
- the extraction gate (definedHere) collected function/method names only
- resolution accepted function/method targets only (matchFunctionRef +
the function_ref import fast path)
Capture return_statement for Python (single expression; tuple returns not
descended), admit same-file CLASS names to the gate, and accept class
targets for Python bare identifiers — scoped to Python so the TS/JS KIND
FILTER contract is untouched. The docopt false-positive mechanism behind
the function-only rule (lowercase locals vs same-named methods) doesn't
transfer: methods stay excluded for bare ids, and the same-file/import
gate + unique-or-drop rules still apply.
Probed on django-rest-framework (~250 files): 559 new references→class
edges, 10/10 sampled genuine (serializer_class = AuthTokenSerializer, the
ModelSerializer field-mapping registry, aliases, ctor args, isinstance).
EXTRACTION_VERSION 24 → 25.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
codegraph_node / codegraph_explore read CURRENT bytes but slice them at
INDEXED line ranges; after an un-synced edit that slice can be a DIFFERENT
symbol's code served under the requested name — isError: false, introduced
by the 'verbatim … do not Read' guarantee. The watcher-based pending (#403)
and degraded (#876) banners cannot cover a project reached via projectPath:
cross-project instances have no watcher, by construction.
Freshness is now verified at the point of emission from data the index
already stores: one stat per rendered file (size + floored mtime, the sync
fast path's own test), sha256 content-hash compare only on stat mismatch
(so a touch/identical rewrite never false-positives), memoized briefly per
handler. On drift:
- codegraph_node: small files ship WHOLE and CURRENT (Read-parity, still
no Read needed); large ones omit the body with an explicit notice
steering to the tool's file-read mode or Read. Location/signature stay,
flagged as possibly shifted.
- codegraph_explore: the whole-file render (already correct by
construction) is kept and flagged; adaptive/skeleton/cluster slicing is
disabled for drifted files — a too-big drifted file is omitted with a
notice instead. The verbatim/do-not-Read header gains a per-file
exception, and a trailing note flags shifted line references (flow,
blast radius, symbol lists).
The guarantee itself is preserved: everything actually rendered is still
byte-accurate — drifted files ship whole or not at all, never as a
possibly-wrong slice. A re-sync of the target project restores normal
output (covered by test).
Adds __setLoadCodeGraphForTests (same seam pattern as __setFsWatchForTests)
so in-process tests can exercise a genuine cross-project open, which
vitest's transform cannot service through the lazy require.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A SIGKILL'd process (the #850 liveness watchdog, OOM, a crash) leaves its WAL
on disk; the next session appends to the same file; and nothing ever truncated
it — PASSIVE checkpoints fold frames but keep the file at its high-water mark,
and the one shrinking path (a clean last-connection close) is exactly what a
killed-daemon world never takes. Observed at 25.6 GB on a 5.46 GB DB, growing
until the disk filled.
- journal_size_limit on every connection: resetting checkpoints now clip the
WAL back to the cap instead of leaving it at its high-water mark.
- healOversizedWal() fired from every DatabaseConnection.open: off-thread
PASSIVE fold + TRUNCATE when the leftover WAL exceeds the cap (64 MB,
CODEGRAPH_WAL_HEAL_MB to override). Single-flight per connection with
bounded retries — concurrent passes defeat each other (each checkpoint sees
the other as a busy reader).
- Daemon/direct MCP watchdogs now pass progressPaths (DB + WAL), extending the
#1231 slow-disk deferral to the long-lived server so a healthy daemon mid
slow statement isn't SIGKILL'd — fewer kills, fewer leaked WALs.
- codegraph status shows WAL size (human + JSON) and warns when it dwarfs the
DB; daemon.log lines and the watchdog kill notice now carry ISO timestamps
so kills can be placed in time.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The standalone bundle's bin dir exposes only codegraph.cmd, and Claude
Code executes UserPromptSubmit hooks through Git Bash, which applies no
PATHEXT — so the bare `codegraph prompt-hook` the installer wrote was
"command not found" (exit 127) on every prompt. Write the platform-correct
spelling, recognize both spellings on uninstall/opt-out, and self-heal an
installer-written entry from the other platform in place on install/upgrade
re-runs (npx/hand-edited variants stay untouched).
Reproduced and validated on the Windows VM: bare form exits 127 under Git
Bash on a standalone-only PATH, codegraph.cmd exits 0; full installer suite
(165 tests, including the new migration coverage) green on Windows + macOS.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A user-level mcp.json entry using ${workspaceFolder} makes VS Code
refuse to start the server in ANY window without a folder open (loose
files, welcome tab), toasting "Variable workspaceFolder can not be
resolved" — recurring error-noise, hit live during validation.
The pin was never needed for VS Code: unlike Cursor, VS Code documents
stdio-server cwd as the workspace folder, and the codegraph server
resolves its project via roots/list with a cwd fallback. Global entries
are now variable-free (`serve --mcp`); local installs keep the absolute
--path. This supersedes the "open a folder" install note from the
previous commit, which is removed again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VS Code refuses to start a user-level MCP server whose entry uses
${workspaceFolder} in a window with no folder open, surfacing only a
cryptic "Variable workspaceFolder can not be resolved" toast (hit live
during validation). Global installs now note this up front.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The VS Code Copilot Chat extension writes MCP socket-handoff lock files
into ~/.copilot/ide/ on launch, so `existsSync(~/.copilot)` reported the
Copilot CLI as installed on any machine that merely has the VS Code
extension (caught live on the maintainer's Mac). Detection now counts
the dir as a CLI footprint only when it holds something besides `ide`.
Also: uninstalling a from-scratch install now deletes mcp-config.json
instead of leaving a `{}` husk that would keep detect() reporting the
CLI as installed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds three new installer targets so `codegraph install` can wire the
MCP server into GitHub Copilot surfaces:
- copilot-vscode: .vscode/mcp.json (local) or the VS Code User-dir
mcp.json (global), JSONC-surgical edits, `--path` pinned via
${workspaceFolder} for global installs
- copilot-cli: ~/.copilot/mcp-config.json
- copilot-jetbrains: github-copilot config dir (XDG / %LOCALAPPDATA%)
Detection, install, uninstall, and --print-config are covered for all
three in installer-targets.test.ts, including platform-specific path
resolution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Codex installer's `findNextTableHeader` skipped `[[array-of-tables]]` headers instead of treating them as a block boundary, so any `[[...]]` block after `[mcp_servers.codegraph]` in ~/.codex/config.toml was silently deleted on install/upgrade/uninstall. Now treats both `[...]` and `[[...]]` as boundaries, with a small line lexer so header-shaped text inside multiline strings/arrays isn't mistaken for a boundary. Adds round-trip regression coverage (install → reinstall → uninstall) + CHANGELOG entry.
Fixes#1351. Supersedes #624.
Thanks @KtzeAbyss.
Two changes to the watcher path (the always-on daemon every agent
session uses), which previously paid a flat 2s debounce plus a full-tree
scan-diff on every save even though the OS events name the exact files:
1. Adaptive debounce: a pending set of ≤2 files fires after a 300ms
quiet window; ≥3 keeps the full configured window so agent
multi-file bursts coalesce exactly as before. Re-arming preserves
trailing-edge semantics; a user-set CODEGRAPH_WATCH_DEBOUNCE_MS
remains the authoritative upper bound (quick window never exceeds
it, floor 100ms).
2. Scoped sync: watcher-triggered syncs pass their pending paths, and
the reconciler stats exactly those — per-path logic identical to the
full walk (stat pre-filter, hash confirm, the #1240
removal/resurrection flow) — skipping the O(repo) scan and
tracked-load. Strict fallbacks keep the full scan-diff as ground
truth: directory removals (#1285 — the events can't name the
children), empty pending sets (retry paths), and >500-file storms
(branch checkouts, which also self-heal anything event coalescing
dropped). filesChecked counts examined PATHS so a deletion-only
scoped sync can't mimic the #449 lock-unavailable signature.
Measured (warm in-process, the daemon path): dubbo one-file sync work
512→335ms, Swift compiler (27k files) 884→385ms — save-to-fresh-graph
≈0.6-0.7s end-to-end including the quick debounce, from ~2.5-6s
perceived before. Gates: scoped-vs-full dumps byte-identical on dubbo
AND the Swift compiler; watcher suite 30/30 (3 new: scoped pass-through,
dir-removal fallback, quick-fire timing); sync suite 34/34 (4 new
scoped-parity cases incl. delete-resurrection and the lock signature);
full suite 2,696 ×2 with CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The swiftc =2/nm:mc-* attribution located the wall: the
getSupertypes conformance walk ran 971,200 times (565s of combined
worker time, 581µs each) — every resolveMethodOnType miss re-queried
implements/extends edges for every same-named type node, recursing
depth-4 through Swift's protocol landscape with no memoization, and
post-inference resolveMethodOnType averaged 1,912µs per call.
Fix 1 — generation-tagged getSupertypes memo. Supertype edges GROW
during the resolution loop (batch k persists its edges BEFORE batch
k+1 fans out — the #1320 ordering), so a plain cache would freeze an
early batch's emptier answer. Within a batch the edge state is fixed
by that same ordering, so memo entries carry a generation that
advances at every batch entry point (resolveBatchYielding /
resolveListForAdmission — covering the sequential loop, pool workers,
sync admission, and the conformance pass); a stale-gen entry
recomputes. Behavior-identical to no memo at every point in time;
walk invocation counts match the unmemoized run exactly (971,200 /
24,336 / 76,415).
Fix 2 — per-(language, method-name) owner index in getMethodMatches:
candidates bucket once by their qualifiedName's last two segments
(exactly the span the match predicate tests), so a (type, method)
query is a map lookup instead of an O(candidates) scan per
methodMatchCache miss. ObjC selectors and multi-segment typeNames
keep the legacy linear path. Also ships nm:mc-rmot / nm:rmot-supers
=2 attribution rows.
swiftc: settle 100.2→31.3s, resolveMethodOnType 1,912→202µs, wall
183.5→97.8s (was 185s at the head-to-head; cbm's same-box number is
119.1s). Gates: swiftc old-vs-new dump byte-identical (1,837,235
rows), swiftc pooled-vs-CODEGRAPH_NO_PARALLEL_RESOLVE=1 identical
(the generation-semantics risk surface), dubbo old-vs-new identical
(49k Java instance-method hits share both paths), Alamofire
identical; suite 2,689 ×2 with CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The kong/tokio matcher-chain residue attributed (nm:mc-* sub-stage rows,
shipped here too): matchMethodCall's cost is ~entirely
inferLocalReceiverType — 61µs per miss on kong, 99% miss rate (39k `self:`
calls hunting a local declaration Lua never writes), re-scanning the same
scope lines for every ref.
Two pure memos, both semantics-preserving by construction:
- Compiled-pattern memo: localReceiverTypePatterns/phpPropertyTypePatterns
built 2-4 fresh RegExp objects per call; patterns are a pure function of
(language, receiver) and non-global, so instances are shared via a
FIFO-capped map (no per-get mutation — the §7a.6 LRU-churn lesson).
- Incremental scan memo: refs for the same (file, scope, receiver) arrive
in ~ascending line order and the backward declaration scan is a pure
function of immutable file lines — a per-context watermark scans each
line once per key (query(c) = highest match in [start..c]; monotonic
calls extend the watermark over (hi..c]; non-monotonic calls fall back
to the plain bounded scan). componentScoped (CFML/PHP whole-file sweep)
is keyed out. States drop with the context's file caches via
clearNameMatcherMemos, wired into ReferenceResolver.clearCaches.
kong mc-infer misses 61→20µs (2.4s→0.8s combined); fresh index 3.43 →
3.03-3.20s (−8%; 4.07 → 3.14 cumulative with #1391). tokio unchanged
(tight scopes). Gates: dubbo (49k Java instance-method HITS ride this
scan), kong, tokio, Fusion dumps all byte-identical; suite 2,689 ×2 with
CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
ctype_lab
YucandGitHub