cc9ce09256a8e824f93e00a2af302bd33ae3360d
385
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colby McHenry
|
ccb0295259 |
fix(explore): reliably pin extension-less kebab-case file basenames in queries
Previously, naming a kebab-case file without its extension (e.g., `background-image-table` vs. `background-image-table.tsx`) in a `codegraph_explore` query would shred the name into fragments (`background`, `image`, `table`), admitting irrelevant sibling files and crowding out the intended target. This change introduces a new resolution pass in `extractQueryPaths` specifically for extension-less kebab basenames. Queries now accurately identify and pin these files. Unresolved hyphenated prose (e.g., `cross-call`) is left in the query for FTS without being flagged as an unknown path. Resolution prioritizes explicit slashed/dotted paths and respects an ambiguity budget for common stems to prevent over-pinning. |
||
   
|
81e1f4a92f |
fix: harden daemon and large-index recovery paths (#1562)
* fix: harden indexing recovery and daemon liveness * test: cover daemon and recovery review gaps * test: pin that a failure marker never blocks a later successful parse (#1557 retry-discard guard) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Colby McHenry <me@colbymchenry.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d8f2eeaddf |
fix(db): loop-append dense unresolved-ref result rows; make stripped-salvage visible (#1558) (#1576)
Real-world validation of #1575 on indexes damaged by the released v1.5.0 binary surfaced both of these. getUnresolvedReferencesByFiles chunked its INPUT under SQLite's parameter limit but appended each chunk's RESULT rows with a spread — every row becomes a call argument, so a dense recovery sync (the #1541 self-heal re-indexing 919 files produced 234,440 rows) exceeded V8's argument limit and killed resolution mid-sync with "Maximum call stack size exceeded", leaving the graph 226k edges short until another sync resumed the orphans (and that sweep resolves measurably worse than the batched path — see the follow-up issue). The failed-ref retry loader had the identical pattern on unbounded result rows. Both append with a loop now (#1558). The #1575 stripped-salvage warning also never rendered: init's summary prints only index_partial warnings and counts only hard errors, so a run with salvaged files still read as fully clean — and with no hard errors the detail wasn't written to errors.log either. Salvage entries now carry code 'salvaged_stripped', the summary prints a visible warning naming the files, and errors.log is written for salvage-only runs. Validated on real corpora with full-graph dumps: healthy-path inits stay byte-identical to the pre-#1575 baseline (cpython Lib, Alamofire, with a determinism control); a realistically-damaged index (41 wiped + 5 missing files, damage generated by the released binary) heals in one plain sync to identical per-file counts and an edge set within the normal incremental residual; pathological mass damage (52% of the repo) completes without crashing. New regression test reproduces the RangeError on the old code with 200k pending refs. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
26045b3159 |
fix(extraction): decode kernel results in indexAll retry passes; self-heal wiped rows (#1541) (#1575)
The parse-pool workers return kernel-language extractions as an undecoded buffer transport (nodes/edges EMPTY, tables in kernelBuffers). indexAll's main loop decodes them (or hands the buffers to the store worker), but its two retry passes — plain retry and the comments-stripped last resort — stored the transport as-is: the storage gate passed via errors.length === 0, zero nodes were inserted, and the files row was written with node_count = 0 while the original error was spliced out of the summary. Any worker crash/timeout whose in-flight file was a kernel-routed language permanently recorded that file as "(0 symbols)" — silently, and immune to later syncs because the stored hash matches the on-disk bytes (#1541; v1.4.1 predates the kernel path, which is why it was unaffected). - Both retry passes now materialize kernel results before the gate, store, counters, and log lines. - storeExtractionResult materializes at entry as defense-in-depth, so no storage path can persist an undecoded transport again. - Zero-node rows on symbol-bearing languages (only the wipe produces these — every real extraction stores at least the file node) are dropped during full-reconcile sync and indexAll so already-affected files re-index automatically after upgrading. Scoped watcher syncs leave rows outside their scope untouched. - The comments-stripped salvage now downgrades the failure to a visible warning instead of erasing it: the recovered result can be incomplete, and reporting clean success made a fresh index quietly disagree with a later per-file re-parse of the same bytes (#1565's init-vs-sync divergence). Repro (released 1.5.0): CODEGRAPH_PARSE_TIMEOUT_MS=1 codegraph init on any Python project → "Retry OK: <file> (0 nodes)" and permanent "(python, 0 symbols)" rows. Fixed build stores real symbols under the same forcing, and heals rows wiped by prior runs. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby McHenry
|
d806317897 |
fix(explore): improve query accuracy for file paths, camelCase, and variables
`codegraph_explore` previously struggled with accurately interpreting user queries. Explicitly named file paths were shredded, making it hard to target specific files; natural language queries often missed camelCase identifiers; and state held in variables was overlooked as starting symbols. This commit introduces several improvements: - **Reliable File Path Resolution:** Naming a file by its path in a `codegraph_explore` query now works reliably. The path is resolved against the index, and that file is guaranteed a place at the top of the answer. Previously, paths were broken into fragments — bracketed route segments like SvelteKit's `[id]` made this worst — and pieces like `page` or `runs` matched every sibling file, so the file you actually named could be crowded out. A path that doesn't match any indexed file is now called out instead of silently ignored. - **CamelCase Matching for Queries:** Plainly-worded `codegraph_explore` questions now find camelCase code. A query like "auto-scroll to bottom" can reach a function named `scrollFeedToBottom`, because query words are matched against the words inside identifiers, not just whole names. - **Variable and Constant Seeding:** Variables and constants now count when `codegraph_explore` picks its starting symbols, so state held in plain variables — `$state`-style variables in Svelte, for example — no longer gets overlooked. |
||
|
Colby McHenry
|
d289bf84d3 |
Merge main into fix/union-declarations-not-indexed
Resolves the CHANGELOG conflict — main and this branch each prepended a bullet to [Unreleased] > Fixes; both are kept. Everything else auto-merged, including src/mcp/tools.ts, which main reworked heavily for the explore allocation/displacement work (CG-28/31/36/38) while this branch added the `union` kind to its container sets. Verified on the merged tree with the native kernel built: 3070 passed, 9 skipped, 0 failed. |
||
|
Colby McHenryandClaude Opus 5
|
5b0c4b8b93 |
fix(resolution): trait dispatch reaches union implementors (#1515)
Making unions first-class nodes leaves the third loss in #1515 open: interfaceOverrideEdges enumerates its concrete side as ['class','struct'], so a union implementor is skipped even though it now has a real node and a real `implements` edge. "Who implements this trait" then answers wrongly rather than incompletely — the struct beside it bridges and the union does not. Add 'union' to that tuple, plus a regression test that pins the Rust trait -> union-impl hop (the struct implementor is the control proving the synthesizer ran). Verified the test fails on the union assertion alone before this change. No EXTRACTION_VERSION bump: main is already at 25 against v1.5.0's 24, so existing indexes are flagged stale for the next release regardless, and over-bumping is what turns the re-index hint into noise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenry
|
493d4210f1 |
Merge branch 'main' into feat/copilot-installer-targets
# Conflicts: # CHANGELOG.md |
||
|
Colby McHenryandClaude Fable 5
|
2962e7e1f4 |
feat(mcp): surface supported languages in MCP server instructions (#671)
Recut of #678 against the current instructions — the original predated the explore-first rewrite and conflicted in both files it touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandGitHub
|
99f2ebf0d1 |
Merge pull request #1527 from colbymchenry/bugfix/CG-38
CG-38: guarantee an agent-named symbol renders, wherever it sits |
||
|
Colby McHenry
|
2c708caf7c | Merge branch 'main' into feature/CG-35 | ||
|
Colby McHenry
|
89c53ddf24 |
fix(explore): guarantee an agent-named symbol renders, wherever it sits (CG-38)
`codegraph_explore` never returned `queueMessage` (L1087) or `flushQueuedMessages` (L1102) from a 1,414-line file, on a symbol bag or a prose question, even with that file at rank #1 holding 67% of the envelope — the agent got a same-stem `QueuedMessage` interface at L70 and had to Read the file for the functions it had named. Pre-existing at every build including pre-epic (controlled bisect, index held fixed). Two independent causes: 1. `buildFlowFromNamedSymbols` returns the Flow prose AND the set of node ids the agent named — and the latter is the whole guarantee, since it injects a named def into its file's cluster ranges at importance 9. Its bail-outs returned EMPTY, zeroing the identity whenever there was nothing to PRINT. Two sibling closures that never call each other produce no chain, no synth hop and no boundary, so both defs lost importance 9 and the file rendered from its head. `identityOnly()` now separates the two, gated on shape-precise tokens so a prose word that exact-matches a callable cannot promote itself. 2. The ceiling trim filled in SOURCE order, so an over-ceiling render always dropped the END of a large file first. The shrink HAD kept both symbols (1022-1121); the trim cut back to 839. `windowToCeiling` now takes the spine call site plus every importance>=9 member as focus lines, tries the full ceiling first, and splits the held-back reserve evenly with carry-forward — greedy-in-source-order reproduced the bug one level down. The shrink's loose size estimate is left alone deliberately, and the comment now says why: making it exact was built and measured WORSE (it stops at the last member that fits whole and the released bytes carry forward to lower-ranked files, costing payroll-go's `s.store.Upsert`). `bound()` clamps to the ceiling anyway, so the slack costs no bytes; it just must not pick the survivors, which is what the trim now handles. The measurement gap this closes: every existing probe is aggregate — envelope share, per-file spend, source totals, file counts — and all are green on a response that returns 25K from the right file and omits the named function. `probe-named-symbol.mjs` checks the definition LINE against the response's rendered lines, per symbol. Suite envelope byte-identical to main on all six repos; probe-allocation 4/4, no starvation flags; 180 files / 2,997 tests green. Fixture: 7/7 fail on main, 7/7 pass here, deterministic over 4 runs per arm. |
||
|
Colby McHenry
|
10f1ac601a |
docs(benchmarks): record the CG-36 cluster-starvation measurement
The issue blamed the density tiebreak; both real cases lost on maxImportance, so ranking was left alone. Full before/after table, the one cost (okhttp's rank-6 file, squeezed out by reservations that were already structurally over-subscribed), and what ships to keep it measurable. |
||
|
Colby McHenryandClaude Opus 5
|
9efae0f8f2 |
fix(explore): damp ambient declaration files on flow queries (CG-28)
A file that declares nothing but types and that nothing in the index depends on — a hand-written ambient `.d.ts` of global shims, vendored typings, module augmentation — cannot answer a flow question: no bodies, no call edges, no behaviour, nothing typed by it. But the identifiers it declares are exactly the generic ones a prose question uses (`Body`, `Message`, `ImageMetadata`, `ReadableStream`), so on term overlap it out-scored the implementation. Measured on the new fixture: rank #1 and 51% of delivered source, with the flow's own entry file pushed out of the response entirely. Measured first, per the issue: the Wrangler `worker-configuration.d.ts` that opened this is already handled by CG-25's banner detection, worth 15-46 points of envelope share across four flow queries. CG-25 credited; only the un-bannered case needed anything. `rankPenalty` now multiplies score and graph mass by 0.5 for such files, taken as the STRONGER of it and the generated penalty rather than multiplied — one property two signals see must not be charged twice. Detection is structural, not by extension, and four conditions deep. Two of them were forced by measurement: requiring every symbol to be type-level takes the corpus flag rate from 1-18% (which swept in Kotlin sealed classes, Rust mod.rs re-exports and django's locale tables) down to 0-4%; requiring that nothing depends on the file separates an ambient shim from a working types module, and without it the rule demoted displacement-ts's pipeline `types.ts` and broke the CG-31 gate. A query that NAMES a declared type is exempt, so a question about a type still reaches its declaration at full weight. Precise tokens only, so "…the file body…" cannot exempt a `Body` interface it never meant to name; this needs its own set because `namedSeedIds` is callable-only and a type never becomes one. Regression evidence in docs/benchmarks/explore-declaration-only-cg28.md: 6-repo envelope sweep byte-identical against a clean baseline build, zero ambient files reach the candidate set on VS Code across five queries, corpus flag rate 0-0.74%, both allocation fixtures PASS, full suite 2,978 green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
dc4fd755ef |
merge: recognize Wrangler-style generated banners (CG-25)
A generated Cloudflare Wrangler ambient-types file was not flagged generated, so
it ranked with no penalty and competed with hand-written source on generic token
overlap. The banner shape it uses — "Generated by <tool> by running <command>" —
matched none of the existing content patterns, all of which require DO NOT EDIT,
a standalone @generated, or the "auto(matically) generated by" phrasings.
Precision is held by requiring TWO 'by' clauses: the banner must name a tool and
then say 'by running'. Ordinary prose ("the report is generated by running the
nightly job") has only one and does not match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Colby McHenryandClaude Opus 5
|
57e0854213 |
fix(explore): recognize Wrangler-style "generated by … by running" banners (CG-25)
Cloudflare Wrangler's `worker-configuration.d.ts` (~12k lines of ambient types) carried no banner any GENERATED_CONTENT_PATTERNS entry matched: every existing marker requires `DO NOT EDIT`, a standalone `@generated`, `<auto-generated>`, or the literal `automatically/auto-generated by` phrasings. Wrangler emits a bare `Generated by Wrangler by running `wrangler types``, so the file ranked with pen 1.00 and won 79.4% of an explore envelope on generic token overlap alone (CG-24). The discriminator is the reproduction instruction, not the word "generated": the banner must name a tool AND then say `by running`, i.e. two separate "by" clauses. That keeps prose out — "the nightly summary is generated by running the ETL job" has only one — while catching every CLI-driven emitter that tells you how to regenerate. Precision swept over 441,856 files across the whole local source tree: 5 hits, all genuine Wrangler output, no false positives. Isolated before/after on the CG-24 repro (same query, same index, only the `files.generated` flag differing): before pen 1.00 score 115.0 share 79.4% 3 files rendered after pen 0.30 score 35.4 share 21.1% 4 files rendered The new pattern stays in the existing table position, below the header window the detector scans, so the module still does not classify itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
03893b0ab9 |
CG-33: converge incremental sync with a full rebuild
A live, auto-synced index did not converge to a clean rebuild of the same tree — 4.3% of distinct edges wrong in both directions on this repo's own index, overwhelmingly `calls`, which is what flow queries traverse and what explore's file ranking weights. Silent: nothing warned, and the symptom read as "codegraph isn't very good" rather than "this index needs rebuilding." Two causes, and the fix needed both. Resolution binds a reference to one of the same-named definitions PROJECT-WIDE, so a definition appearing or vanishing changes the correct answer for references in files the sync never touches — and those references resolved successfully once, which deletes their unresolved_refs row, leaving nothing to revisit them with (#1240's retry only revisits refs parked as failed). Separately, when nothing disambiguated the candidates the winner came down to rowid, i.e. the order files happened to be WRITTEN, which differs between a scan-order full index and a sync that appends each file as it changes. That second one is why re-resolution alone could not converge: re-resolving against the identical graph still picked a different candidate. So getNodesByName now orders by (file_path, start_line) — a property of the code, not of the write order — and sync computes a definitionDelta and re-opens the resolution edges whose answer it may have invalidated, re-inserting each as the reference that created it for the orphan sweep to bind against the post-sync graph. The delta compares `file\0name` pairs per file rather than one name set over the batch: a commit that adds `collect` to a new file while an unrelated changed file already defines `collect` cancels out of a batch-wide set, and that miss was the largest residual class in the first measurement. Conservative where the failure modes are asymmetric — a wrong deletion is a permanent edge loss, a missed rebind is only residual drift. Edges without a refName stamp are never touched (nothing to restore them from), sources the sync already re-extracted are skipped, and a per-name ceiling declines the generic names. Edges are deleted before the sweep re-inserts, since INSERT OR IGNORE against idx_edges_identity would otherwise keep both rows when a reference rebinds elsewhere. Replaying real commits of this repo through sync, then diffing against a rebuild: 16 commits 48 -> 0; 80 commits 1,634 -> 361, with the actively misleading direction (stale edges the index keeps asserting) 671 -> 2. Index and sync wall-clock are unchanged; the ORDER BY costs 18% per uncached name lookup, which never reaches wall-clock because the resolver memoizes it. The 357-edge residual at 80 commits is one pre-existing class: refs to generic names (`push`, `join`) parked above #1240's per-name retry ceiling, which a rebuild resolves into cross-language garbage — a TS test file "calling" an R method. Converging there would mean manufacturing wrong edges, so it is left alone. And no drift metric in `codegraph status`: it cannot be computed without the rebuild it would be recommending, and a proxy would fire on that residual and train users to ignore it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
5f32478b57 |
docs(benchmarks): record the CG-26 A/B — the invariant holds on every path
Deterministic 6-repo table, the three agent A/Bs (django, excalidraw, okhttp, 2 runs/arm, Read 0 in all 12 runs), and an honest read of the two repos that deliver a few hundred fewer source chars: at the CG-31 tip both were over-filled by the flat-200 section overhead and paid for it by discarding their epilogue whole. Also: CHANGELOG entries for the two user-visible changes, and the memory note now carries the fourth accounting gap plus the two lessons — hold the REMAINDER when a full reservation no longer fits, and never skip a file over an accounting difference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
be7c968439 |
docs(benchmarks): record the CG-31 A/B — no regression, four repos stop truncating
Deterministic (6 repos, clean rebuilds, both builds): four deliver more source and one more file each, two are byte-identical, none deliver less. Agent A/B (django n=3, okhttp n=2, gin n=2, sonnet/effort high, both arms codegraph-on, 0 contamination): the new arm is faster on all three, Read at or below baseline, occupancy lower. Also records the two corrections the suite forced on the first cut of the guard, and the two residuals CG-26 inherits — the render loop's 600-char epilogue margin (a sweep was run and deliberately NOT shipped) and the BUY arm's source-space-only guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
 ctype_lab
|
ba58365c6f | docs(union): describe first-class union nodes | ||
|
Colby McHenryandClaude Opus 5
|
089dcc276f |
fix(explore): hold back what is still owed below a clustered render (CG-31)
Carry-forward slack let a file spend what the files ABOVE it left on the
table. Nothing held back what was promised BELOW it. The whole-file BUY arm
has always refused that trade (`owedBelow`); the cluster path read `headroom`
— what is left before the hard ceiling — instead of what is still owed, so
`fileBudget` and `SPINE_CEILING` could pay a 1.5x overshoot out of another
file's reservation.
`fundedHeadroom` is the same inequality in the units the cluster path spends
in: source PLUS the per-section overhead each unreached file will charge.
Floored at the file's own reservation — a kept promise is not a displacement —
and it is <= `headroom` by construction, so it is the only bound the three
render sites need. The skeleton path's `bodyCap` takes it too.
Measured on `__tests__/fixtures/displacement-ts` (a 4-stage pipeline padded
past 500 files, where the 24K envelope genuinely saturates the 24.4K render
ceiling):
before ingest.ts emitted 9,301 on a 6,289 spendable, then lost the whole
section to the final ceiling — 0 delivered. types.ts and sink.ts
skipped `budget-whole-file`. 3 of 6 admitted files delivered.
after ingest.ts bounded to the 4,913 actually free. 6 of 6 delivered,
envelope 14,908 -> 22,066.
The self-query allocation fixture flips back to PASS with it, on a clean full
rebuild of this repo's index (CG-33). Its `afterCG30` verdict blamed an
over-RESERVED incidental file; the reservation was identical in both arms —
the file was over-SPENDING. Recorded honestly in `afterCG31`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
ctype_labandClaude Opus 5
|
86854cd0d7 |
docs(union): changelog entry + port-checklist annotations
The two kernel port checklists record the extractor configs as surveyed at porting time; their structTypes lines are marked superseded rather than rewritten, so the surveys stay readable as history. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
d652c148f6 |
docs(cg-30): changelog entry + record the self-query probe flip honestly
The self-query allocation probe fixture's delivered-share gates now fail. The cause is not the new bound: allocation is unchanged between arms (parse-run.mjs 32.3% vs 33.9% on main) and tools.ts delivers the same 8,282 chars in both. What changed is that the incidental file now DELIVERS — on main its whole section was cut by the hard-ceiling truncation, so the fixture passed on truncation luck. Every file on this repo obeys the new bound (max 1.40x of spendable). Recorded as `afterCG30` with that reasoning rather than tuning the bound to restore the pass. The over-reservation it exposes is epic CG-24's subject. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenry
|
7a7ea30cbd |
docs: cross-call dedup — its gates, where the bytes go, and the all-pointer guard (CG-18)
Extends the session-state design doc with the layer built on it: what gates a withheld span (session, content fingerprint, two size floors, kill switch), why the fingerprint and not #1474's drift flag, the two channels the reclaimed bytes leave by, and why "no duplicate ranges across calls" holds for every call that had something new to say rather than universally. |
||
|
Colby McHenryandClaude Opus 5
|
c65d56ceba |
docs: CG-22 — the epic's gate, re-run at CG-15's exact setup (#1500)
CG-21 fixed the unspent-reservation defect and re-ran the A/B itself. CG-22 is
the gate proper: CG-15's setup, unchanged, measured independently of the task
that wrote the fix. RUNS=3, both arms codegraph-on, sonnet/high,
CODEGRAPH_NO_PROMPT_HOOK=1 on both, baseline pinned to
|
||
|
Colby McHenry
|
51cd053d85 |
docs: CG-21 — spending the reservation (design record + CHANGELOG precision)
Records both levers, the funding-pool design (and the per-file version that dropped payslip_builder.go), the resolved memory-budget.ts exception, and the two hermetic fixtures with their mutation matrix. The CHANGELOG clause 'no longer trimmed while a smaller, weakly-related one is included whole' was imprecise after CG-21: the smaller file often IS still included whole now, when its share nearly covers it. Reworded to say what the fix actually guarantees. |
||
|
Colby McHenryandClaude Opus 5
|
8077b83eb3 |
docs: consolidate the #1500 CHANGELOG entries into user-facing shape (CG-16)
Three separate engineer-shaped entries (CG-5 generated detection, CG-10 scoring, CG-12 allocation) become two user-facing bullets under Fixes, in the shape #1500 actually reported: explore concentrates on the code that answers the question, and a generated CRUD/protobuf layer no longer crowds out the hand-written code beside it. Per the CHANGELOG rules: strips the benchmark counts and percentages (the client-go 2,001-file count, the quarter-to-four-fifths envelope shift) and the internal symbol names, keeps the re-index note, and credits the reporter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
5f7f5f59df |
feat(mcp): score-proportional byte allocation for explore, with a relative cliff (CG-12, #1500)
The explore envelope used to follow FILE SIZE, not relevance. Every admitted
file was capped at the same flat `maxCharsPerFile`, while the whole-file rule
handed anything under `maxCharsPerFile * 3` its entire contents — a 3x swing
decided by how big a file happened to be:
- self-query: `memory-budget.ts` (score 18) shipped whole and took 51.2% of
the response; `src/mcp/tools.ts` (score 41, 4x the graph mass, 3x the term
hits — it holds the allocator itself) was clipped at 3,800 and got 32.9%.
- #1500 Go fixture: two generated CRUD files shipped whole at ~4.5K each AND
consumed two of the tier's four file slots, so `BuildPayslip` — the
hand-written "calculate" half of the question — ranked #6 and never
rendered at all.
`allocateExploreBudget` now reserves each ranked file a share of the envelope
before anything renders, so the render loop spends a reservation instead of
racing for whatever the files above it left:
- weight = score x worth x (spine ? 2 : 1), where `worth` is `rankPenalty`
applied a SECOND time — ranking answers "is this file about the query",
allocation answers "will these bytes teach the agent anything", and
generated CRUD can legitimately rank while its bytes stay boilerplate;
- a relative cliff at 15% of the top weight (capped at SCORE_FLOOR_MAX, so a
god-file can't silence peers the score floor just admitted) gives a file
ZERO source — path, symbols and line numbers only — and crucially frees its
`maxFiles` slot for a file that earns its bytes;
- every admitted file gets MIN_CHARS, then the remainder splits by weight:
the floor keeps a diffuse survey question returning a spread, the remainder
concentrates a precise one;
- the flat per-file cap is retired as the primary guard, leaving a 70%-of-
envelope safety valve.
Two changes were needed to make the reservation bite: an oversize cluster now
shrinks by whole MEMBER symbol ranges (a single-cluster god-file previously
took ~40% more than allotted, and the file below it was dropped for lack of
room), and the arrival-order budget stops are gone — they cut files by the
order they were reached rather than by merit.
Measured: payroll-go answer group 25.6% -> 78.7%, generated 57.4% -> 0%, and
`func (s *Service) BuildPayslip` now delivered; self-query `tools.ts` 18.5% ->
60.6%, past the epic's >50% bar. Controls hold: cobra/gin diffuse survey
queries keep their file spread (3->3, 3->4), express's middleware query is
byte-identical, and gin's flow query moves its top file from the thin `ginS`
singleton wrapper to `routergroup.go`.
One documented exception to "no previously-unclipped file becomes clipped":
`memory-budget.ts` was unclipped-whole at 5,672 and now clusters within its
3.1K reservation. That is the epic's own diagnosis of the bug — it scored 18
against 58 and was taking the larger slice purely for being small.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Colby McHenryandClaude Opus 5
|
a3898cdc70 |
feat(mcp): relevance scoring overhaul for explore — kill incidental name-collision matches (CG-10, #1500)
Explore's per-file relevance awarded +50/+10/+3/+1 by match class and admitted
anything scoring >= 3. Neither half held up: the tier said HOW a symbol reached
us, never whether the match was evidence, and an absolute floor admits noise on
any repo where the top file scores 50+. Three scripts/agent-eval/*.mjs harnesses
took 63% of this repo's own "how does explore allocate its output budget" answer
on nothing but an unused `const explore` and a `const BUDGET`.
Four levers:
- KIND WEIGHT (RELEVANCE_KIND_WEIGHT): callables and types 1.0, members ~0.5,
variable/constant/parameter 0.15-0.35. A weak-kind symbol with no usage edge
anywhere in the graph (`contains` excluded — nesting is not usage) drops to
0.08. Only weak kinds in the top two tiers pay for the DB probe; the subgraph's
own edges answer most cases free. No measurable latency change (210 vs 211
ms/call, n=12 interleaved).
- PERIPHERAL CAP: nodes >=2 hops from any match accumulate into a bucket capped
at 5. Uncapped they added a flat +1 each, so a file grew more relevant by being
bigger — parse-session.mjs reached 22 off one constant plus twelve unrelated
symbols.
- RANK PENALTY: generated files x0.3, low-value x0.5, applied to the score AND
the graph mass. Score alone would not have fixed #1500 — the generated CRUD
carries MORE graph mass than the hand-written use-case, and graph mass outranks
score in the comparator. Self-normalizing, never a hard exclusion.
- RELATIVE FLOOR: clamp(topScore * 0.2, 1, 10). Capped at one full-strength
direct match so concentration elsewhere can never exclude one (without it a
named-seed-heavy file pushed the floor to 21 and dropped a file the agent had
named by class name). Backfills to 3 candidates when it would leave fewer, and
drops the evidence requirement rather than return nothing at all.
excludeLowValueFiles was dead config — declared per tier, read nowhere; the
test/spec exclusion has been unconditional for a while. Removed. The real gap was
the detector: `isLowValue` anchored on a leading `/`, so a repo-ROOT `test/` dir
(express, cobra, most of npm and Go) never matched — express's routing question
spent 59% of its envelope on three test files. Anchored at `^` too, and the
filter now runs before the floor and judges "are there other candidates?" on the
whole gather.
Measured before/after on the same indexes (baseline
|
||
|
Colby McHenryandClaude Opus 5
|
16e17495f4 |
feat(extraction): content-based generated-file detection (CG-5, #1500)
`isGeneratedFile` was path-only, but Go's own convention is a CONTENT marker (`// Code generated by <tool>. DO NOT EDIT.`), not a filename one. A Go monorepo with generated CRUD in ordinarily-named files sitting beside hand-written use-cases was therefore invisible to every generated-file down-rank in the codebase — that is #1500. Measured on kubernetes/client-go (2,453 Go files): the canonical banner appears in 2,001 of them, the path check flags 0, the new content check flags exactly those 2,001 — no false positives, no misses. Design: decide at INDEX time (content is already in memory for parsing), persist on `files.generated`, read from the DB. Explore never reads file headers per request. - `hasGeneratedHeader(content)` recognizes the standard banners — Go's, protoc's, `@generated`, `<auto-generated>`, Thrift, OpenAPI Generator, FlatBuffers, bindgen, ANTLR. Precision-first and fenced three ways: an 8KB/60-line header window, a comment-line requirement (leader or open block comment), and markers tight enough that prose can't trip them. A generator's own source, holding the banner as a string constant in its body, is not flagged; neither is this module itself (pinned by test). - `isGeneratedFile(path)` is unchanged — cheap, sync, still the fallback. - Schema v9 adds `files.generated` + a PARTIAL index. DDL only, no backfill: the flag derives from content the migration cannot see, so rows stay 0 until a re-index and every reader unions the flag with the path check — an un-migrated index keeps pre-#1500 behavior rather than regressing. Re-index required; noted in the CHANGELOG. - `generatedPredicateFor(paths)` gives ranking a bounded probe + O(1) lookups. Bounded, not cached: no invalidation, so a ranking call can never serve a verdict the last sync already replaced. Wired into explore ranking, findSymbolMatches, findAllSymbols, search (MCP + CLI), the context formatter, and the dominant-file/route-file hygiene filters. Cost (acceptance bar was no measurable index-time regression): a single unanchored `/generat/i` test over the header rejects ~every hand-written file before any line splitting. 4.6 µs/file on client-go (worst case — 82% generated). End-to-end `codegraph init` on client-go, n=3 alternating arms: 5.73s median with detection vs 5.76s path-only baseline; the arms cross over between runs, so the difference is inside run-to-run noise. Scope note: generated status remains a stable TIEBREAK at equal score, exactly where it was. Making it a strong negative signal is CG-10, which this unblocks by making the signal correct and available. Two pre-existing tests hard-coded schema version 8; both now track CURRENT_SCHEMA_VERSION (or the migration table) so future migrations don't require editing them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
49c11fc2e0 |
Self-hosted telemetry on Cloudflare D1 + password-gated admin dashboard (CG-7) (#1497)
* feat(telemetry): D1 schema + migrations for raw events and daily rollups First step of replacing PostHog with self-hosted telemetry on Cloudflare D1. Creates the codegraph-telemetry database binding and the initial migration; no worker code paths change yet (the ingest write path and the nightly rollup cron land next). Schema is raw events plus daily rollups: `events` holds one row per sanitized event with the envelope broken out into columns and event-specific props as JSON; `daily_machines`, `daily_event_counts` and `daily_dim_counts` are the nightly rollups the dashboard reads; `machine_first_seen` and `machine_days` carry the retention cohorts and are never purged. One generic dimension table covers every bar and pie, so a new breakdown is a cron change rather than a migration. The migration is commented as an audit surface, like the rest of this worker — every column, and which dashboard chart each rollup table serves. Three judgment calls worth flagging, all documented in the file: - `events` gets `(day, event)` instead of the separate `(day)` and `(event, day)` indexes. D1 bills a row write per index touched, so a third index on the hot table costs ~97k writes/day, and `(day, event)` is a covering index for plain day-range scans anyway (verified with EXPLAIN QUERY PLAN). - `daily_event_counts` and `daily_dim_counts` carry a `machines` column, and `machine_days` a `prod` flag. The "users by ..." panels and the production-user count are distinct-machine numbers, not event counts, and they are unrecoverable once raw events are purged. - No CHECK constraint on `event`: the worker's allowlist is the source of truth and the write path is fail-silent, so a rejected INSERT would lose data quietly instead of erroring loudly. Volume note in the migration footer: ~30M row writes/month against the 50M included on Workers Paid. Storage is the tighter constraint — raw events grow ~74 MB/day, so retention should start at 90 days (~6.7 GB) rather than 180, which would exceed D1's 10 GB per-database cap. * feat(telemetry): admin dashboard worker — scaffold + shared-password auth New Cloudflare Worker at telemetry-dashboard/, sibling of telemetry-worker/ and bound read-only to the same D1 database. Serves a static frontend plus a JSON API behind a shared password, on stats.getcodegraph.com. Auth is the simplest thing that is actually safe for exactly two users: one password in a secret, compared in constant time over SHA-256 digests, and an HMAC-signed cookie (HttpOnly; Secure; SameSite=Lax; Path=/) with a one-year expiry so you sign in once per browser. The cookie is a signed assertion, not a lookup key — no session store. Its payload carries a fingerprint of the password it was minted against, so rotating ADMIN_PASSWORD signs everyone out. Login attempts are capped at 5/min per IP via a ratelimit binding. Everything is deny-by-default: assets.run_worker_first routes every request through the worker before the static-asset server sees it, so the dashboard HTML, its JS, its CSS and the chart library are all behind the session check. The login page is rendered inline by the worker rather than served from public/, which leaves no "is this file public?" judgement calls in the asset directory. Unauthenticated pages 302 to /login, unauthenticated /api/* gets 401. A missing secret fails closed rather than opening the dashboard. scripts/smoke-auth.sh is the regression net — 54 assertions against a throwaway `wrangler dev` covering the gate, cookie flags and persistence, forged/flipped/ truncated cookies, open-redirect refusal, brute-force capping, and password rotation invalidating live sessions. Refs CG-11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(telemetry-dashboard): simplify the chart-library probe in the shell Refs CG-11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(telemetry): nightly rollup cron + raw-event retention purge (CG-10) Adds a scheduled() handler to the ingest worker that recomputes daily_event_counts / daily_dim_counts / daily_machines for the just-completed UTC day plus a 2-day overlap (late-arriving offline buffers), then purges raw events past the retention window. Rollup writes are idempotent upserts, so a re-run never double-counts. Also adds an ADMIN_TOKEN-guarded POST /admin/rollup?day=YYYY-MM-DD for backfill/repair, and drops the PostHog forwarding path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(telemetry): dashboard charts — SQL API over D1 + the Chart.js views (CG-12, CG-13) Replaces the scaffold page with the dashboard proper: 19 panels covering every view of the PostHog dashboard this retires, driven by one filter row. src/api.ts is the read API CG-12 specified: /api/{meta,summary,timeseries, breakdown,activation,retention}, all range-scoped, all parameterized against a closed set of dims and metrics, all shaped labels[] + datasets[] so the frontend does no arithmetic. Rollups answer everything except the activation funnel, which needs raw events and says where they start. The frontend splits into a DOM-free panel registry (public/panels.js) and the page that mounts it (public/app.js), so the render check can drive the same registry the browser rendered from. Panels fail alone, refetch dims rather than flashing, and every chart carries a table twin. Two numbers are labelled rather than rounded off: range-wide "users" per dimension is machine-days (the rollups cannot give distinct machines, and per-day counts are taken as the largest single-event count so one machine's install + index + usage is not counted three times), and recent activation and retention cohorts are marked as still-converting instead of drawn as a cliff. Both colour scales were run through the data-viz validator against the panel surface, not picked by eye; the results are recorded in public/theme.js. Verification, all against the committed fixture (12 machines over 10 days, every expected number worked out by hand from the events, not recorded from a run): scripts/smoke-api.sh 98 assertions scripts/render-check.mjs 79 assertions — real Chromium over CDP, no new deps scripts/smoke-auth.sh 54 assertions (unchanged, still green) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(telemetry): cutover runbook + the end-to-end gate that de-risks it (CG-14) The account-level steps of the PostHog cutover are the maintainer's to run, so this lands the runbook they follow and the check that has to pass first. The runbook (telemetry-worker/README.md) walks the six steps in the order that keeps them reversible: Workers Paid → migrate → deploy → watch 24h → verify the first rollup and the dashboard → only then delete POSTHOG_KEY and cancel the subscription. Step 3 records the outgoing version id because `wrangler rollback` is the escape hatch for the whole verification window, and that window is precisely why the PostHog key is deleted last rather than first. The new gate (scripts/smoke-cutover.sh, `npm run smoke:cutover`) covers the one seam nothing else did. Both workers declare the same D1 database_id, so pointing them at a single --persist-to directory runs the real chain: a client batch → the ingest worker → D1 → the nightly rollup → the dashboard API reading the numbers back. Every other suite stops at one link — smoke-ingest at the events table, smoke-rollup at hand-checked SQL, smoke-api at a hand-written fixture that the cron never touched. That left the dimension names the rollup WRITES versus the ones the dashboard READS agreeing by convention across two branches, where a mismatch is silent: no error, no failed request, just a panel reading zero forever. 61 assertions, all 13 dimensions, and three deliberate traps — a ci machine that is active but not a production user, usage_rollup counts that must be summed rather than tallied, and an uninstall's `targets` that must not leak into the install-scoped breakdown. Writing it caught that the activation funnel's denominator is first-seen machines, not install events (deliberate — a reinstall must not re-enter the funnel), so the suite now pins that distinction rather than assuming it. Also rewords the last PostHog reference in dashboard code: a comment justifying the 14-day retention curve by pointing at a dashboard step 6 deletes. The reasoning now stands on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(telemetry): tell the truth about where events are stored (CG-15) The telemetry docs are a privacy contract, and they still described a managed analytics store that no longer receives anything. Replace that with what actually happens now — events land in our own D1 database on Cloudflare, the endpoint makes no outbound requests, raw events are purged after 90 days and only anonymous daily rollups outlive them. This strengthens the guarantee rather than restating it: there is no second party to share with. - TELEMETRY.md: new "Where it is stored" section; the never-collected IP bullet no longer leans on a vendor-side setting to hold. - docs/design/telemetry.md: ingest section rewritten around D1 + the nightly rollup/retention cron; volume math redone on Workers Paid and the D1 quota (storage, not writes, is what sets the 90-day window); new section documenting the dashboard worker and cross-linking it. - Fixed three drifts from the worker allowlist the sweep surfaced: schema_version was still 1, client_name/client_version was still marked "plumbing to add" though session.ts passes it today, and the legacy sqlite_backend field the worker still accepts was undocumented. - telemetry-worker/README.md: step 6 claimed a repo-wide grep came back clean, which this runbook itself falsifies. Added step 7 — deleting the runbook is what makes that grep true, and is the completion check. - smoke-cutover.sh: the vendor guarantee is now asserted by class (no analytics-ingest endpoint referenced) rather than by one vendor's name, so it keeps working once the name is gone. Verified it still catches a planted forwarding URL. 61/61 pass. Retention is documented as 90 days, not the 180 in the task notes: 180 days of raw events exceeds D1's 10 GB per-database cap, and the code purges at 90. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: untrack local Kommandr issue DB and ignore its sqlite artifacts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
f6ac7b36e6 |
fix(mcp): blast radius follows caller chains before claiming no test coverage (#1475) (#1494)
The "no covering tests found" flag only inspected a symbol's direct callers, so helpers exercised transitively by tests (logDebug runs 1,471x under npm test) were reported untested — wrong for ~40% of flagged symbols per the issue's measurement. The check now BFSes up the caller graph (3 hops, 64-lookup budget per entry) and reports indirect coverage as "tested via callers: <files>". When nothing is found it claims only what was measured — "no tests found within 3 caller hops", or the weaker "no test calls this directly" if the budget ran out — and drops the warning glyph. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
38580e0b04 |
fix(python): bare class references produce references edges to classes (#1478) (#1493)
Python's class-as-value idioms (return SomeClass, x = SomeClass, registry dicts, classes passed as arguments) produced no references edges, so callers/impact on a Django/DRF serializer missed the views that consume it. Three gates dropped them: - return_statement was never dispatched by PYTHON_SPEC (kernel mirrored) - the extraction gate (definedHere) collected function/method names only - resolution accepted function/method targets only (matchFunctionRef + the function_ref import fast path) Capture return_statement for Python (single expression; tuple returns not descended), admit same-file CLASS names to the gate, and accept class targets for Python bare identifiers — scoped to Python so the TS/JS KIND FILTER contract is untouched. The docopt false-positive mechanism behind the function-only rule (lowercase locals vs same-named methods) doesn't transfer: methods stay excluded for bare ids, and the same-file/import gate + unique-or-drop rules still apply. Probed on django-rest-framework (~250 files): 559 new references→class edges, 10/10 sampled genuine (serializer_class = AuthTokenSerializer, the ModelSerializer field-mapping registry, aliases, ctor args, isinstance). EXTRACTION_VERSION 24 → 25. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
f2a5df34de |
fix(mcp): never serve a mis-sliced symbol body from a file that drifted from its index (#1474) (#1492)
codegraph_node / codegraph_explore read CURRENT bytes but slice them at INDEXED line ranges; after an un-synced edit that slice can be a DIFFERENT symbol's code served under the requested name — isError: false, introduced by the 'verbatim … do not Read' guarantee. The watcher-based pending (#403) and degraded (#876) banners cannot cover a project reached via projectPath: cross-project instances have no watcher, by construction. Freshness is now verified at the point of emission from data the index already stores: one stat per rendered file (size + floored mtime, the sync fast path's own test), sha256 content-hash compare only on stat mismatch (so a touch/identical rewrite never false-positives), memoized briefly per handler. On drift: - codegraph_node: small files ship WHOLE and CURRENT (Read-parity, still no Read needed); large ones omit the body with an explicit notice steering to the tool's file-read mode or Read. Location/signature stay, flagged as possibly shifted. - codegraph_explore: the whole-file render (already correct by construction) is kept and flagged; adaptive/skeleton/cluster slicing is disabled for drifted files — a too-big drifted file is omitted with a notice instead. The verbatim/do-not-Read header gains a per-file exception, and a trailing note flags shifted line references (flow, blast radius, symbol lists). The guarantee itself is preserved: everything actually rendered is still byte-accurate — drifted files ship whole or not at all, never as a possibly-wrong slice. A re-sync of the target project restores normal output (covered by test). Adds __setLoadCodeGraphForTests (same seam pattern as __setFsWatchForTests) so in-process tests can exercise a genuine cross-project open, which vitest's transform cannot service through the lazy require. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
02c0e2c935 |
fix(db): stop watchdog-killed sessions from leaking the SQLite WAL without bound (#1431) (#1490)
A SIGKILL'd process (the #850 liveness watchdog, OOM, a crash) leaves its WAL on disk; the next session appends to the same file; and nothing ever truncated it — PASSIVE checkpoints fold frames but keep the file at its high-water mark, and the one shrinking path (a clean last-connection close) is exactly what a killed-daemon world never takes. Observed at 25.6 GB on a 5.46 GB DB, growing until the disk filled. - journal_size_limit on every connection: resetting checkpoints now clip the WAL back to the cap instead of leaving it at its high-water mark. - healOversizedWal() fired from every DatabaseConnection.open: off-thread PASSIVE fold + TRUNCATE when the leftover WAL exceeds the cap (64 MB, CODEGRAPH_WAL_HEAL_MB to override). Single-flight per connection with bounded retries — concurrent passes defeat each other (each checkpoint sees the other as a busy reader). - Daemon/direct MCP watchdogs now pass progressPaths (DB + WAL), extending the #1231 slow-disk deferral to the long-lived server so a healthy daemon mid slow statement isn't SIGKILL'd — fewer kills, fewer leaked WALs. - codegraph status shows WAL size (human + JSON) and warns when it dwarfs the DB; daemon.log lines and the watchdog kill notice now carry ISO timestamps so kills can be placed in time. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
0682137a42 |
fix(installer): write the Claude prompt hook as codegraph.cmd on Windows (#1466) (#1489)
The standalone bundle's bin dir exposes only codegraph.cmd, and Claude Code executes UserPromptSubmit hooks through Git Bash, which applies no PATHEXT — so the bare `codegraph prompt-hook` the installer wrote was "command not found" (exit 127) on every prompt. Write the platform-correct spelling, recognize both spellings on uninstall/opt-out, and self-heal an installer-written entry from the other platform in place on install/upgrade re-runs (npx/hand-edited variants stay untouched). Reproduced and validated on the Windows VM: bare form exits 127 under Git Bash on a standalone-only PATH, codegraph.cmd exits 0; full installer suite (165 tests, including the new migration coverage) green on Windows + macOS. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Fable 5
|
490791c07a |
feat(installer): GitHub Copilot targets — VS Code, Copilot CLI, JetBrains
Adds three new installer targets so `codegraph install` can wire the
MCP server into GitHub Copilot surfaces:
- copilot-vscode: .vscode/mcp.json (local) or the VS Code User-dir
mcp.json (global), JSONC-surgical edits, `--path` pinned via
${workspaceFolder} for global installs
- copilot-cli: ~/.copilot/mcp-config.json
- copilot-jetbrains: github-copilot config dir (XDG / %LOCALAPPDATA%)
Detection, install, uninstall, and --print-config are covered for all
three in installer-targets.test.ts, including platform-specific path
resolution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
 YucandGitHub
|
572d22bfbe |
fix(installer): Codex TOML block finder preserves trailing array-of-tables siblings (#1351) (#1370)
The Codex installer's `findNextTableHeader` skipped `[[array-of-tables]]` headers instead of treating them as a block boundary, so any `[[...]]` block after `[mcp_servers.codegraph]` in ~/.codex/config.toml was silently deleted on install/upgrade/uninstall. Now treats both `[...]` and `[[...]]` as boundaries, with a small line lexer so header-shaped text inside multiline strings/arrays isn't mistaken for a boundary. Adds round-trip regression coverage (install → reinstall → uninstall) + CHANGELOG entry. Fixes #1351. Supersedes #624. Thanks @KtzeAbyss. |
||
 github-actions[bot]
|
ea72e1b190 |
docs(changelog): promote [Unreleased] into [1.5.0]
[skip ci] Auto-generated by Release workflow. |
||
|
|
0f1096e238 |
docs: Opus 4.8 benchmark re-validation + release-notes headline (#1399)
README Benchmark Results re-run 2026-07-21 on the current build (Rust kernel + this cycle's resolution overhaul), Claude Opus 4.8, 7 repos, median of 4 runs/arm: 89% fewer tool calls, 60% cheaper, 69% fewer tokens, 20% faster on average, file reads 0-vs-1..24 on ALL seven repos. Per-repo floor effects reported honestly (excalidraw/alamofire wall, okhttp cost wash). Cost note updated to match the measured data. Changelog [Unreleased] headline now co-leads with near-instant sync. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
c74e8b05e0 |
perf(sync): adaptive quick-fire debounce + scoped watcher sync — save-to-graph well under a second at any scale (#1397)
Two changes to the watcher path (the always-on daemon every agent session uses), which previously paid a flat 2s debounce plus a full-tree scan-diff on every save even though the OS events name the exact files: 1. Adaptive debounce: a pending set of ≤2 files fires after a 300ms quiet window; ≥3 keeps the full configured window so agent multi-file bursts coalesce exactly as before. Re-arming preserves trailing-edge semantics; a user-set CODEGRAPH_WATCH_DEBOUNCE_MS remains the authoritative upper bound (quick window never exceeds it, floor 100ms). 2. Scoped sync: watcher-triggered syncs pass their pending paths, and the reconciler stats exactly those — per-path logic identical to the full walk (stat pre-filter, hash confirm, the #1240 removal/resurrection flow) — skipping the O(repo) scan and tracked-load. Strict fallbacks keep the full scan-diff as ground truth: directory removals (#1285 — the events can't name the children), empty pending sets (retry paths), and >500-file storms (branch checkouts, which also self-heal anything event coalescing dropped). filesChecked counts examined PATHS so a deletion-only scoped sync can't mimic the #449 lock-unavailable signature. Measured (warm in-process, the daemon path): dubbo one-file sync work 512→335ms, Swift compiler (27k files) 884→385ms — save-to-fresh-graph ≈0.6-0.7s end-to-end including the quick debounce, from ~2.5-6s perceived before. Gates: scoped-vs-full dumps byte-identical on dubbo AND the Swift compiler; watcher suite 30/30 (3 new: scoped pass-through, dir-removal fallback, quick-fire timing); sync suite 34/34 (4 new scoped-parity cases incl. delete-resurrection and the lock signature); full suite 2,696 ×2 with CODEGRAPH_KERNEL_EXPECT=1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
157c8e735d |
perf(resolution): generation-tagged supertype memo + method owner index — Swift compiler 185→98s, byte-identical (#1395)
The swiftc =2/nm:mc-* attribution located the wall: the getSupertypes conformance walk ran 971,200 times (565s of combined worker time, 581µs each) — every resolveMethodOnType miss re-queried implements/extends edges for every same-named type node, recursing depth-4 through Swift's protocol landscape with no memoization, and post-inference resolveMethodOnType averaged 1,912µs per call. Fix 1 — generation-tagged getSupertypes memo. Supertype edges GROW during the resolution loop (batch k persists its edges BEFORE batch k+1 fans out — the #1320 ordering), so a plain cache would freeze an early batch's emptier answer. Within a batch the edge state is fixed by that same ordering, so memo entries carry a generation that advances at every batch entry point (resolveBatchYielding / resolveListForAdmission — covering the sequential loop, pool workers, sync admission, and the conformance pass); a stale-gen entry recomputes. Behavior-identical to no memo at every point in time; walk invocation counts match the unmemoized run exactly (971,200 / 24,336 / 76,415). Fix 2 — per-(language, method-name) owner index in getMethodMatches: candidates bucket once by their qualifiedName's last two segments (exactly the span the match predicate tests), so a (type, method) query is a map lookup instead of an O(candidates) scan per methodMatchCache miss. ObjC selectors and multi-segment typeNames keep the legacy linear path. Also ships nm:mc-rmot / nm:rmot-supers =2 attribution rows. swiftc: settle 100.2→31.3s, resolveMethodOnType 1,912→202µs, wall 183.5→97.8s (was 185s at the head-to-head; cbm's same-box number is 119.1s). Gates: swiftc old-vs-new dump byte-identical (1,837,235 rows), swiftc pooled-vs-CODEGRAPH_NO_PARALLEL_RESOLVE=1 identical (the generation-semantics risk surface), dubbo old-vs-new identical (49k Java instance-method hits share both paths), Alamofire identical; suite 2,689 ×2 with CODEGRAPH_KERNEL_EXPECT=1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
974e6c8b95 |
perf(resolution): incremental receiver-inference scan memo + compiled-pattern memo — kong −8% more (−23% cumulative), byte-identical (#1392)
The kong/tokio matcher-chain residue attributed (nm:mc-* sub-stage rows, shipped here too): matchMethodCall's cost is ~entirely inferLocalReceiverType — 61µs per miss on kong, 99% miss rate (39k `self:` calls hunting a local declaration Lua never writes), re-scanning the same scope lines for every ref. Two pure memos, both semantics-preserving by construction: - Compiled-pattern memo: localReceiverTypePatterns/phpPropertyTypePatterns built 2-4 fresh RegExp objects per call; patterns are a pure function of (language, receiver) and non-global, so instances are shared via a FIFO-capped map (no per-get mutation — the §7a.6 LRU-churn lesson). - Incremental scan memo: refs for the same (file, scope, receiver) arrive in ~ascending line order and the backward declaration scan is a pure function of immutable file lines — a per-context watermark scans each line once per key (query(c) = highest match in [start..c]; monotonic calls extend the watermark over (hi..c]; non-monotonic calls fall back to the plain bounded scan). componentScoped (CFML/PHP whole-file sweep) is keyed out. States drop with the context's file caches via clearNameMatcherMemos, wired into ReferenceResolver.clearCaches. kong mc-infer misses 61→20µs (2.4s→0.8s combined); fresh index 3.43 → 3.03-3.20s (−8%; 4.07 → 3.14 cumulative with #1391). tokio unchanged (tight scopes). Gates: dubbo (49k Java instance-method HITS ride this scan), kong, tokio, Fusion dumps all byte-identical; suite 2,689 ×2 with CODEGRAPH_KERNEL_EXPECT=1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
abb0a916f7 |
perf(resolution): per-context basename index for Lua/Luau require resolution — kong fresh index −16% (#1391)
The full-README competitor matrix ranked lua/kong as the largest legitimate fresh-index gap (2.71×). Stage attribution (RESOLVE_PROFILE=2) pinned it: resolveLuaRequire ran getAllFiles().filter(endsWith) FOUR times per require ref — ~7.5k string suffix scans each, measured at ~0.9ms/ref, hit or miss (2.7s combined over kong's 3k requires). Replace the per-ref full-list scans with a per-context basename → file-paths index (the cobolCopybookIndexes pattern). Buckets preserve getAllFiles() iteration order, so each suffix's candidate filter yields exactly the array the full scan produced — identical matches, identical stable sort, identical winner, dump-proven. kong: 4.07-4.27 → 3.40-3.63s (−16%). Gates: kong old-vs-new dump byte-identical (157,650 rows), kong pooled-vs-sequential identical, Fusion (luau instance-path requires) old-vs-new identical; suite 2,689 ×2 with CODEGRAPH_KERNEL_EXPECT=1. Also registers cobolCopybookIndexes in clearImportResolverMemos — it was never dropped on cache clears, so post-sync copybook lookups could serve a stale file list; cache-drop is the always-safe direction. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
1aa4de6eaa |
perf(resolution): adaptive pool engagement — projected-settle bar replaces the fixed 150k-ref gate for mid-run boot; tokio −23% (#1390)
The 9-language competitor matrix exposed tokio as the worst fresh-index gap: 77% of its wall was resolution running SEQUENTIALLY — 56k Rust refs sit under the fixed 150k pool gate while costing 36µs each (9× Go's 4µs/ref on prometheus). A ref-count gate can't see per-ref cost. After each sequential batch the loop now projects the remaining sequential settle from the measured rate and boots the pool mid-run when it clears 400ms. The switch rides machinery that already existed: pool boot is async and fan-out engages only when ready, admission order is mode-independent, and the #1320 edges-before-fanout invariant holds at every batch boundary regardless of when the pool arrives. Up-front engagement at >=150k refs is unchanged; 2-core/low-memory hosts still decline inside tryCreate's sizing; CODEGRAPH_NO_PARALLEL_RESOLVE still disables; downgrade permanence is preserved (one engage attempt per run). Measured (n=3, interleaved, caffeinated): tokio 3.06-3.12 → 2.40-2.57s (resolution 2,443→~1,330ms); express (tiny control) unchanged with zero engagements; dubbo unchanged (ref-count path). Gates: tokio + excalidraw adaptive-vs-sequential dumps byte-identical (87,302 / 89,903 rows), dubbo dump identical to the session baseline, suite 2,689 ×2 with CODEGRAPH_KERNEL_EXPECT=1. Known follow-up: sampling the rate mid-first- batch would close the remaining ~0.3s to the forced-engage ceiling. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
082ea65f3a |
perf(synthesis): provably-empty pass gates + prefilters — render/expo/rn/mybatis stop scanning repos they can't match; iface memo (#1389)
Store-arc round 2 (#1388 follow-up). The synthesis pool barrier on dubbo carried ~1.4s of passes that provably could not emit an edge for the project: reactRenderEdges fanned out over every class before checking for a render method (now: one indexed name lookup bounds candidates — not a language gate, Java Litho-style render+setState still matches); expo/rn cross-platform pairing streamed every method row without the languages their edges require (now registry-gated: expo needs swift AND kotlin file-languages, rn needs a JS-family caller for isBridge); mybatis built its full java-method index before discovering there were no mapper-XML methods (now collects the XML side first). ifaceEdges — real work — stops re-fetching a hub interface's methods once per implementer and skips supertype-less classes before any per-class lookup. dubbo warm wall 8.49-8.79 → 8.14-8.24s (n=3/arm, caffeinated); barrier 784→435ms; the full removed pass work lands on low-core envelopes where synthesis runs sequentially. Dumps byte-identical: dubbo old-vs-new, pooled-vs-sequential, kernel-vs-wasm (441,270 rows) + excalidraw JSX-live control (89,903 rows, 46 react-render edges reproduced). Suite 2,689 ×2 with CODEGRAPH_KERNEL_EXPECT=1. Also ships the diagnostics that located the round (zero cost when off): CODEGRAPH_RESOLVE_PROFILE=2 attributes per-ref time to resolveOne's strategies (stage:*) and the name-matcher's sub-matchers (nm:*); CODEGRAPH_SYNTH_TIMINGS now prints the store worker's decode-vs-SQL split. Killed by measurement, recorded in the PR: import-failure negative cache (both-outcome names exist — static imports resolve via instance-method on jvm-miss), jvm-miss early return (1,939 later-strategy edges), jsxEdges language gate (Java generics text produces jsx edges), and §4d buffer→bind on Spring repos (extract() hook forces the decoded path — kernel=0 bundles measured). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
27c3c55436 |
perf(resolution): darwin-honest memory budget — vm_stat-based availability unstrangles the resolver pool on macOS (#1388)
Post-R7b store-arc round 1, found by the dubbo warm-wall decomposition (the cbm bar): resolution's loop-stage profile showed settle=3.0s — the main thread idling on TWO resolver workers on an 11-core Mac. Pool sizing logged `size=2 (budget=1068MB)`: memoryBudgetBytes() falls back to os.freemem() when uncontained, and macOS keeps RAM deliberately full of reclaimable cache, so freemem reads ~1GB on a mostly-idle 64GB machine. The memory term then capped the pool at 2 where the CPU term allowed 6 — the macOS sibling of §7a.1's os.cpus() cpuset-blindness (that round fixed the CPU term; this fixes the memory term). Fix: darwinMemoryAvailable() reads /usr/bin/vm_stat once per sizing call and reports free + inactive + speculative + purgeable pages — what Activity Monitor calls available, the same reclaimable-inclusive convention the Linux branch already uses by crediting inactive_file back. Parse failure → null → freemem fallback; Linux/cgroup and Windows paths untouched. Measured (dubbo 4,402 files, warm, caffeinated, n=3 each): pool now self-sizes to 6 (budget 5.7-6.3GB) — wall 8.62-8.83s vs 9.67-10.87s baseline, resolution phase 6.9→5.3s, loop settle 3.0→1.9s. Matches the CODEGRAPH_RESOLVE_WORKERS=6 probe exactly (probe-before-build). Dumps byte-identical pool-6 vs sequential (441,270 lines). Second consumer unblocked: the cFnPtr LRU cache cap no longer spuriously degrades to 128 on Macs (its full-cache tier is worth ~60s at kernel scale). Suite: resolver-pool-sizing gains a darwin-gated reclaimable-pages test + an off-darwin null pin; full suite 2,689 green ×2 with CODEGRAPH_KERNEL_EXPECT=1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d1b75a1a27 |
feat(kernel): R7b Dart walker — dart module, vendored-grammar-C d4d8f3e + wasm byte-copy vendor, dart default-routed (#1386)
R7b batch 4 #4 — the FINAL R7b language (docs/design/dart-kernel-port-checklist.md is the authoritative quirk list). The fourth vendored-grammar-C language, with a twist: production dart resolved its wasm from tree-sitter-wasms, whose dart dependency is an UNPINNED github:UserNobody14/tree-sitter-dart — a routine dependency update would have silently changed dart's grammar. This PR byte-copies the shipping 0.1.13 artifact into src/extraction/wasm/ (VENDORED_WASM_LANGS += dart) and compiles the same-commit (d4d8f3e337d8) parser.c/scanner.c in the kernel — table identity proven by the kernel-grammar-parity row. crates.io tree-sitter-dart is the nielsenko fork (different lineage) — rejected. The center of gravity is THE SIBLING-BODY DOUBLE-WALK, reproduced bug-for-bug: dart attaches every function/method body as a NEXT SIBLING of its signature, and the TS walkers consume each body TWICE — once via resolveBody (attributed to the function/method) and once via the enclosing generic walk (attributed to the file/class). Duplicate local-function nodes with the SAME id under different parents, duplicated calls/instantiates refs, and file/class-attributed fn-ref twins all emit in the exact observed interleave (a dedicated fixture pins the duplicate-id rows; the bloc kind-census spot-check pins the counts). Also preserved (probe-pinned): the extractBareCall selector matrix (the first callTypes=[] language — cascades completely invisible, `?.` encodes like `.`, the `ConfigT.load()` calls+references double emission with no callee-of-call skip, capitalized-chain `Foo.create().run` re-encode, const-object callee names); the constructor hooks (unnamed ctor skipped, named ctors/factories renamed to the CTOR name with the class as returnType, `@override (T) m()` record-misparse rescued by class-name validation); operator methods minting `method "<anonymous>"`; static_final_declaration constants via the visitNode hook while instance fields mint NOTHING; the prefixed-return-type prefix bug (`other.OtherClass f()` → returnType `other`); enum `with` mixins silent vs `implements` working; anonymous extensions named after the ON type; deferred imports invisible; named-argument callbacks NOT fn-ref-captured (the Flutter `onPressed:` idiom — future accuracy PR, TS-side first); `async*`/`sync*` NOT async; value-refs with the LIVE dart sibling-body pull and the `$X`-vs-`${X}` interpolation asymmetry; dartdoc kept in all three comment forms with the annotation-broken chain. Gates: parity sweeps first-run 0-diff on shelf/bloc/flutter — 5,815 clean files byte-parity, deferrals 10/21/1341 ≈ the survey's 10/21/~1340 (both-arm grammar reality: empty object patterns — the sealed-class idiom — and unnamed `library;` dominate; --max-deferral 0.3); full-init dumps byte-identical ×3 (shelf 7,959 / bloc 40,026 / flutter 1,855,319 dump lines); bloc per-kind node census identical across arms (the double-walk duplicate rows survive the store identically); kernel-dart-parity suite (7 fixtures + in-memory CRLF variants + double-walk duplicate-id pin + generated-file skip pin + two defer pins); full suite 2,688 green ×2 with CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += dart (20 langs — R7b COMPLETE). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
bdd687b49f |
feat(kernel): R7b Scala walker — scala module, vendored-grammar-C master@0aca5d0a6f, scala default-routed (#1385)
R7b batch 4 #3 (docs/design/scala-kernel-port-checklist.md is the authoritative quirk list). The third vendored-grammar-C language and the biggest grammar in the tree (35MB parser.c): the vendored wasm is tree-sitter/tree-sitter-scala master@0aca5d0a6f — a post-v0.26.0 generation sync that is not a release (the 0.26.0 crate is 30 states BEHIND, so a crate pin would be a silent downgrade). NO wasm change: production has parsed with this exact revision since #91 — the kernel-grammar-parity row (ABI 15, 26,650 states, 32 fields, id-by-id tables) is the whole alignment proof. Preserved bug-for-bug (all probe-pinned): the leak-through asymmetries — extension methods mint NO nodes (first def's body calls leak to the enclosing scope, later defs invisible, and the braced form resolves its body field to the `{` TOKEN via first-match-wins field lookup → whole extension invisible); anonymous `new T { … }` template_body members leak to the enclosing scope (findAnonymousClassBody misses template_body); the bodied-vs-bodiless class asymmetry (bodiless headers walk class_parameters → default-value calls emit FROM the class; bodied ones never see them) — plus first-segment import names (`import com.example.C` → `com`), the val/var hook keyed on the enclosing-definition NODE TYPE (object vals → constants/value-ref targets, class/trait/enum/given vals → fields) with consumed initializers, every def routed through extractMethod with the top-level function fallback, nested defs in bodies minting NOTHING (the inverse of kotlin) while body-local classes extract fully, curried signatures keeping only the FIRST parameter list (type params win the `parameters` field), enum cases positioned at the CASE node with invisible params/extends tails, extends with-chains via scalaBaseTypeName, `@deprecated(args)` decorates, the #750 capitalized-chain re-encode (`WidgetS.create().render`), literal-receiver silence, static-member reads AND writes, infix invisibility, `derives` silence, scaladoc retention with the CRLF `\r` pin, full value-reference machinery (shadow prune, last-wins same-name targets, `$X`/`${X}` interpolation reads), and SCALA_SPEC fn-refs (bare ids + postfix eta unwrap + varinit, var-init non-capture). Gates: parity sweeps first-run 0-diff on os-lib/cats/scala3-compiler-src/ scala3-library-src — 1,935 clean files byte-parity, deferrals 0/15/57/116 matching the survey's predictions exactly (scala-3's PHANTOM hasError files — flag-true, zero ERROR nodes, capture-checking `^` — defer on the FLAG); full-init dumps byte-identical ×3 (os-lib, cats, scala3 whole-repo 950,889 dump lines); kernel-scala-parity suite (9 fixtures + 9 in-memory CRLF variants incl. Scala-3 indentation through the external scanner + phantom/real-error defer pins + first-segment/namespace/value-ref pins); full suite 2,669 green ×3 with CODEGRAPH_KERNEL_EXPECT=1 (kernel-scaffold's stays-wasm example moved scala → pascal). DEFAULT_ROUTED += scala (19 langs). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
e32135171e |
feat(kernel): R7b Lua+Luau walker — one lua module, vendored-grammar-C lua v0.4.1, tree-sitter-luau 1.2.0 pin, both default-routed (#1384)
R7b batch 4 #2 (docs/design/lua-luau-kernel-port-checklist.md is the authoritative quirk list). ONE walker for both dialects (ccpp precedent) — the differences are exactly four: luau's type_definition aliases, the `export `-slice isExported hook, the return-type signature suffix, and the grammar handle. Grammar prep is kernel-side only, no wasm change: lua is the SECOND vendored-grammar-C language (the vendored wasm is the v0.4.1 tag, a revision not on crates.io — tag artifacts compiled via build.rs, shas pinned); luau is a plain crate pin =1.2.0 whose tarball is sha-identical to the tag (the swift tag≠crate divergence does not recur). Grammar-parity rows replace the bump gate entirely. Preserved bug-for-bug (all probe-pinned): the require/visitNode-hook ASYMMETRIES (top-level requires — including inside top-level if/for/while — mint import nodes while the identical body-level statement emits `calls "require"`; top-level `local x = foo()` initializers are invisible while global `x = foo()` calls emit), the BFS string-win inside require args (`require(script:WaitForChild("Kid"))` → import Kid) and Roblox instance paths, receiver-QN methods (`M.sub.deep::chained`, `_G::installed`, stack-QN nested globals like `render::leakedGlobal`), the raw-text callee world (colon forms with `self` never stripped, bracket callees, newline-glued chains byte-verbatim, the `(handler)` paren-conversion), LUA_SPEC function-as-value capture with the `M.cb = cb` param-storage skip and first-occurrence dedupe, LuaDoc `---` keeping a leading `- ` plus `--!strict` joining docstring chains (block-comment docstrings keep interior CRLF bytes), variable nodes at the IDENTIFIER with positional value pairing, duplicate same-(kind,name,line) ids, and the lua↔luau isExported wire divergence (lua functions: flag absent; luau functions: present-false; methods: absent in both; variables: present-false in both; `export type`: true). Gates: parity sweeps first-run 0-diff on kong/lazy.nvim/lua-resty-core (lua) + lune/Fusion (luau) — 1,734 clean files byte-parity, deferrals 1/0/0/3/8 matching the survey's both-arm predictions exactly (kong's 1 = a deliberately invalid fixture; luau's = grammar-inherent generic type packs and default type params); full-init dumps byte-identical kernel-vs-wasm ×4 (kong 157,650 dump lines); kernel-lua-parity suite (both torture fixtures + in-memory CRLF variants + glue-chain, duplicate-id, and cross-dialect defer pins + kernel-arm wire-flag pins); full suite 2,647 green ×2 with CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += lua, luau (18 langs). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |