fix(prompt-hook): make the structural-question gate multilingual (#1126) (#1134)

* fix(prompt-hook): fire the structural gate for Latin-script, Cyrillic, and JA/KO prompts (#1126)

The prompt-hook's keyword gate only knew English and simplified-Chinese
keywords, so a structural question in French (or Spanish, German, Italian,
Portuguese, Russian, Japanese, Korean, traditional Chinese) silently
no-op'd unless it happened to contain an identifier-shaped code token —
the #994 symptom, resurfaced for every other language.

Root causes fixed:
- JS \b is ASCII-only: a keyword whose first/last char is accented or
  non-Latin (où, qué, Cyrillic, kana) can never match \bkeyword\b —
  the same mechanism behind #994. Keyword matching now uses Unicode
  lookaround boundaries ((?<![\p{L}\p{N}_]) … (?![\p{L}\p{N}_])).
- Bare-stem English entries never matched their own derived forms
  (\barchitect\b can't match "architecture", \bdepend\b can't match
  "dependencies"). Stems are now matched as word prefixes (leading
  boundary only), which also lets one shared stem cover the Romance/
  Germanic spellings that coincide.
- The "CJK" set was simplified-Chinese-only: Japanese (呼び出し, 仕組み,
  実装 — and 追跡 ≠ 追踪), Korean, and traditional-Chinese terms are now
  in the unsegmented substring set.

Code-token extraction and the graph-verification path are unchanged;
non-structural prose stays a zero-cost no-op in every language.

Fixes #1126

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(prompt-hook): extend the gate to tier-2 languages (VI/TR/ID/PL/UA/NL/CS/RO/HU/EL/Nordics/FI/HI/AR/FA/HE/TH)

The first pass covered the 10 largest languages; this closes the rest of
the major-developer-population set (~29 total). Notable per-language
mechanics the curation had to respect:

- Agglutinative languages (Turkish, Finnish, Hungarian) need stems, not
  exact words — suffixes attach to everything (akışı, riippuu, működik).
- Indonesian me-/di-/ber- prefixes block leading-boundary stems, so
  affixed forms are listed explicitly (memanggil, dipanggil, berfungsi).
- Arabic/Farsi/Hebrew are spaced but proclitics attach to the word
  (وكيف = and-how), so they join the substring class with Thai.
- Ukrainian і/и spellings diverge from Russian (архітектур ≠ архитектур).
- Excluded terms that collide with English or code words: NL "pad",
  SV "var", CS "tok", Catalan "com" (matches every .com domain) — with
  regression tests pinning the exclusions.

Vietnamese was the sharpest gap: spaced Latin with heavy diacritics —
exactly the ASCII-\b failure class #1126 reports.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-02 14:34:17 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 04e23917d0
commit 317e7f4d3d
4 changed files with 292 additions and 22 deletions
+2 -2
View File
@@ -1072,8 +1072,8 @@ program
// Gate: only structural / flow / impact / where-how prompts get context, so
// every other prompt ("fix this typo") stays a zero-cost no-op. Language-aware
// (English + CJK keywords, plus code-shaped tokens) so it fires for non-English
// prompts too (issue #994). A keyword fires on its own; a code-token is only a
// (multilingual keywords, plus code-shaped tokens) so it fires for non-English
// prompts too (#994, #1126). A keyword fires on its own; a code-token is only a
// CANDIDATE — verified against the graph below, so a tech brand ("JavaScript")
// that looks like a symbol but isn't one here doesn't inject spurious context.
const keyworded = hasStructuralKeyword(prompt);