# Slug consistency — unify on transliteration **Date:** 2026-06-09 **Backlog item:** roadmap.md → Infra/dette → "Cohérence des slugs". **Decisions:** canonical = **transliterate** (é→e); architecture = **Rust core + parity-tested TS mirror** (Approach B). --- ## Problem (corrected diagnosis) The roadmap framed this as "analytics drops accents, GSC transliterates." **That is backwards.** Empirically (verified on real terms): | term | `wasm.slugify` (Rust, GSC) | analytics `processor.ts` | `actions.ts` local (manual add) | |---|---|---|---| | éclairage | `clairage` | `clairage` | `eclairage` | | café crème | `caf-cr-me` | `caf-cr-me` | `cafe-creme` | | électricité | `lectricit` | `lectricit` | `electricite` | So **wasm and analytics already agree** (both drop accents); the **outlier is `actions.ts`** (transliterates). Two real defects follow: 1. **Mangled FR slugs.** Rust `slugify` keeps only `is_ascii_alphanumeric()`, so every accented char becomes a separator: "courtier en énergie indépendant" → `courtier-en-nergie-ind-pendant`. 2. **Accented vs. unaccented don't merge.** "électricité" → `lectricit` but "electricite" → `electricite` — different keywords for the same intent. A French user omitting accents (common) splits the keyword. Transliteration is the only behavior that merges them. We make **transliteration canonical** in all three paths and migrate existing data. ## Scope of existing data (dev, measured) 467 keywords total · **231 ids change** (all accented, all on Acieb — real FR data) · **52 collision groups / 112 rows** collapse to a shared id (e.g. `comparateur electricité` + `comparateur electricite` + `comparateur électricité` → one `comparateur-electricite`). The migration must **merge**, not just rename. --- ## 1. Canonical slugify — Rust core `crates/import-core/src/normalize.rs::slugify(s)` transliterates before the existing ASCII filter: - Add dependency `unicode-normalization` (first dep for import-core; install with `sfw cargo add unicode-normalization`). - Decompose `s` to NFD, drop chars with `canonical_combining_class != 0` (the combining diacritics), then run the **existing** keep-`[a-z0-9]` / collapse-to-`-` / trim logic unchanged. - This matches JS `s.normalize("NFD").replace(/\p{Diacritic}/gu, "")` for the practical Latin character set (é/è/ê/à/ç decompose + strip; œ/ß have no decomposition → fall through to the separator rule, same as JS). - `slug_id` (backlink URL slugs) calls the same `slugify` — unchanged call site, behavior stays correct for domains. - Rebuild: `bun run build:wasm` + `bun install` + restart dev server (the wasm is external, not HMR'd — see `[[import-core-wasm-rebuild]]`). **Files:** `crates/import-core/src/normalize.rs`, `crates/import-core/Cargo.toml`, regenerated `src/lib/backlink/import-core/import_core_bg.wasm` (+ `.js`/`.d.ts`). ## 2. App convergence Delete the local `slugify` in `src/lib/keywords/actions.ts` (≈ lines 297-305) and replace its two call sites (manual add, ≈ 320/356) with `wasm.slugify`. After this, **every** app path — GSC sync/discovery/import, keyword import, manual add, exclusions — uses the single Rust `wasm.slugify`. The `org-bootstrap.ts` slugify is for org slugs (different purpose) and is **not** touched. **Files:** `src/lib/keywords/actions.ts`. ## 3. Analytics TS mirror + parity test `services/analytics/src/processor.ts::slugify` becomes the NFD-transliterate version (identical algorithm to the old `actions.ts` one): ```ts function slugify(s: string): string { return s .toLowerCase() .normalize("NFD") .replace(/\p{Diacritic}/gu, "") .replace(/[^a-z0-9]+/g, "-") .replace(/(^-|-$)/g, ""); } ``` A **shared golden fixture** (~25 terms → expected slugs; covers é è ê à ç œ ß, leading accent, apostrophes, collapsed whitespace, leading/trailing separators) is asserted in **both**: - `crates/import-core` Rust tests (against `slugify_public`), and - a new `services/analytics/test/slugify.test.ts`. Identical expected outputs on both sides pin the implementations together; drift fails CI on whichever side regresses. The fixture lives in the spec/plan as the source of truth and is copied verbatim into both test files. **Files:** `services/analytics/src/processor.ts`, `services/analytics/test/slugify.test.ts`, `crates/import-core/src/normalize.rs` (test module). ## 4. Data migration (re-slug + merge) One-off script `src/lib/keywords/migrate-slugs.ts`, run via a package script (pattern of the existing lang migration), **env-guarded** (`MIGRATE_SLUGS=1`) so it never runs implicitly. Algorithm (single transaction, or per-site transactions): 1. Load all `keywords`. 2. Group by `newId = ${siteId}-${wasm.slugify(term)}`. 3. For each group with any id-change or `size > 1`: - **Survivor** = the member already at `newId` if one exists; else the first member, whose row is established at `newId` (insert-new-then-delete-old, never an in-place `id` UPDATE — avoids the FK-without-ON-UPDATE-CASCADE problem). - **Merge** members into the survivor record: - `metrics`: shallow-merge by source key (union; on same source, keep the entry with the newer `updatedAt`, else survivor's). - `trend`: keep the longest array (most months); tie → survivor's. - `stages`: set-union. - `gsc`: coalesce — prefer a non-null with the better (lower) position; else survivor's. - `status`: most-advanced by the existing keyword status rank; `theme`/`intent`/`locale`: coalesce (survivor first, then any non-null); `briefs`: merge by country key. - **Repoint FK children** from every loser id → survivor id: - `keyword_targets`: upsert by (keyword_id, country); on conflict keep the most-advanced status (reuse `STATUS_RANK`), then delete loser rows. - `keyword_shortlist`: insert (survivor, …) `onConflictDoNothing`, delete loser rows. - `content_items.primary_keyword_id`: update loser→survivor. - **Delete** loser keyword rows. 4. `keyword_exclusions` (slug-keyed; the term is not stored): a drop-accent slug like `lectricit-entreprise-devis` can't be transliterated directly, but it **is** recoverable from the keyword set. Build `oldToNew: Map` over all keywords as `oldSlugify(term) → wasm.slugify(term)`, where `oldSlugify` is the drop-accent algorithm replicated locally in the script (`s.toLowerCase().replace(/[^a-z0-9]+/g,"-").replace(/^-+|-+$/g,"")`). Update each exclusion whose `slug` is a key in the map (dedup on the `(siteId, slug)` PK). Exclusions whose original keyword was already hard-deleted can't be re-derived and are left unchanged — a rare, self-healing edge: if that term re-ranks it reappears once and is re-excluded under the new slug. (Acieb's `lectricit-entreprise-devis` maps cleanly — that keyword still exists.) 5. **Idempotent:** after one run all slugs are canonical → no groups >1, no id changes → re-run is a no-op. **Files:** `src/lib/keywords/migrate-slugs.ts`, `src/lib/keywords/merge-keywords.ts` (pure merge fn, unit-tested), `package.json` (script). ## 5. Testing - **Rust:** golden-fixture parity test + transliteration unit cases in `normalize.rs`. - **Analytics:** `slugify.test.ts` asserts the same golden fixture. - **App pure:** `merge-keywords.test.ts` — merging two keyword records yields the expected union (metrics/stages/gsc/status/trend). - **App actions:** update existing manual-add test expectations to the transliterated slug; confirm `actions.ts` no longer defines a local slugify. - **Migration (DB):** integration test — seed a site with colliding keywords (different sources) plus `keyword_targets` (conflicting statuses), `keyword_shortlist`, `content_items`, `keyword_exclusions`; run the migration; assert a single survivor at the canonical id, FK children repointed/deduped, most-advanced target status kept, exclusions deduped. - **Live (d3k, static reads):** re-import an Ahrefs export on Acieb; assert an accented term now shares one keyword id with its GSC counterpart and shows cross-source consensus. ## 6. Prod runbook After deploy: rebuild the wasm in the image (build step), then run `MIGRATE_SLUGS=1