# Semantic-Core Crate (S1+S2) Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Le crate Rust WASM `semantic-core` : extraction sémantique par page (zones exclusives, archétype, stemming multilingue, n-grammes) + scoring SERP agrégé (BM25F, courbes CTR, médianes pondérées, intention auto, guide d'écriture, gap). **Architecture:** Nouveau membre du workspace Cargo, dépend d'`import-core` en rlib (parse `tl`, decode_entities). Deux fonctions `#[wasm_bindgen]` pures — `extract_semantic_page` (1 page → `PageSemantics`) et `build_serp_profile` (N pages → `SemanticProfile`). Aucun I/O dans le crate. Spec : `docs/features/analyse-semantique-serp/CDC_v1.3.md` + `design-technique-phase-b.md` (D1). **Tech Stack:** Rust, `tl 0.7`, `rust-stemmers`, `wasm-bindgen`/`serde-wasm-bindgen`, wasm-pack `--target nodejs`. ## Global Constraints - Poids de zones par défaut (CDC §2.2) : title 3.0, h1 2.5, h2h3 1.5, slug 1.5, alt 1.3, anchor 1.2, meta 1.2, body 1.0 — tous paramétrables, `meta keywords` jamais scorée. - Zones **mutuellement exclusives** : une occurrence comptée une seule fois, précédence `Title > H1 > H2/H3 > alt > ancre > meta description > body`. - BM25F : saturation sur la fréquence pondérée par zone, `k1 = 1.2`, `b = 0.75` (défauts paramétrables). Pas d'IDF (le corpus est la SERP, 10 docs). - Intention auto : seuil de dominance **60 %** du Top 10, sinon `mixed_intent` + courbe standard. - Stemming : jamais le stemmer d'une autre langue ; langue non supportée → repli sans stemming + flag. - Structs serde `#[serde(rename_all = "camelCase")]` ; fonctions wasm : `&str` in, `Result` out via `serde_wasm_bindgen`. - Pas d'I/O, pas de `Date::now`/random dans le crate. - Tests natifs : `cargo test -p semantic-core`. Gate finale : `bun run test:all` vert. - Conventional Commits ; branche **`ytg-like/semantic-core`** depuis `dev`. **AUCUN merge dans `dev`** à la fin de ce plan : la feature complète (S1→S7) vit sur les branches `ytg-like/*` et n'intègre `dev` qu'après validation globale par l'utilisateur. --- ### Task 1: Scaffold du crate + branchement build **Files:** - Create: `crates/semantic-core/Cargo.toml` - Create: `crates/semantic-core/src/lib.rs` - Modify: `Cargo.toml` (racine — members) - Modify: `package.json` (scripts + dépendance `file:`) - Modify: `next.config.ts` (serverExternalPackages) - Modify: `deploy/deploy.sh` (2ᵉ build wasm AVANT bun install) - Create: `src/lib/semantic/semantic-core/.gitignore` **Interfaces:** - Produces: crate compilable `semantic-core`, importable en JS comme `"semantic-core"` (wasm nodejs), script `build:wasm:semantic`. - [ ] **Step 1: Créer la branche** ```bash git checkout dev && git pull && git checkout -b ytg-like/semantic-core ``` - [ ] **Step 2: Manifest du crate** `crates/semantic-core/Cargo.toml` : ```toml [package] name = "semantic-core" version = "0.1.0" edition = "2021" [lib] crate-type = ["cdylib", "rlib"] [dependencies] wasm-bindgen = "0.2" serde = { version = "1", features = ["derive"] } serde_json = "1" serde-wasm-bindgen = "0.6" tl = "0.7" url = "2" rust-stemmers = "1.2" import-core = { path = "../import-core" } ``` `Cargo.toml` racine : ```toml [workspace] members = ["crates/import-core", "crates/gsc-worker", "crates/semantic-core"] resolver = "2" ``` - [ ] **Step 3: lib.rs sanity + test** ```rust // crates/semantic-core/src/lib.rs — moteur d'analyse sémantique SERP (CDC v1.3). // Pur : aucune I/O. Le fetch/R2/DB restent côté TS (design Phase B, D1). pub fn crate_ready() -> bool { true } #[cfg(test)] mod tests { #[test] fn sanity() { assert!(super::crate_ready()); } } ``` - [ ] **Step 4: Vérifier la compile native** Run: `cargo test -p semantic-core` Expected: `test tests::sanity ... ok` - [ ] **Step 5: Branchement build wasm + Next** `package.json` — scripts (à côté de `build:wasm`) et dépendance : ```json "build:wasm:semantic": "wasm-pack build crates/semantic-core --release --target nodejs --out-dir ../../src/lib/semantic/semantic-core --out-name semantic_core", ``` ```json "semantic-core": "file:src/lib/semantic/semantic-core", ``` `src/lib/semantic/semantic-core/.gitignore` : contenu `*` (mirror d'import-core — artefacts jamais committés). `next.config.ts` : ajouter `"semantic-core"` au tableau `serverExternalPackages` existant. `deploy/deploy.sh` : juste après le bloc `wasm-pack build … import-core` existant (lignes ~42-46), même forme : ```bash wasm-pack build crates/semantic-core --release --target nodejs \ --out-dir ../../src/lib/semantic/semantic-core --out-name semantic_core ``` - [ ] **Step 6: Builder le wasm et réinstaller** Run: `bun run build:wasm:semantic && bun install` Expected: dossier `src/lib/semantic/semantic-core/` peuplé (`semantic_core.js`, `.d.ts`, `_bg.wasm`), install OK. - [ ] **Step 7: Commit** ```bash git add -A && git commit -m "feat(semantic): scaffold du crate semantic-core (workspace + build wasm + branchement Next/deploy)" ``` --- ### Task 2: Modèle de données + config (défauts CDC) **Files:** - Create: `crates/semantic-core/src/model.rs` - Create: `crates/semantic-core/src/config.rs` - Modify: `crates/semantic-core/src/lib.rs` (mod) **Interfaces:** - Produces (consommé par toutes les tâches suivantes) : - `Zone` enum : `Title, H1, H2H3, Slug, Alt, Anchor, Meta, Body` ; - `ZoneCounts` : `{ title, h1, h2h3, slug, alt, anchor, meta, body: u32 }` (+ `weighted(&ZoneWeights) -> f64`, `total() -> u32`) ; - `PageSemantics` : `{ lang, lang_supported, archetype, term_freqs: BTreeMap, bigram_freqs, trigram_freqs: BTreeMap, structure: PageStructure, questions: Vec, weighted_len: f64 }` ; - `PageStructure` : `{ word_count, h2_count, hn_depth, lists, tables, paragraphs: u32, schema_types: Vec }` ; - `SemanticConfig` : poids de zones + `k1`, `b`, `top_n_terms` (200), `intent_threshold` (0.6), `curve_override: Option` — `Default` = valeurs CDC ; `SemanticConfig::from_json(&str)` tolérant (champs absents → défauts) ; - `Archetype` enum : `Transactional, Editorial, Unknown`. - [ ] **Step 1: Test de config (défauts + parse partiel)** ```rust // dans config.rs #[cfg(test)] mod tests { use super::*; #[test] fn defaults_match_cdc() { let c = SemanticConfig::default(); assert_eq!(c.weights.title, 3.0); assert_eq!(c.weights.slug, 1.5); assert_eq!(c.weights.body, 1.0); assert_eq!(c.k1, 1.2); assert_eq!(c.intent_threshold, 0.6); assert_eq!(c.top_n_terms, 200); } #[test] fn partial_json_keeps_defaults() { let c = SemanticConfig::from_json(r#"{"weights":{"title":4.0}}"#).unwrap(); assert_eq!(c.weights.title, 4.0); assert_eq!(c.weights.h1, 2.5); // défaut conservé } } ``` - [ ] **Step 2: Vérifier l'échec** — Run: `cargo test -p semantic-core` → erreurs de compile (types absents). - [ ] **Step 3: Implémenter model.rs + config.rs** `config.rs` (structure ; le parse partiel passe par `#[serde(default)]` champ à champ) : ```rust use serde::{Deserialize, Serialize}; #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(rename_all = "camelCase", default)] pub struct ZoneWeights { pub title: f64, pub h1: f64, pub h2h3: f64, pub slug: f64, pub alt: f64, pub anchor: f64, pub meta: f64, pub body: f64, } impl Default for ZoneWeights { fn default() -> Self { Self { title: 3.0, h1: 2.5, h2h3: 1.5, slug: 1.5, alt: 1.3, anchor: 1.2, meta: 1.2, body: 1.0 } } } #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(rename_all = "camelCase", default)] pub struct SemanticConfig { pub weights: ZoneWeights, pub k1: f64, pub b: f64, pub top_n_terms: usize, pub intent_threshold: f64, pub curve_override: Option, } impl Default for SemanticConfig { fn default() -> Self { Self { weights: ZoneWeights::default(), k1: 1.2, b: 0.75, top_n_terms: 200, intent_threshold: 0.6, curve_override: None } } } impl SemanticConfig { pub fn from_json(s: &str) -> Result { if s.trim().is_empty() { return Ok(Self::default()); } serde_json::from_str(s) } } ``` `model.rs` : les structs de l'interface ci-dessus, toutes `#[derive(Debug, Clone, Serialize, Deserialize)] #[serde(rename_all = "camelCase")]`, `BTreeMap` (ordre stable → tests déterministes). `ZoneCounts::weighted` = somme des champs × poids correspondant ; `total` = somme brute. - [ ] **Step 4: Vérifier** — Run: `cargo test -p semantic-core` → PASS. - [ ] **Step 5: Commit** — `git commit -m "feat(semantic): modèle de données + config défauts CDC v1.3"` --- ### Task 3: Tokenisation, stop-words, stemming, n-grammes **Files:** - Create: `crates/semantic-core/src/tokenize.rs` - Modify: `crates/semantic-core/src/lib.rs` (mod) **Interfaces:** - Produces: - `Lang` enum (`En, Fr, Es, De, It, Pt, Nl, Unsupported`) + `Lang::from_html_lang(&str) -> Lang` (sous-tag primaire : `"fr-FR"` → `Fr`) ; - `Tokenizer::new(lang: Lang)` — porte stop-words + stemmer de LA langue (ou aucun si `Unsupported`) ; - `tokenize(&self, text: &str) -> Vec` : lowercase, split unicode alphanumérique, filtre stop-words, stemme chaque token, filtre tokens < 2 chars et purement numériques ; - `tokenize_keep_stops(&self, text: &str) -> Vec` : idem sans filtre stop-words ni stemming (pour n-grammes lisibles) ; - `ngrams(tokens: &[String], n: usize) -> Vec` (fenêtre glissante, jointure par espace) ; - `is_question(text: &str, lang: Lang) -> bool` (préfixes interrogatifs par langue). - [ ] **Step 1: Tests** ```rust #[cfg(test)] mod tests { use super::*; #[test] fn french_stemming_merges_singular_plural() { let t = Tokenizer::new(Lang::Fr); assert_eq!(t.tokenize("cheval"), t.tokenize("chevaux")); } #[test] fn stopwords_filtered_by_lang() { let t = Tokenizer::new(Lang::Fr); assert!(t.tokenize("le chat et la souris").iter().all(|w| w != "le" && w != "et" && w != "la")); } #[test] fn unsupported_lang_no_stemming() { let t = Tokenizer::new(Lang::from_html_lang("ja")); assert_eq!(t.tokenize("fabricants"), vec!["fabricants"]); // brut, pas le stemmer FR } #[test] fn html_lang_subtag() { assert!(matches!(Lang::from_html_lang("fr-FR"), Lang::Fr)); assert!(matches!(Lang::from_html_lang("EN"), Lang::En)); } #[test] fn bigrams() { let toks: Vec = ["assurance", "auto", "cher"].iter().map(|s| s.to_string()).collect(); assert_eq!(ngrams(&toks, 2), vec!["assurance auto", "auto cher"]); } #[test] fn question_detection() { assert!(is_question("Comment choisir son assurance ?", Lang::Fr)); assert!(is_question("How to choose insurance", Lang::En)); assert!(!is_question("Notre catalogue", Lang::Fr)); } } ``` - [ ] **Step 2: Vérifier l'échec** — `cargo test -p semantic-core` → compile errors. - [ ] **Step 3: Implémenter** - Mapping stemmers : `rust_stemmers::Algorithm::{English, French, Spanish, German, Italian, Portuguese, Dutch}`. - Stop-words : constantes embarquées par langue (`const STOP_FR: &[&str] = &["le","la","les","de","des","du","un","une","et","ou","à","au","aux","en","dans","sur","pour","par","avec","ce","cette","ces","que","qui","ne","pas","est","sont","son","sa","ses","plus","d","l","se","il","elle","on","nous","vous","ils","elles","être","avoir", /* ~80 mots total */]` — même approche pour EN/ES/DE/IT/PT/NL, listes ~60-100 mots standard Snowball). Zéro dépendance externe. - Préfixes interrogatifs : FR `comment/pourquoi/quel(le)(s)/où/quand/combien/qui/que`, EN `how/why/what/where/when/which/who`, ES/DE/IT/PT/NL équivalents ; test = premier token lowercase du texte ∈ liste. - Tokenisation : itération `char`, mot = suite de `char::is_alphanumeric` (+ `'` traité comme séparateur), lowercase via `to_lowercase()`. - [ ] **Step 4: Vérifier** — `cargo test -p semantic-core` → PASS. - [ ] **Step 5: Commit** — `git commit -m "feat(semantic): tokenisation multilingue, stop-words, stemming, n-grammes (CDC §3.7)"` --- ### Task 4: Extraction des zones exclusives depuis le HTML **Files:** - Create: `crates/semantic-core/src/zones.rs` - Modify: `crates/semantic-core/src/lib.rs` (mod) **Interfaces:** - Consumes: `tl` (parse une-passe, pattern import-core `onpage.rs`), `import_core` helpers publics si exposés (sinon : dupliquer `decode_entities` localement est INTERDIT — passer par un `pub use`/fonction publique ajoutée à `import-core` dans cette tâche, changement minimal). - Produces: `ZoneExtract { title: String, meta_description: String, h1: Vec, h2h3: Vec, alts: Vec, anchors: Vec, body_text: String, lang: String, schema_types: Vec, hn_depth: u32, h2_count: u32, lists: u32, tables: u32, paragraphs: u32, has_main_or_article: bool, product_grid_signals: u32 }` et `extract_zones(html: &str, archetype_hint: Archetype) -> ZoneExtract` ; `slug_tokens(url: &str) -> Vec`. - [ ] **Step 1: Tests (fixtures HTML inline)** ```rust #[cfg(test)] mod tests { use super::*; use crate::model::Archetype; const HTML: &str = r#"Fabricant de bijoux

Fabricants de bijoux

Guide du bijou

Le bijou artisanal en acier.

bague acier
Mentions légales bijoux
"#; #[test] fn nested_anchor_in_h2_counted_once_in_h2() { let z = extract_zones(HTML, Archetype::Editorial); assert!(z.h2h3.iter().any(|h| h.contains("Guide du bijou"))); assert!(!z.anchors.iter().any(|a| a.contains("Guide du bijou"))); // précédence H2 > ancre } #[test] fn nav_and_footer_excluded_from_body() { let z = extract_zones(HTML, Archetype::Editorial); assert!(!z.body_text.contains("Accueil")); assert!(!z.body_text.contains("Mentions")); assert!(z.body_text.contains("artisanal")); } #[test] fn alt_and_meta_extracted() { let z = extract_zones(HTML, Archetype::Editorial); assert_eq!(z.alts, vec!["bague acier"]); assert_eq!(z.meta_description, "Grossiste bijoux acier"); assert_eq!(z.lang, "fr"); } #[test] fn slug_tokenized() { assert_eq!( slug_tokens("https://ex.com/fabricant-bijoux/acier_inox?utm=1"), vec!["fabricant", "bijoux", "acier", "inox"] ); } #[test] fn transactional_keeps_product_grid() { let shop = r#"
"#; let z = extract_zones(shop, Archetype::Transactional); // en transactionnel les textes de la grille restent dans les ancres (zone anchor) assert!(z.anchors.iter().any(|a| a.contains("Bague or"))); } } ``` - [ ] **Step 2: Vérifier l'échec** — `cargo test -p semantic-core` → compile errors. - [ ] **Step 3: Implémenter** Walker DOM (pattern `onpage.rs`, garde profondeur 256 + fallback plat) avec contexte de zone : - Exclusion structurelle TOUJOURS : `nav, footer, header, aside, script, style, noscript, svg, template` + `role="navigation"|"banner"|"contentinfo"`. - **Éditorial** : si `
` ou `
` existe, le body_text ne provient que de ce sous-arbre (approximation Readability déterministe, design D1). - **Transactionnel** : tout le body hors exclusions structurelles (la grille produits est du contenu, CDC §3.0). - Précédence pendant le walk : dans `

` → h1 ; dans `

/

` → h2h3 ; sinon dans `` → anchors ; texte restant → body_text. `alt` collectés sur `` (`src`/`data-src` non requis ici). `title`/`meta[name=description]` depuis ``. Entités décodées (helper import-core). - Compteurs : `h2_count`, `hn_depth` (max), `lists` (`ul|ol`), `tables`, `paragraphs` (`p`), `schema_types` (réutiliser la logique JSON-LD d'import-core via fonction publique, sinon parse local des `script[type=application/ld+json]` → `@type`). - `product_grid_signals` : nombre de `
  • ` contenant à la fois `` et `` (consommé par Task 5). - `slug_tokens` : `url::Url::parse` → path, percent-decode, split `- / _`, lowercase, filtre vides/numériques purs. - [ ] **Step 4: Vérifier** — `cargo test -p semantic-core` → PASS. - [ ] **Step 5: Commit** — `git commit -m "feat(semantic): extraction zones exclusives + slug + compteurs structure (CDC §2.2, §3.0)"` --- ### Task 5: Détection d'archétype **Files:** - Create: `crates/semantic-core/src/archetype.rs` - Modify: `crates/semantic-core/src/lib.rs` (mod) **Interfaces:** - Consumes: `ZoneExtract.schema_types`, `ZoneExtract.product_grid_signals`, `ZoneExtract.has_main_or_article`. - Produces: `detect_archetype(schema_types: &[String], product_grid_signals: u32, word_count: u32) -> Archetype`. - [ ] **Step 1: Tests** ```rust #[cfg(test)] mod tests { use super::*; use crate::model::Archetype; fn s(v: &[&str]) -> Vec { v.iter().map(|x| x.to_string()).collect() } #[test] fn schema_product_wins() { assert!(matches!(detect_archetype(&s(&["Product", "BreadcrumbList"]), 0, 800), Archetype::Transactional)); } #[test] fn schema_article_wins() { assert!(matches!(detect_archetype(&s(&["Article"]), 0, 1500), Archetype::Editorial)); } #[test] fn heuristic_grid_fallback() { // pas de schema : ≥ 8 items de grille et peu de texte → transactionnel assert!(matches!(detect_archetype(&s(&[]), 12, 300), Archetype::Transactional)); } #[test] fn heuristic_text_fallback() { assert!(matches!(detect_archetype(&s(&[]), 1, 1200), Archetype::Editorial)); } #[test] fn unknown_when_ambiguous() { assert!(matches!(detect_archetype(&s(&[]), 3, 150), Archetype::Unknown)); } } ``` - [ ] **Step 2: Vérifier l'échec** — compile errors attendues. - [ ] **Step 3: Implémenter** — priorité Schema.org (CDC §3.0) : transactionnel si ∩ `{Product, Offer, CollectionPage, ItemList}` ; éditorial si ∩ `{Article, NewsArticle, BlogPosting, FAQPage, HowTo}` ; sinon heuristique : `product_grid_signals >= 8 && word_count < 600` → Transactional ; `word_count >= 600` → Editorial ; sinon Unknown. - [ ] **Step 4: Vérifier** — PASS. - [ ] **Step 5: Commit** — `git commit -m "feat(semantic): détection d'archétype schema-first + heuristique (CDC §3.0)"` --- ### Task 6: `extract_semantic_page` (assemblage + surface wasm) **Files:** - Create: `crates/semantic-core/src/page.rs` - Modify: `crates/semantic-core/src/lib.rs` (mod + `#[wasm_bindgen]`) **Interfaces:** - Consumes: Tasks 2-5. - Produces: - Rust : `analyze_page(html: &str, final_url: &str, config: &SemanticConfig) -> PageSemantics` — pipeline : `extract_zones` (2 passes : hint Unknown → `detect_archetype` → ré-extraction si l'archétype change la stratégie body) → tokenisation par zone (langue de la page) → `term_freqs: BTreeMap` (slug inclus) → bigrams/trigrams (sur tokens avec stop-words retirés mais NON stemmés, fenêtre par zone body) → `questions` (Hn interrogatifs) → `weighted_len` = Σ_z len_tokens(z) × W_z ; - WASM : `extract_semantic_page(html: &str, final_url: &str, config_json: &str) -> Result`. - [ ] **Step 1: Test d'intégration** ```rust #[cfg(test)] mod tests { use super::*; use crate::config::SemanticConfig; const PAGE: &str = r#"Fabricant bijoux acier

    Fabricant de bijoux

    Comment choisir son fabricant ?

    Un fabricant de bijoux propose des bijoux en acier. Les bijoux artisanaux…

    "#; #[test] fn full_page_analysis() { let cfg = SemanticConfig::default(); let p = analyze_page(PAGE, "https://ex.com/fabricant-bijoux", &cfg); let t = crate::tokenize::Tokenizer::new(crate::tokenize::Lang::Fr); let stem = t.tokenize("bijoux").remove(0); let zc = p.term_freqs.get(&stem).expect("terme présent"); assert_eq!(zc.title, 1); assert_eq!(zc.h1, 1); assert_eq!(zc.slug, 1); // /fabricant-bijoux assert!(zc.body >= 2); assert_eq!(zc.meta, 1); assert_eq!(p.questions.len(), 1); assert!(p.lang_supported); assert!(p.weighted_len > 0.0); } } ``` - [ ] **Step 2: Vérifier l'échec.** — compile errors. - [ ] **Step 3: Implémenter** `page.rs` + exposer dans `lib.rs` : ```rust #[wasm_bindgen] pub fn extract_semantic_page(html: &str, final_url: &str, config_json: &str) -> Result { let cfg = SemanticConfig::from_json(config_json).map_err(|e| JsValue::from_str(&e.to_string()))?; let page = page::analyze_page(html, final_url, &cfg); serde_wasm_bindgen::to_value(&page).map_err(|e| JsValue::from_str(&e.to_string())) } ``` - [ ] **Step 4: Vérifier** — `cargo test -p semantic-core` PASS, puis `bun run build:wasm:semantic` OK. - [ ] **Step 5: Commit** — `git commit -m "feat(semantic): extract_semantic_page — analyse complète d'une page (S1 done)"` --- ### Task 7: Scoring — courbes W_P, BM25F, agrégats **Files:** - Create: `crates/semantic-core/src/scoring.rs` - Modify: `crates/semantic-core/src/lib.rs` (mod) **Interfaces:** - Consumes: `PageSemantics`, `SemanticConfig`. - Produces: - `position_weight(pos: u32, curve: &Curve) -> f64` — courbes `Standard/Informational/Transactional/Navigational` (tables CDC §2.1) + fallback `1.0 / (pos as f64 + 1.0).log2()` pour pos > 10 ; - `bm25f(ftf_weighted: f64, weighted_len: f64, avg_weighted_len: f64, k1: f64, b: f64) -> f64` = `ftf_n / (k1 + ftf_n)` avec `ftf_n = ftf_weighted / (1.0 - b + b * weighted_len / avg_weighted_len)` ; - `TermAggregate { term, score, median_weighted, is_outlier, zones: ZoneCounts /* sommes Top10 */, top3: TermRange { min, median, max } }` ; - `aggregate_terms(pages: &[(u32, &PageSemantics)], cfg: &SemanticConfig) -> Vec` (trié score desc, tronqué `top_n_terms`). - [ ] **Step 1: Tests mathématiques** ```rust #[cfg(test)] mod tests { use super::*; #[test] fn saturation_caps_stuffing() { // 200 répétitions ne valent pas 200× une répétition (CDC §2.2) let one = bm25f(1.0, 100.0, 100.0, 1.2, 0.75); let stuffed = bm25f(200.0, 100.0, 100.0, 1.2, 0.75); assert!(stuffed < one * 3.0); assert!(stuffed < 1.0); // asymptote } #[test] fn length_normalization_penalizes_long_docs() { let short = bm25f(5.0, 50.0, 100.0, 1.2, 0.75); let long = bm25f(5.0, 400.0, 100.0, 1.2, 0.75); assert!(short > long); } #[test] fn position_one_outweighs_ten() { let c = Curve::Standard; assert!(position_weight(1, &c) > position_weight(10, &c) * 5.0); assert_eq!(position_weight(1, &c), 1.0); } #[test] fn outlier_flagged_when_one_page_dominates() { // construire 3 pages : le terme n'apparaît (massivement) que chez pos 3 // → contribution unique > 60 % du score → is_outlier // (fixture via PageSemantics construits à la main) let aggs = aggregate_fixture_one_dominant(); assert!(aggs.iter().find(|t| t.term == "outlier_term").unwrap().is_outlier); } } ``` (La fixture `aggregate_fixture_one_dominant()` construit des `PageSemantics` minimaux à la main — pas de HTML.) - [ ] **Step 2: Vérifier l'échec.** - [ ] **Step 3: Implémenter** — courbes en tables constantes (pos 1-10, valeurs CDC §2.1 ; interpolation linéaire 4-5 et 6-10), médiane pondérée (tri des scores par page, cumul des W_P jusqu'à 50 %), outlier si `max_contribution / total > 0.6`, `top3` = min/médiane/max des fréquences brutes totales (`ZoneCounts::total`) sur les pages de position ≤ 3. - [ ] **Step 4: Vérifier** — PASS. - [ ] **Step 5: Commit** — `git commit -m "feat(semantic): scoring BM25F + courbes CTR + agrégats et outliers (CDC §2)"` --- ### Task 8: `build_serp_profile` — intention, couverture, guide, gap **Files:** - Create: `crates/semantic-core/src/profile.rs` - Modify: `crates/semantic-core/src/model.rs` (structs profil) - Modify: `crates/semantic-core/src/lib.rs` (wasm) **Interfaces:** - Consumes: Tasks 2-7. - Produces: - Entrée : `PageInput { position: u32, url: String, included: bool, fetch_ok: bool, semantics: Option }` — le TS envoie `serde_json::to_string(&Vec)` ; - `SemanticProfile` (shape du design §3, sans `analyzedAt/jobId/exports` — ajoutés côté TS) : `intent { curve, mode, mixed_intent }`, `coverage { analyzed, total, excluded_by_user: Vec, missing: Vec, flags: Vec }`, `terms: Vec`, `ngrams { bigrams, trigrams }`, `questions`, `structure` (percentiles word_count p25/median/p75, moyennes h2/lists/tables, `schema_types` %, `archetypes` %, comparatif top3 vs top10), `guide` (termes avec fourchettes top3 + zone recommandée = zone au max de freq, ngrams top, questions, objectifs structurels = médianes), `gap: Option` ; - Intention : archétypes des pages incluses → dominance ≥ `intent_threshold` → courbe ; `curve_override` → mode `manual` ; sinon `inferred` ; mixte → `Standard` + `mixed_intent: true` ; - Renormalisation : W_P calculés sur les positions **d'origine**, division par Σ W_P des pages analysées (CDC §6 : le n°3 ne devient pas n°2) ; - Gap : `client: Option` en 3ᵉ argument ; deltas par terme (`expected_score - client_score`, tri desc), `intent_mismatch` si archétype client ≠ dominant, delta structurel ; - WASM : `build_serp_profile(pages_json: &str, client_json: &str, config_json: &str) -> Result` (`client_json` vide → pas de gap). - [ ] **Step 1: Tests** ```rust #[cfg(test)] mod tests { use super::*; #[test] fn intent_transactional_over_threshold() { // 7/10 pages Transactional → curve transactionnelle, pas de mixed_intent let p = build_profile(fixture_pages(7, 3), None, &SemanticConfig::default()); assert_eq!(p.intent.curve, "transactionnelle"); assert!(!p.intent.mixed_intent); assert_eq!(p.intent.mode, "inferred"); } #[test] fn mixed_intent_flagged() { let p = build_profile(fixture_pages(5, 5), None, &SemanticConfig::default()); assert!(p.intent.mixed_intent); assert_eq!(p.intent.curve, "standard"); } #[test] fn coverage_separates_user_exclusion_from_missing() { // pos 2 included=false ; pos 7 fetch_ok=false let p = build_profile(fixture_with_holes(), None, &SemanticConfig::default()); assert_eq!(p.coverage.excluded_by_user, vec![2]); assert_eq!(p.coverage.missing, vec![7]); } #[test] fn gap_intent_mismatch() { let client = fixture_editorial_client(); let p = build_profile(fixture_pages(8, 2), Some(client), &SemanticConfig::default()); let gap = p.gap.unwrap(); assert!(gap.intent_mismatch); // Article vs SERP transactionnelle assert!(!gap.terms.is_empty()); } } ``` (fixtures = `PageSemantics` construits à la main, pattern Task 7.) - [ ] **Step 2: Vérifier l'échec.** - [ ] **Step 3: Implémenter** `profile.rs` + surface wasm + noms de courbes sérialisés en français (`"transactionnelle"`, `"informationnelle"`, `"standard"`, `"navigationnelle"` — cohérence CDC/UI). - [ ] **Step 4: Vérifier** — `cargo test -p semantic-core` PASS ; `bun run build:wasm:semantic` OK. - [ ] **Step 5: Commit** — `git commit -m "feat(semantic): build_serp_profile — intention, couverture, guide, gap (S2 done)"` --- ### Task 9: Types TS + wrapper + gate finale **Files:** - Create: `src/lib/semantic/types.ts` (miroir TS des structs camelCase : `PageSemantics`, `SemanticProfile`, `SemanticConfig`, `PageInput` — pattern `OnpageDoc` de `src/lib/keywords/onpage.ts`) - Create: `src/lib/semantic/engine.ts` (wrapper typé) - Create: `src/lib/semantic/engine.test.ts` **Interfaces:** - Produces (consommé par S3 — le runner du pipeline) : - `extractSemanticPage(html: string, finalUrl: string, config?: Partial): PageSemantics` - `buildSerpProfile(pages: PageInput[], client: PageSemantics | null, config?: Partial): SemanticProfile` - [ ] **Step 1: Test bun** ```typescript // src/lib/semantic/engine.test.ts import { describe, expect, test } from "bun:test"; import { buildSerpProfile, extractSemanticPage } from "./engine"; const HTML = `Fabricant bijoux

    Fabricant de bijoux

    Bijoux en acier pour revendeurs. Bijoux gravés.

    `; describe("semantic engine (wasm)", () => { test("extractSemanticPage renvoie zones et structure", () => { const p = extractSemanticPage(HTML, "https://ex.com/fabricant-bijoux"); expect(p.langSupported).toBe(true); expect(Object.keys(p.termFreqs).length).toBeGreaterThan(0); expect(p.structure.wordCount).toBeGreaterThan(0); }); test("buildSerpProfile agrège et pondère", () => { const page = extractSemanticPage(HTML, "https://ex.com/fabricant-bijoux"); const profile = buildSerpProfile( [{ position: 1, url: "https://ex.com/fabricant-bijoux", included: true, fetchOk: true, semantics: page }], null, ); expect(profile.coverage.analyzed).toBe(1); expect(profile.terms.length).toBeGreaterThan(0); expect(profile.terms[0].score).toBeGreaterThan(0); }); }); ``` - [ ] **Step 2: Vérifier l'échec** — Run: `bun run test src/lib/semantic/engine.test.ts` → module absent. - [ ] **Step 3: Implémenter** `engine.ts` : ```typescript // src/lib/semantic/engine.ts — wrapper typé du wasm semantic-core. // Serveur uniquement (wasm nodejs, serverExternalPackages). import * as wasm from "semantic-core"; import type { PageInput, PageSemantics, SemanticConfig, SemanticProfile } from "./types"; export function extractSemanticPage( html: string, finalUrl: string, config: Partial = {}, ): PageSemantics { return wasm.extract_semantic_page(html, finalUrl, JSON.stringify(config)) as PageSemantics; } export function buildSerpProfile( pages: PageInput[], client: PageSemantics | null, config: Partial = {}, ): SemanticProfile { return wasm.build_serp_profile( JSON.stringify(pages), client ? JSON.stringify(client) : "", JSON.stringify(config), ) as SemanticProfile; } ``` - [ ] **Step 4: Gates** Run: `bun run test src/lib/semantic/engine.test.ts` → PASS Run: `cargo test -p semantic-core` → PASS Run: `bun run test:all` → PASS (aucune régression) Run: `bun run lint` → clean - [ ] **Step 5: Commit + push de la branche (PAS de merge dev)** ```bash git add -A && git commit -m "feat(semantic): wrapper TS typé + tests bun du moteur wasm" git push -u origin ytg-like/semantic-core ``` La branche reste ouverte : les phases suivantes (S3 pipeline, S4+ UI…) continuent sur `ytg-like/*`. Merge dans `dev` uniquement sur validation globale de l'utilisateur. --- ## Self-review (fait à l'écriture) - Couverture spec S1+S2 : zones exclusives ✔ (T4), slug ✔ (T4), archétype ✔ (T5), mobile-UA = hors crate (S3, fetch TS) ✔ n/a, stemming/stop-words/questions ✔ (T3), n-grammes ✔ (T3/T6), BM25F + courbes + médiane + outliers ✔ (T7), intention/couverture/guide/gap ✔ (T8), co-occurrences — **reportées à S5** (nécessitent une déf. produit fine ; le champ existe dans le profil, vide en S1+S2, documenté ici). - Types cohérents entre tâches : `ZoneCounts`/`PageSemantics`/`SemanticConfig` définis T2, consommés T4-T9 sous les mêmes noms. - Chaque tâche : test → fail → impl → pass → commit.