# Content SERP — On-page Crawl (Slice 2) Design

- **Date**: 2026-06-08
- **Status**: approved (design)
- **Section**: contenu / planning (content-card sheet → SERP panel)
- **Depends on**: SERP competitors slice 1 (`keywords.serp` jsonb overlay, `SerpCompetitor`, `SerpPanel`) on `dev`.

## Goal

Fill the four on-page columns (**Meta · Img · Alt · JSON-LD**) of each SERP competitor by crawling its URL with the self-hosted **Firecrawl** instance, on demand. Delivers the user's original ask: meta title/description, image count, **alt text per image (per language)**, and JSON-LD / rich-snippet presence — for the top-10 competitors of a keyword's locale.

## Trigger

A **« Crawler on-page »** button in `SerpPanel` (next to « Importer export SERP »), shown when a snapshot exists and Firecrawl is configured (`env.firecrawlEnabled`). Click → server action crawls the snapshot's ≤10 competitor URLs → fills the fields → re-render. No worker infra.

## Data model

Extend the on-page fields of `SerpCompetitor` (`src/lib/keywords/serp-types.ts`). All are `null` today (slice 1 never populated them), so **no migration** — the change is jsonb-shape only.

```ts
title: string | null;            // Firecrawl metadata.title
metaDescription: string | null;  // metadata.description
lang: string | null;             // metadata.language  (the "par langue")
images: number | null;           // total <img> count
imagesWithoutAlt: number | null; // <img> with missing/empty alt
imageAlts: { src: string; alt: string }[] | null; // per image, capped 50 (opt.1)
missingAltSrcs: string[] | null; // srcs of images without alt, capped 50 (opt.3)
hasJsonLd: boolean | null;       // any <script type="application/ld+json">
jsonLdTypes: string[] | null;    // distinct schema.org @type values found
crawledAt: string | null;        // ISO timestamp of the crawl
```

`SerpSnapshot` gains an optional `crawledAt?: string` (snapshot-level last-crawl stamp). `imageAlts` changes from `string[]` to `{src,alt}[]`; update slice-1's `parseSerpExport` null defaults accordingly (`imageAlts: null`, plus `missingAltSrcs/lang/jsonLdTypes/crawledAt: null`).

## Extraction (two sources, mirroring slice 1)

- **Firecrawl** `POST {FIRECRAWL_API_URL}/v1/scrape` with `{ url, formats: ["rawHtml"] }`, `Authorization: Bearer {FIRECRAWL_API_KEY}`. Read `data.metadata.{title,description,language}` and `data.rawHtml`. (Self-hosted mirrors the documented Firecrawl API — **validated live against `crawl.elevaseo.app` before writing the parser**, see plan.)
- **Rust** (`import-core`, rust-maximal): `parse_onpage_html(html) -> OnpageHtml` using the **`tl`** crate (lightweight, wasm-friendly HTML parser). Extracts:
  - `images`: count of `<img>`.
  - per `<img>`: `src` (or `data-src` fallback) + `alt`; empty/missing alt → `imagesWithoutAlt++` and src into `missingAltSrcs`.
  - `imageAlts`: `{src, alt}` for each image (capped 50 in the TS wrapper).
  - `hasJsonLd` + `jsonLdTypes`: presence of `<script type="application/ld+json">` and the `@type` strings inside (best-effort JSON parse; nested/array @type flattened).

Title/description/language come from Firecrawl metadata (robust); images/JSON-LD come from rust parsing `rawHtml`.

## Components / files

```
src/env.ts                            — FIRECRAWL_API_URL/KEY + firecrawlEnabled (DONE)
.env.example                          — Firecrawl vars (DONE)
src/lib/keywords/firecrawl.ts         — scrapeForOnpage(url): {title,description,language,rawHtml}|null
crates/import-core/src/onpage.rs      — parse_onpage_html + wasm export (tl crate)
crates/import-core/Cargo.toml         — add `tl` dependency
src/lib/keywords/onpage.ts            — wrapper: Firecrawl metadata + wasm parse → competitor on-page partial (cap 50)
src/lib/keywords/serp-actions.ts      — + crawlSerpForKeywordAction(keywordId)
src/components/content/serp-panel.tsx — « Crawler on-page » button + filled columns + per-row expand
```

## Flow

`crawlSerpForKeywordAction(keywordId)` (`"use server"`):
1. Guard `env.firecrawlEnabled` → else `{ ok:false, error:"Firecrawl non configuré" }`.
2. Read keyword `serp` + `country` → pick `serp[country]` snapshot. No snapshot → `{ ok:false }`.
3. Crawl the competitors' URLs with a **concurrency pool of 4** (`scrapeForOnpage` → `enrichCompetitor`). A failed URL leaves that competitor's on-page fields `null` and increments a `failed` counter — never aborts the batch.
4. Write back `serp[country]` with enriched competitors + snapshot `crawledAt`; `revalidatePath("/content/planning")`.
5. Return `{ ok:true, crawled, failed }`.

## UI

In `SerpPanel`:
- **« Crawler on-page »** button (disabled when `!firecrawlEnabled` with a hint, or during crawl → « Crawl… »). Errors shown inline like the import error.
- The four columns fill:
  - **Meta**: `✓` when `metaDescription` present (title attr/tooltip = the description); else `—`.
  - **Img**: total count, or `—`.
  - **Alt**: `N sans alt` (red-tinted when > 0), or `—`.
  - **JSON-LD**: `✓` (+ `jsonLdTypes` as small text/tooltip) or `—`.
- **Per-row expand**: clicking a competitor row toggles a detail block listing the **alt text per image** (`src` → `alt`, empty alts flagged), and the `missingAltSrcs` highlighted — this is where the user reads/audits the alts. `crawledAt` shown in the panel footer.

## Error / empty states

- Firecrawl not configured → button disabled + tooltip "Configure FIRECRAWL_API_URL/KEY".
- Whole-crawl failure (all URLs fail) → inline error with `failed` count; existing data untouched.
- Per-URL failure → that competitor stays at `—` (uncrawled); `failed` counter reported.
- Crawl on a snapshot whose competitors were already crawled → re-crawls and overwrites (fresh `crawledAt`).

## Testing

- **Unit (rust)** `onpage.rs`: image count, missing-alt detection + src capture, `imageAlts` pairs, JSON-LD presence + `@type` extraction (object, array, `@graph`), no-img / no-jsonld → zeros/false.
- **Unit (TS)** `onpage.test.ts`: wrapper merges Firecrawl metadata + parsed html → fields; caps `imageAlts`/`missingAltSrcs` at 50.
- **Action test** `serp-actions.test.ts` (extend): `mock.module` the firecrawl client to return canned metadata+html → `crawlSerpForKeywordAction` enriches `serp[locale]` competitors; per-URL failure leaves nulls + counts `failed`; `mock.module("next/cache")`.
- **Manual (d3k)**: with the real key in `.env.local`, open the comparateur card → « Crawler on-page » → columns fill from `crawl.elevaseo.app`; expand a row → real alt texts; `d3k errors` clean.

## Non-goals (YAGNI)

- Background / scheduled crawl worker; auto-crawl on SERP import.
- Screenshots, full-text content, comparison vs. our own page.
- Re-crawl scheduling / freshness tracking beyond the `crawledAt` stamp.
- The slice-1 backlog (URL filtering, chunked import) — tracked separately.