# Content SERP On-page Crawl (Slice 2) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add a "Crawler on-page" button to the SERP panel that crawls each competitor URL via the self-hosted Firecrawl, filling Meta/Img/Alt(per-image)/JSON-LD for the keyword's locale snapshot.

**Architecture:** Firecrawl `/v1/scrape` returns `metadata` (title/description/language) + `rawHtml`; a new rust `parse_onpage_html` (`import-core`, `tl` crate) extracts image/alt counts, per-image alts, and JSON-LD `@type`s from the html. A TS wrapper merges both into the existing `SerpCompetitor` on-page fields; a server action crawls ≤10 URLs (concurrency 4) and writes the enriched snapshot back to `keywords.serp[locale]`. No migration (jsonb shape change only).

**Tech Stack:** Rust `tl` HTML parser → wasm; Next 16 server actions + `fetch`; Drizzle/Postgres jsonb; React client; bun:test + cargo test.

---

### Task 1: Extend SERP types + slice-1 wrapper null defaults

**Files:**
- Modify: `src/lib/keywords/serp-types.ts`
- Modify: `src/lib/keywords/serp-import.ts`
- Modify: `src/lib/keywords/serp-import.test.ts`

- [ ] **Step 1: Update the types**

In `src/lib/keywords/serp-types.ts`, replace the `SerpCompetitor` on-page block and add `ImageAlt` + snapshot `crawledAt`:

```ts
// src/lib/keywords/serp-types.ts — competitor SERP snapshot stored per locale on a keyword.
export interface ImageAlt {
  src: string;
  alt: string;
}

export interface SerpCompetitor {
  position: number; // organic rank
  domain: string;
  url: string;
  backlinks: number | null; // per-URL backlinks (decision: backlinks vers l'URL)
  refDomains: number | null; // "Ref.Domains"
  pageAuthority: number | null; // "Page AS"
  searchTraffic: number | null; // "Search Traffic"
  urlKeywords: number | null; // "URL Keywords"
  // slice 2 (crawl) — null until "Crawler on-page" runs:
  title: string | null;
  metaDescription: string | null;
  lang: string | null; // page language (Firecrawl metadata.language)
  images: number | null;
  imagesWithoutAlt: number | null;
  imageAlts: ImageAlt[] | null; // per-image {src, alt}, capped 50
  missingAltSrcs: string[] | null; // srcs of images without alt, capped 50
  hasJsonLd: boolean | null;
  jsonLdTypes: string[] | null; // distinct schema.org @type values
  crawledAt: string | null;
}

export interface SerpSnapshot {
  source: "se-ranking";
  importedAt: string; // ISO date
  features: string[]; // distinct non-organic SERP feature types (PAA, Video…)
  competitors: SerpCompetitor[]; // top 10 organic, sorted by position asc
  crawledAt?: string; // ISO date of the last on-page crawl
}
```

- [ ] **Step 2: Update the slice-1 wrapper to set the new null defaults**

In `src/lib/keywords/serp-import.ts`, update the `.map` in `parseSerpExport` to include every new on-page field as null:

```ts
  const competitors: SerpCompetitor[] = raw.competitors.map((c) => ({
    ...c,
    title: null,
    metaDescription: null,
    lang: null,
    images: null,
    imagesWithoutAlt: null,
    imageAlts: null,
    missingAltSrcs: null,
    hasJsonLd: null,
    jsonLdTypes: null,
    crawledAt: null,
  }));
```

- [ ] **Step 3: Update the wrapper test for the new field**

In `src/lib/keywords/serp-import.test.ts`, the existing assertion `expect(snap.competitors[0].imageAlts).toBeNull();` still holds. Add one line after it:

```ts
    expect(snap.competitors[0].missingAltSrcs).toBeNull();
```

- [ ] **Step 4: Run the wrapper test + typecheck**

Run: `bun --env-file=.env.local test src/lib/keywords/serp-import.test.ts`
Expected: PASS (1 test). Then `bunx tsc --noEmit` → PASS.

- [ ] **Step 5: Commit**

```bash
git add src/lib/keywords/serp-types.ts src/lib/keywords/serp-import.ts src/lib/keywords/serp-import.test.ts
git commit -m "feat(content): extend SERP competitor types for on-page crawl"
```

---

### Task 2: Rust on-page HTML parser + wasm export

**Files:**
- Create: `crates/import-core/src/onpage.rs`
- Modify: `crates/import-core/src/lib.rs`
- Modify: `crates/import-core/Cargo.toml`

- [ ] **Step 1: Add the `tl` dependency**

In `crates/import-core/Cargo.toml`, under `[dependencies]`, add:

```toml
tl = "0.7"
```

- [ ] **Step 2: Create the parser with tests**

Create `crates/import-core/src/onpage.rs`:

```rust
// crates/import-core/src/onpage.rs — extract on-page SEO signals from raw HTML.
use serde::Serialize;

#[derive(Debug, Clone, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct ImageAlt {
    pub src: String,
    pub alt: String,
}

#[derive(Debug, Clone, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct OnpageHtml {
    pub images: i64,
    pub images_without_alt: i64,
    pub image_alts: Vec<ImageAlt>,
    pub missing_alt_srcs: Vec<String>,
    pub has_json_ld: bool,
    pub json_ld_types: Vec<String>,
}

fn empty() -> OnpageHtml {
    OnpageHtml {
        images: 0,
        images_without_alt: 0,
        image_alts: vec![],
        missing_alt_srcs: vec![],
        has_json_ld: false,
        json_ld_types: vec![],
    }
}

/// Collect schema.org `@type` strings from a parsed JSON-LD value
/// (handles a bare object, an array of nodes, and `@graph`).
fn collect_from_value(v: &serde_json::Value, out: &mut Vec<String>) {
    match v {
        serde_json::Value::Array(a) => {
            for e in a {
                collect_from_value(e, out);
            }
        }
        serde_json::Value::Object(o) => {
            if let Some(t) = o.get("@type") {
                match t {
                    serde_json::Value::String(s) => out.push(s.clone()),
                    serde_json::Value::Array(a) => {
                        for e in a {
                            if let Some(s) = e.as_str() {
                                out.push(s.to_string());
                            }
                        }
                    }
                    _ => {}
                }
            }
            if let Some(g) = o.get("@graph") {
                collect_from_value(g, out);
            }
        }
        _ => {}
    }
}

pub fn parse_onpage_html(html: &str) -> OnpageHtml {
    let dom = match tl::parse(html, tl::ParserOptions::default()) {
        Ok(d) => d,
        Err(_) => return empty(),
    };
    let parser = dom.parser();

    let mut images = 0i64;
    let mut images_without_alt = 0i64;
    let mut image_alts = Vec::new();
    let mut missing_alt_srcs = Vec::new();

    if let Some(iter) = dom.query_selector("img") {
        for nh in iter {
            let Some(tag) = nh.get(parser).and_then(|n| n.as_tag()) else {
                continue;
            };
            images += 1;
            let attrs = tag.attributes();
            let src = attrs
                .get("src")
                .flatten()
                .map(|b| b.as_utf8_str().to_string())
                .or_else(|| {
                    attrs
                        .get("data-src")
                        .flatten()
                        .map(|b| b.as_utf8_str().to_string())
                })
                .unwrap_or_default();
            let alt = attrs
                .get("alt")
                .flatten()
                .map(|b| b.as_utf8_str().to_string())
                .unwrap_or_default();
            if alt.trim().is_empty() {
                images_without_alt += 1;
                missing_alt_srcs.push(src.clone());
            }
            image_alts.push(ImageAlt { src, alt });
        }
    }

    let mut has_json_ld = false;
    let mut json_ld_types = Vec::new();
    if let Some(iter) = dom.query_selector("script") {
        for nh in iter {
            let Some(node) = nh.get(parser) else { continue };
            let Some(tag) = node.as_tag() else { continue };
            let is_ld = tag
                .attributes()
                .get("type")
                .flatten()
                .map(|b| b.as_utf8_str().to_string())
                .unwrap_or_default()
                .eq_ignore_ascii_case("application/ld+json");
            if !is_ld {
                continue;
            }
            has_json_ld = true;
            collect_from_value(
                &serde_json::from_str(&node.inner_text(parser)).unwrap_or(serde_json::Value::Null),
                &mut json_ld_types,
            );
        }
    }
    json_ld_types.sort();
    json_ld_types.dedup();

    OnpageHtml {
        images,
        images_without_alt,
        image_alts,
        missing_alt_srcs,
        has_json_ld,
        json_ld_types,
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    const HTML: &str = r#"<html><head>
      <script type="application/ld+json">{"@type":"FAQPage","name":"x"}</script>
      <script type="application/ld+json">{"@graph":[{"@type":"BreadcrumbList"},{"@type":["Product","Thing"]}]}</script>
      </head><body>
      <img src="/a.png" alt="logo alpha">
      <img src="/b.png" alt="">
      <img data-src="/c.png">
      </body></html>"#;

    #[test]
    fn counts_images_and_missing_alt() {
        let r = parse_onpage_html(HTML);
        assert_eq!(r.images, 3);
        assert_eq!(r.images_without_alt, 2);
        assert_eq!(r.image_alts.len(), 3);
        assert_eq!(r.image_alts[0].alt, "logo alpha");
        assert_eq!(r.missing_alt_srcs, vec!["/b.png", "/c.png"]);
    }

    #[test]
    fn extracts_json_ld_types() {
        let r = parse_onpage_html(HTML);
        assert!(r.has_json_ld);
        assert_eq!(
            r.json_ld_types,
            vec!["BreadcrumbList", "FAQPage", "Product", "Thing"]
        );
    }

    #[test]
    fn no_images_no_jsonld() {
        let r = parse_onpage_html("<html><body><p>hi</p></body></html>");
        assert_eq!(r.images, 0);
        assert!(!r.has_json_ld);
        assert!(r.json_ld_types.is_empty());
    }
}
```

- [ ] **Step 3: Register module + wasm export in `crates/import-core/src/lib.rs`**

Add `mod onpage;` with the other `mod` lines, and this export after `parse_serp_urls_csv`:

```rust
#[wasm_bindgen]
pub fn parse_onpage_html(html: &str) -> Result<JsValue, JsValue> {
    let res = onpage::parse_onpage_html(html);
    serde_wasm_bindgen::to_value(&res).map_err(|e| JsValue::from_str(&e.to_string()))
}
```

- [ ] **Step 4: Run rust tests**

Run: `cargo test --manifest-path crates/import-core/Cargo.toml onpage`
Expected: PASS (3 tests). If `tl` fails to compile for the wasm target later, that surfaces in Step 5 — report BLOCKED with the error.

- [ ] **Step 5: Rebuild wasm + relink + verify export**

Run:
```bash
bun run build:wasm
bun install
grep parse_onpage_html src/lib/backlink/import-core/import_core.d.ts
```
Expected: `bun run build:wasm` succeeds and the grep prints one match.

- [ ] **Step 6: Restart d3k so the new wasm loads** (controller task — the dev server caches wasm)

Run:
```bash
pkill -f dev3000; sleep 2
cd /Users/humperdoo/Github/seo-insight && d3k --startup-timeout 60 --browser-navigation-timeout 60 --browser-tool agent-browser --date-time local --agent-name claude-yolo --no-tui > /tmp/d3k-launch.log 2>&1 &
sleep 13; grep -m1 "ready\|Your App" /tmp/d3k-launch.log
```
Expected: "Development environment ready!".

- [ ] **Step 7: Commit**

```bash
git add crates/import-core/src/onpage.rs crates/import-core/src/lib.rs crates/import-core/Cargo.toml
git commit -m "feat(content): rust on-page HTML parser (images/alt/JSON-LD) + wasm export"
```

---

### Task 3: Firecrawl client

**Files:**
- Create: `src/lib/keywords/firecrawl.ts`

- [ ] **Step 1: Write the client**

```ts
// src/lib/keywords/firecrawl.ts — scrape one URL via the self-hosted Firecrawl /v1/scrape.
import { env } from "@/env";

export interface OnpageScrape {
  title: string | null;
  description: string | null;
  language: string | null;
  rawHtml: string;
}

interface ScrapeResponse {
  data?: {
    rawHtml?: string;
    metadata?: { title?: string; description?: string; language?: string };
  };
}

export async function scrapeForOnpage(url: string): Promise<OnpageScrape | null> {
  const base = env.FIRECRAWL_API_URL;
  const key = env.FIRECRAWL_API_KEY;
  if (!base || !key) return null;
  try {
    const res = await fetch(`${base}/v1/scrape`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${key}`,
      },
      body: JSON.stringify({ url, formats: ["rawHtml"] }),
    });
    if (!res.ok) return null;
    const json = (await res.json()) as ScrapeResponse;
    const d = json.data;
    if (!d?.rawHtml) return null;
    return {
      title: d.metadata?.title ?? null,
      description: d.metadata?.description ?? null,
      language: d.metadata?.language ?? null,
      rawHtml: d.rawHtml,
    };
  } catch {
    return null;
  }
}
```

- [ ] **Step 2: Validate the real Firecrawl response shape (if key is set)**

If `FIRECRAWL_API_KEY` is present in `.env.local`, confirm the instance matches this shape before relying on it. Run:
```bash
bun --env-file=.env.local -e 'const {scrapeForOnpage}=await import("./src/lib/keywords/firecrawl.ts"); console.log(JSON.stringify((await scrapeForOnpage("https://opera-energie.com/comparateur/electricite-pro/")), (k,v)=>k==="rawHtml"&&typeof v==="string"?v.slice(0,120):v, 2))'
```
Expected: an object with `title`/`description`/`language` and a `rawHtml` snippet. If the key isn't set yet, skip — the action degrades gracefully and this is re-checked live in Task 7. If the shape differs (e.g. metadata under a different path), adjust `ScrapeResponse` accordingly and note it.

- [ ] **Step 3: Typecheck**

Run: `bunx tsc --noEmit`
Expected: PASS.

- [ ] **Step 4: Commit**

```bash
git add src/lib/keywords/firecrawl.ts
git commit -m "feat(content): Firecrawl scrape client (env-gated)"
```

---

### Task 4: On-page wrapper (`crawlOnpage`)

**Files:**
- Create: `src/lib/keywords/onpage.ts`
- Test: `src/lib/keywords/onpage.test.ts`

- [ ] **Step 1: Write the failing test**

```ts
// src/lib/keywords/onpage.test.ts
import { describe, expect, it, mock } from "bun:test";

mock.module("./firecrawl", () => ({
  scrapeForOnpage: async (url: string) =>
    url.includes("fail")
      ? null
      : {
          title: "Titre",
          description: "Desc",
          language: "fr",
          rawHtml:
            '<img src="/a.png" alt="x"><img src="/b.png" alt=""><script type="application/ld+json">{"@type":"FAQPage"}</script>',
        },
}));

const { crawlOnpage } = await import("./onpage");

describe("crawlOnpage", () => {
  it("merges Firecrawl metadata + parsed html into on-page fields", async () => {
    const f = await crawlOnpage("https://ok.test/", "2026-06-08T00:00:00.000Z");
    expect(f).not.toBeNull();
    expect(f?.title).toBe("Titre");
    expect(f?.lang).toBe("fr");
    expect(f?.images).toBe(2);
    expect(f?.imagesWithoutAlt).toBe(1);
    expect(f?.imageAlts?.[0]).toEqual({ src: "/a.png", alt: "x" });
    expect(f?.missingAltSrcs).toEqual(["/b.png"]);
    expect(f?.hasJsonLd).toBe(true);
    expect(f?.jsonLdTypes).toContain("FAQPage");
    expect(f?.crawledAt).toBe("2026-06-08T00:00:00.000Z");
  });

  it("returns null when the scrape fails", async () => {
    expect(await crawlOnpage("https://fail.test/", "x")).toBeNull();
  });
});
```

- [ ] **Step 2: Run it to verify it fails**

Run: `bun --env-file=.env.local test src/lib/keywords/onpage.test.ts`
Expected: FAIL ("Cannot find module './onpage'").

- [ ] **Step 3: Write the wrapper**

```ts
// src/lib/keywords/onpage.ts — crawl one competitor URL → its on-page SerpCompetitor fields.
import * as wasm from "import-core";
import { scrapeForOnpage } from "./firecrawl";
import type { ImageAlt, SerpCompetitor } from "./serp-types";

interface OnpageHtml {
  images: number;
  imagesWithoutAlt: number;
  imageAlts: ImageAlt[];
  missingAltSrcs: string[];
  hasJsonLd: boolean;
  jsonLdTypes: string[];
}

export type OnpageFields = Pick<
  SerpCompetitor,
  | "title"
  | "metaDescription"
  | "lang"
  | "images"
  | "imagesWithoutAlt"
  | "imageAlts"
  | "missingAltSrcs"
  | "hasJsonLd"
  | "jsonLdTypes"
  | "crawledAt"
>;

const CAP = 50;

export async function crawlOnpage(
  url: string,
  nowIso: string,
): Promise<OnpageFields | null> {
  const scrape = await scrapeForOnpage(url);
  if (!scrape) return null;
  const html = wasm.parse_onpage_html(scrape.rawHtml) as OnpageHtml;
  return {
    title: scrape.title,
    metaDescription: scrape.description,
    lang: scrape.language,
    images: html.images,
    imagesWithoutAlt: html.imagesWithoutAlt,
    imageAlts: html.imageAlts.slice(0, CAP),
    missingAltSrcs: html.missingAltSrcs.slice(0, CAP),
    hasJsonLd: html.hasJsonLd,
    jsonLdTypes: html.jsonLdTypes,
    crawledAt: nowIso,
  };
}
```

- [ ] **Step 4: Run the test to verify it passes**

Run: `bun --env-file=.env.local test src/lib/keywords/onpage.test.ts`
Expected: PASS (2 tests). (Requires Task 2 wasm rebuild.)

- [ ] **Step 5: Commit**

```bash
git add src/lib/keywords/onpage.ts src/lib/keywords/onpage.test.ts
git commit -m "feat(content): on-page crawl wrapper (Firecrawl + wasm parse)"
```

---

### Task 5: `crawlSerpForKeywordAction`

**Files:**
- Modify: `src/lib/keywords/serp-actions.ts`
- Test: `src/lib/keywords/serp-actions.test.ts`

- [ ] **Step 1: Write the failing DB test (append to the existing describe block)**

In `src/lib/keywords/serp-actions.test.ts`, at the very top of the file (before any import) set the Firecrawl env so the action's guard passes, and mock the wrapper so no real network/wasm is needed:

```ts
process.env.FIRECRAWL_API_URL = "https://crawl.test";
process.env.FIRECRAWL_API_KEY = "test-key";
mock.module("./onpage", () => ({
  crawlOnpage: async (url: string, nowIso: string) =>
    url.includes("opera")
      ? {
          title: "T",
          metaDescription: "D",
          lang: "fr",
          images: 3,
          imagesWithoutAlt: 1,
          imageAlts: [{ src: "/a", alt: "x" }],
          missingAltSrcs: ["/b"],
          hasJsonLd: true,
          jsonLdTypes: ["FAQPage"],
          crawledAt: nowIso,
        }
      : null, // non-opera URL "fails"
}));
```

(Keep the existing `mock.module("server-only"...)` / `mock.module("next/cache"...)` lines.) Then add this test inside the `describe.skipIf(!RUN_DB)(...)` block:

```ts
  test("crawlSerpForKeywordAction enriches competitors and counts failures", async () => {
    await db.update(schema.keywords).set({
      serp: {
        FR: {
          source: "se-ranking",
          importedAt: "2026-06-08T00:00:00.000Z",
          features: [],
          competitors: [
            { position: 1, domain: "opera-energie.com", url: "https://opera-energie.com/x", backlinks: 1, refDomains: 1, pageAuthority: 1, searchTraffic: 1, urlKeywords: 1, title: null, metaDescription: null, lang: null, images: null, imagesWithoutAlt: null, imageAlts: null, missingAltSrcs: null, hasJsonLd: null, jsonLdTypes: null, crawledAt: null },
            { position: 2, domain: "other.com", url: "https://other.com/y", backlinks: 0, refDomains: 0, pageAuthority: 0, searchTraffic: 0, urlKeywords: 0, title: null, metaDescription: null, lang: null, images: null, imagesWithoutAlt: null, imageAlts: null, missingAltSrcs: null, hasJsonLd: null, jsonLdTypes: null, crawledAt: null },
          ],
        },
      },
    }).where(schema.eq(schema.keywords.id, "kw-1"));

    const res = await actions.crawlSerpForKeywordAction("kw-1");
    expect(res.ok).toBe(true);
    expect(res.crawled).toBe(1);
    expect(res.failed).toBe(1);
    const serp = await actions.getSerpForKeywordAction("kw-1");
    expect(serp.FR.competitors[0].images).toBe(3);
    expect(serp.FR.competitors[0].jsonLdTypes).toContain("FAQPage");
    expect(serp.FR.competitors[1].images).toBeNull(); // failed URL untouched
    expect(serp.FR.crawledAt).toBeTruthy();
  });
```

Notes:
- Add `import { eq } from "drizzle-orm";` to the static imports at the top of the file, and use `eq(schema.keywords.id, "kw-1")` (the `schema.eq(...)` written above is shorthand — replace it with `eq(...)`).
- The `process.env.FIRECRAWL_*` lines must sit at the very top so they run before the dynamic `await import("./serp-actions")` in `beforeEach` (which transitively loads `@/env`). Because `@/env` is evaluated once per process, **run `serp-actions.test.ts` standalone** (as the steps below do), not folded into a multi-file directory run, or `firecrawlEnabled` may be cached `false`.

- [ ] **Step 2: Run it to verify it fails**

Run: `bun run test src/lib/keywords/serp-actions.test.ts`
Expected: FAIL (`crawlSerpForKeywordAction` is not a function).

- [ ] **Step 3: Implement the action**

In `src/lib/keywords/serp-actions.ts`, add the env import and the action. Add to the imports at the top:

```ts
import { env } from "@/env";
import { crawlOnpage } from "./onpage";
```

Add a small concurrency helper and the action at the end of the file:

```ts
/** Run `fn` over `items` with at most `limit` in flight; preserves order. */
async function mapPool<T, R>(
  items: T[],
  limit: number,
  fn: (item: T, index: number) => Promise<R>,
): Promise<R[]> {
  const out = new Array<R>(items.length);
  let next = 0;
  async function worker() {
    while (next < items.length) {
      const i = next++;
      out[i] = await fn(items[i], i);
    }
  }
  await Promise.all(
    Array.from({ length: Math.min(limit, items.length) }, worker),
  );
  return out;
}

export async function crawlSerpForKeywordAction(
  keywordId: string,
): Promise<{ ok: boolean; crawled?: number; failed?: number; error?: string }> {
  if (!env.firecrawlEnabled) {
    return { ok: false, error: "Firecrawl non configuré" };
  }
  const [row] = await db
    .select({ serp: keywords.serp, country: keywords.country })
    .from(keywords)
    .where(eq(keywords.id, keywordId));
  if (!row) return { ok: false, error: "Mot-clé introuvable" };

  const locale = row.country || "US";
  const snap = (row.serp ?? {})[locale];
  if (!snap || snap.competitors.length === 0) {
    return { ok: false, error: "Aucune SERP à enrichir" };
  }

  const now = new Date().toISOString();
  let failed = 0;
  const competitors = await mapPool(snap.competitors, 4, async (c) => {
    const fields = await crawlOnpage(c.url, now);
    if (!fields) {
      failed++;
      return c;
    }
    return { ...c, ...fields };
  });

  const next = {
    ...(row.serp ?? {}),
    [locale]: { ...snap, competitors, crawledAt: now },
  };
  await db.update(keywords).set({ serp: next }).where(eq(keywords.id, keywordId));
  revalidatePath("/content/planning");
  return { ok: true, crawled: competitors.length - failed, failed };
}
```

- [ ] **Step 4: Run the test to verify it passes**

Run: `bun run test src/lib/keywords/serp-actions.test.ts`
Expected: PASS (3 tests: the 2 existing import tests + the new crawl test).

- [ ] **Step 5: Commit**

```bash
git add src/lib/keywords/serp-actions.ts src/lib/keywords/serp-actions.test.ts
git commit -m "feat(content): crawlSerpForKeywordAction (concurrency 4, per-URL failure tolerant)"
```

---

### Task 6: SerpPanel — crawl button, filled columns, per-row expand

**Files:**
- Modify: `src/components/content/serp-panel.tsx`

- [ ] **Step 1: Rewrite the component**

Replace the whole file with this (adds `crawl` button, fills the 4 columns, and a per-row expand listing alts):

```tsx
// src/components/content/serp-panel.tsx — competitor SERP table for a keyword locale.
"use client";

import { useRef, useState } from "react";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import {
  crawlSerpForKeywordAction,
  importSerpForKeywordAction,
} from "@/lib/keywords/serp-actions";
import type { SerpCompetitor, SerpSnapshot } from "@/lib/keywords/serp-types";

function fmt(n: number | null): string {
  return n == null ? "—" : n.toLocaleString("fr-FR");
}

export function SerpPanel({
  keywordId,
  snapshot,
  onImported,
}: {
  keywordId: string;
  snapshot: SerpSnapshot | null;
  onImported: () => void;
}) {
  const inputRef = useRef<HTMLInputElement>(null);
  const [busy, setBusy] = useState(false);
  const [crawling, setCrawling] = useState(false);
  const [error, setError] = useState<string | null>(null);
  const [note, setNote] = useState<string | null>(null);
  const [expanded, setExpanded] = useState<string | null>(null);

  async function onFile(e: React.ChangeEvent<HTMLInputElement>) {
    const file = e.target.files?.[0];
    e.target.value = "";
    if (!file) return;
    if (file.size > 1_000_000) {
      setError("Fichier trop lourd (max 1 Mo) — attendu : un export « SERP URLs ».");
      return;
    }
    setBusy(true);
    setError(null);
    setNote(null);
    try {
      const text = await file.text();
      const res = await importSerpForKeywordAction(keywordId, text);
      if (!res.ok) setError(res.error ?? "Import échoué");
      else onImported();
    } catch {
      setError("Import impossible (fichier trop volumineux ou invalide).");
    } finally {
      setBusy(false);
    }
  }

  async function crawl() {
    setCrawling(true);
    setError(null);
    setNote(null);
    try {
      const res = await crawlSerpForKeywordAction(keywordId);
      if (!res.ok) setError(res.error ?? "Crawl échoué");
      else {
        setNote(`Crawl: ${res.crawled} ok${res.failed ? `, ${res.failed} échec(s)` : ""}`);
        onImported();
      }
    } catch {
      setError("Crawl impossible.");
    } finally {
      setCrawling(false);
    }
  }

  return (
    <div className="rounded-lg border p-3">
      <div className="mb-2 flex items-center justify-between gap-2">
        <div className="font-medium text-sm">SERP / Concurrents</div>
        <div className="flex gap-2">
          {snapshot ? (
            <Button size="sm" variant="outline" disabled={crawling} onClick={crawl}>
              {crawling ? "Crawl…" : "Crawler on-page"}
            </Button>
          ) : null}
          <Button
            size="sm"
            variant="outline"
            disabled={busy}
            onClick={() => inputRef.current?.click()}
          >
            {busy ? "Import…" : "Importer export SERP"}
          </Button>
          <input
            ref={inputRef}
            type="file"
            accept=".csv,text/csv"
            className="hidden"
            onChange={onFile}
          />
        </div>
      </div>

      {error ? <p className="mb-2 text-destructive text-xs">{error}</p> : null}
      {note ? <p className="mb-2 text-muted-foreground text-xs">{note}</p> : null}

      {!snapshot ? (
        <p className="text-muted-foreground text-sm">
          Aucune SERP importée. Dépose un export “SERP URLs”.
        </p>
      ) : (
        <>
          {snapshot.features.length > 0 ? (
            <div className="mb-2 flex flex-wrap gap-1">
              {snapshot.features.map((f) => (
                <Badge key={f} variant="secondary">
                  {f}
                </Badge>
              ))}
            </div>
          ) : null}
          <div className="overflow-x-auto">
            <table className="w-full text-xs">
              <thead className="text-muted-foreground">
                <tr className="border-b text-left">
                  <th className="py-1 pr-2">#</th>
                  <th className="py-1 pr-2">Domaine</th>
                  <th className="py-1 pr-2 text-right">Backlinks</th>
                  <th className="py-1 pr-2 text-right">Ref.dom</th>
                  <th className="py-1 pr-2 text-right">Page AS</th>
                  <th className="py-1 pr-2 text-right">Trafic</th>
                  <th className="py-1 pr-2">Meta</th>
                  <th className="py-1 pr-2 text-right">Img</th>
                  <th className="py-1 pr-2 text-right">Alt</th>
                  <th className="py-1">JSON-LD</th>
                </tr>
              </thead>
              <tbody>
                {snapshot.competitors.map((c) => (
                  <CompetitorRows
                    key={c.url}
                    c={c}
                    open={expanded === c.url}
                    onToggle={() =>
                      setExpanded((cur) => (cur === c.url ? null : c.url))
                    }
                  />
                ))}
              </tbody>
            </table>
          </div>
          <p className="mt-2 text-muted-foreground text-[11px]">
            {snapshot.crawledAt
              ? `On-page crawlé le ${new Date(snapshot.crawledAt).toLocaleString("fr-FR")} — clique une ligne pour les alt par image.`
              : "Meta / Img / Alt / JSON-LD : lance « Crawler on-page »."}
          </p>
        </>
      )}
    </div>
  );
}

function CompetitorRows({
  c,
  open,
  onToggle,
}: {
  c: SerpCompetitor;
  open: boolean;
  onToggle: () => void;
}) {
  const crawled = c.crawledAt != null;
  return (
    <>
      <tr className="cursor-pointer border-b last:border-0" onClick={onToggle}>
        <td className="py-1 pr-2">{crawled ? (open ? "▾" : "▸") : ""} {c.position}</td>
        <td className="py-1 pr-2">
          <a
            href={c.url}
            target="_blank"
            rel="noopener noreferrer"
            className="text-primary hover:underline"
            onClick={(e) => e.stopPropagation()}
          >
            {c.domain}
          </a>
        </td>
        <td className="py-1 pr-2 text-right">{fmt(c.backlinks)}</td>
        <td className="py-1 pr-2 text-right">{fmt(c.refDomains)}</td>
        <td className="py-1 pr-2 text-right">{fmt(c.pageAuthority)}</td>
        <td className="py-1 pr-2 text-right">{fmt(c.searchTraffic)}</td>
        <td className="py-1 pr-2" title={c.metaDescription ?? undefined}>
          {c.metaDescription ? "✓" : "—"}
        </td>
        <td className="py-1 pr-2 text-right">{fmt(c.images)}</td>
        <td
          className={`py-1 pr-2 text-right ${c.imagesWithoutAlt ? "text-destructive" : ""}`}
        >
          {c.imagesWithoutAlt == null
            ? "—"
            : `${c.imagesWithoutAlt} sans alt`}
        </td>
        <td className="py-1" title={c.jsonLdTypes?.join(", ") || undefined}>
          {c.hasJsonLd == null ? "—" : c.hasJsonLd ? "✓" : "✗"}
        </td>
      </tr>
      {open && crawled ? (
        <tr className="border-b last:border-0 bg-muted/30">
          <td colSpan={10} className="px-2 py-2">
            <div className="mb-1 text-muted-foreground text-[11px]">
              {c.title ?? "—"}
              {c.lang ? ` · ${c.lang}` : ""}
              {c.jsonLdTypes?.length ? ` · JSON-LD: ${c.jsonLdTypes.join(", ")}` : ""}
            </div>
            {c.imageAlts?.length ? (
              <table className="w-full text-[11px]">
                <tbody>
                  {c.imageAlts.map((img) => (
                    <tr key={img.src} className="align-top">
                      <td className="w-1/2 truncate pr-2 text-muted-foreground">
                        {img.src}
                      </td>
                      <td
                        className={img.alt.trim() ? "" : "text-destructive"}
                      >
                        {img.alt.trim() ? img.alt : "(alt vide)"}
                      </td>
                    </tr>
                  ))}
                </tbody>
              </table>
            ) : (
              <p className="text-muted-foreground text-[11px]">Aucune image.</p>
            )}
          </td>
        </tr>
      ) : null}
    </>
  );
}
```

- [ ] **Step 2: Typecheck + lint**

Run: `bunx tsc --noEmit && bunx biome check --write src/components/content/serp-panel.tsx`
Expected: tsc PASS; biome fixes formatting only.

- [ ] **Step 3: Commit**

```bash
git add src/components/content/serp-panel.tsx
git commit -m "feat(content): on-page crawl button + filled columns + per-image alt expand"
```

---

### Task 7: Live verification (d3k + Chrome) + full sweep

**Files:** none (verification only). Requires `FIRECRAWL_API_KEY` in `.env.local`.

- [ ] **Step 1: Full test + typecheck sweep**

Run:
```bash
bunx tsc --noEmit
cargo test --manifest-path crates/import-core/Cargo.toml onpage
bun --env-file=.env.local test src/lib/keywords/serp-import.test.ts src/lib/keywords/onpage.test.ts
bun run test src/lib/keywords/serp-actions.test.ts
```
Expected: all green.

- [ ] **Step 2: Seed a SERP snapshot + crawl it live**

With the dev DB keyword `ac5b0f4a-c760-459d-80e2-43786951381f-comparateur-energie-pro`: re-import the real `comparateur-energie-pro_serp_urls…csv` (via a tmp script or the UI) so `serp.FR` has 10 competitors, link a content item to it, open the card.

- [ ] **Step 3: Click "Crawler on-page" + verify via d3k agent-browser**

```bash
d3k agent-browser open "http://localhost:3000/content/planning"
# open the card (eval click on the card title), then:
d3k agent-browser eval '(()=>{const b=[...document.querySelectorAll("[role=dialog] button")].find(x=>/Crawler on-page/.test(x.textContent)); b?.click(); return !!b})()'
d3k agent-browser wait 20000
d3k agent-browser screenshot /tmp/onpage-crawled.png
```
Verify (read the screenshot + eval): Img/Alt/JSON-LD columns now show numbers/✓ for crawled rows; Meta shows ✓ with the description as tooltip; clicking a row expands the per-image alt list (empty alts flagged). `note` shows "Crawl: N ok". Run `d3k errors` → no unhandledRejection / 500.

- [ ] **Step 4: Clean up** any throwaway content item / SERP seed used for the check (leave real data only).

---

## Done criteria

- `tl`-based `parse_onpage_html` shipped; cargo onpage tests green; wasm rebuilt.
- `scrapeForOnpage`, `crawlOnpage`, `crawlSerpForKeywordAction` shipped + tested (real-shape validated against `crawl.elevaseo.app`).
- `SerpPanel` has a working "Crawler on-page" button that fills Meta/Img/Alt/JSON-LD and a per-row expand listing alt-per-image; degrades to a friendly error when Firecrawl is unconfigured.
- tsc clean, all touched tests pass, d3k live crawl verified, `d3k errors` clean.
- Branch `feature/content-serp-onpage-crawl` merged into `dev` (ff-only), deleted, pushed; demo data cleaned.
```