# Keywords — competitor top-10 content scraping (Slice 3)

**Date:** 2026-06-18
**Status:** design (decisions baked in), proceeding to implementation
**Scope:** Slice 3 of the keywords/planning cockpit. Closes the loop on the
original ask: give an agent/writer the real top-10 competitor content
(meta, title, markdown, word count, Hn outline) instead of delegating the scrape.

## Goal

Extend the existing competitor on-page crawl so each `SerpCompetitor` also carries
the page **markdown**, **word count**, and **Hn outline**, and embed that real
competitor analysis in the generated prompt (Slice 1's builder).

## Key decision: JS-only, no rust/wasm

The crawl already runs Firecrawl (`scrapeForOnpage` in `firecrawl.ts`) but only
requests `rawHtml`. Firecrawl can also return **markdown**. So we request markdown
and compute word count + Hn outline from it in **pure TypeScript** — no change to
the rust `import-core` crate, no `build:wasm`, no dev-server restart, fully
unit-testable.

## Existing surface (from inventory)

- Crawl entry: `crawlSerpForKeywordAction(keywordId)` in
  `src/lib/keywords/serp-actions.ts` → `mapPool(competitors, 4, …)` →
  `crawlOnpage(url, now)` in `src/lib/keywords/onpage.ts` → `scrapeForOnpage(url)`
  in `src/lib/keywords/firecrawl.ts` (POST Firecrawl `/v1/scrape`, currently
  `formats: ["rawHtml"]`). Fields merge onto each competitor via spread.
- `SerpCompetitor` / `SerpSnapshot`: `src/lib/keywords/serp-types.ts`; persisted in
  the `keywords.serp` jsonb (lz4-compressed, migration 0024). No DB migration needed.
- Prompt builder: `src/lib/keywords/prompt/build-prompt.ts` — competitor table +
  a delegation section ("scrape each URL yourself"). Bundle
  (`getKeywordPromptBundle`) passes `competitors` through untouched, so new fields
  auto-surface.

## Baked-in decisions

- New `SerpCompetitor` fields are **optional & additive** (`markdown?`,
  `wordCount?`, `hnOutline?`) so existing constructors/fixtures keep compiling.
- Stored markdown is **capped ~8 KB per competitor** (jsonb already lz4 → ~20 KB
  per keyword compressed). `hnOutline` is a **flat list** `{ level, text }[]`.
- Prompt: crawled competitors get real embedded analysis (word count + Hn outline
  + short excerpt); **non-crawled** competitors keep the "scrape yourself"
  delegation. Claude-code keeps its `fetch`/MCP framing.

## Architecture / changes

1. **Type contract** (`serp-types.ts`) — DONE in this branch: add `HeadingNode`
   and optional `markdown` / `wordCount` / `hnOutline` to `SerpCompetitor`.
2. **Pure helper** (`src/lib/keywords/markdown-analysis.ts`, new):
   - `outlineFromMarkdown(md: string): HeadingNode[]` (parse `#`–`######` headings,
     in order; ignore code fences).
   - `wordCountFromMarkdown(md: string): number` (strip md syntax, count tokens).
   - `capMarkdown(md: string, maxBytes = 8192): string` (truncate on a char
     boundary, append an elision marker).
   Pure, deterministic, unit-tested.
3. **Crawl populate** (`firecrawl.ts` + `onpage.ts`):
   - `scrapeForOnpage` requests `formats: ["rawHtml", "markdown"]`; `OnpageScrape`
     gains `markdown: string | null`.
   - `crawlOnpage` / `OnpageFields` gain `markdown` (capped), `wordCount`,
     `hnOutline`, computed via the pure helper. Existing fields unchanged.
4. **Prompt embed** (`build-prompt.ts`): a competitor-detail block for crawled
   competitors (domain — N mots, indented Hn outline, short excerpt); keep the
   delegation wording for uncrawled ones. Update `build-prompt.test.ts`.

Bundle/types (`prompt/types.ts`, `prompt/actions.ts`): no change — `competitors`
carries the new fields automatically.

## Error handling

- Firecrawl returns no markdown → fields stay null (competitor still gets the
  rawHtml-derived fields); no failure.
- Crawl failure → competitor untouched (existing behaviour).
- Markdown cap never splits a multibyte char mid-sequence.

## Testing

- `markdown-analysis.test.ts`: outline (levels, order, code-fence ignore),
  word count, cap (boundary + marker).
- `build-prompt.test.ts` (extend): a bundle with crawled competitors embeds word
  count + Hn outline; uncrawled competitors keep the delegation line.
- Crawl populate: covered by the helper tests + integration typecheck (Firecrawl
  is network-gated; no live call in tests).

## Implementation fan-out (subagents)

Type contract landed first (this commit). Then parallel:
- V1: `markdown-analysis.ts` + tests (pure, independent).
- V2: `firecrawl.ts` + `onpage.ts` populate (uses V1's helper interface).
- V3: `build-prompt.ts` competitor embed + test update (uses the new type fields).
Integrate: typecheck + `bun run test:all` + biome, then commit + ff-merge dev.