# headless-seo-check — design notes

**Nothing here is a contract; `docs/contract.md` is.** This document explains
*why* the contract says what it says: the failure the tool comes from, the
reasoning behind each rule that looks arbitrary until you know its story, and
the limits the tool accepts on purpose. It may name modules; it promises
nothing. `docs/decisions.md` records when each choice was made or reversed.

---

## 1. Where the tool comes from

The tool was born from the deindexation audit of elevaseo.com (May 2026). The
site went from 288 to 79 indexed pages in two months. The audit surfaced a
technical fault: a single `await headers()` in the root `not-found.tsx` had
flipped all 355 routes of the application to dynamic rendering; only 4 pages
were still prerendered. Nothing in the build output said so, the pages looked
identical in a browser, and the HTML source was unchanged.

Being precise about what this tool would **not** have prevented:

- the **trigger** of the deindexation was a Google core update — the tool
  does not see core updates;
- the **root cause** was an editorial corpus produced at scale — the tool
  does not judge content quality; it counts words, it does not read them.

What it would have caught is the **aggravating technical fault** underneath:
invisible in the page source, invisible in the browser, invisible at build
time, and absent from every classic SEO crawler. That is a narrower claim,
and it is the true one. It is also the admission bar for any fifth check: a
real failure that is invisible where people normally look.

## 2. Each diagnostic, and the real bug it derives from

| Diagnostic | Original bug | Why it was invisible | Module |
| --- | --- | --- | --- |
| **Rendering mode** | An `await headers()` in the root `not-found.tsx` flipped 355 routes to dynamic rendering; 4 pages stayed prerendered. | No build warning, identical pages in the browser, unchanged HTML source. Only the response headers said so. | `checks/render` |
| **Indexability** | An `X-Robots-Tag: noindex` set by a middleware, an edge function or a CDN rule left over from a staging deploy, contradicting a `<meta robots>` that says `index`. | The header appears nowhere in the page source; no framework exposes it in a template; and it wins. | `checks/indexability` |
| **hreflang drift** | A framework derives header alternates by swapping a locale prefix; with translated slugs, the derived URL does not exist while the HTML carries the right one. | No validator compares the `Link:` header to the HTML. Those that read only the HTML declare the page perfect. | `checks/hreflang` |
| **Served content** | A page whose server HTML is a shell: Googlebot must execute JavaScript before seeing anything, and that rendering is queued, budgeted, never guaranteed. | The browser shows the full page; it is the HTML **before** hydration that is empty. | `checks/content` |

The rendering verdict rests on response headers and nothing else, which is
what makes it a fact rather than a guess: `no-store` and `private` are the
site telling every CDN and every crawler that each visit costs a render.
It is not a heuristic about a framework's configuration.

## 3. Rationales worth keeping

### 3.1 `js_heavy`, or the CI compromise

A word count alone proves nothing. `example.com` serves twenty-one words and
is a perfectly valid static page; a legal notice runs thirty. Shipping a bare
threshold flagged both — and a tool that fails a build over a thirty-word
legal notice is a tool that gets disabled, after which it catches nothing at
all. So a page is only flagged when it is thin **and** loaded for hydration:
the three `js_heavy` criteria (external bundles, inline bootstrap bytes, an
empty framework root container) all measure the same intention — *this page
expects to be filled in later* — and a short static page triggers none of
them. The stated trade-off, from the module header of `checks/content`:

> A page that is genuinely thin **and** static is not flagged. That is
> accepted on purpose: this check runs in CI, where a false alarm costs more
> than an omission.

`no-internal-links` is gated on the same signal for the same reason: a static
leaf page that links nowhere is a legal notice or a thank-you page, and
flagging it would fire on half of every healthy site. A bundled page with no
links in its HTML is a different animal — its navigation exists, it is just
not in the HTML, so a crawler that does not run JavaScript arrives and stops.

`no-server-text` (Info, exactly zero words, no bundle) is a refinement of
the compromise, not a reversal: a zero-word page with no JavaScript to
explain the emptiness is a soft 404, and saying nothing would suppress a
fact rather than withhold a judgement. Info settles both demands at once —
the information is no longer swallowed, and CI still does not break.

### 3.2 The verification budget is per run, not per page

A site that repeats the same broken alternate on every page would otherwise
multiply one fault by the sample size — twelve verifications per run keeps
the extra traffic bounded and the signal intact. When the budget runs out
with targets unchecked, the run says so: a truncated report that reads as a
clean one is precisely what this tool exists to prevent.

The budget is spent on drifted targets first, but not on them alone. The
header and the HTML come out of the same generator, so a URL derived wrongly
is derived wrongly on both sides at once: the two sources agree, nothing
disagrees, and the dead alternate is exactly the one nobody was checking.
Verifying agreed targets is what makes `hreflang-dead-target` reachable in
the case production produces most often — drift keeps the priority because a
disagreement is the stronger signal, not because agreement is proof.

### 3.3 Sampling is spread, not the first N

The head of a sitemap is the homepage and its immediate neighbours — the
least representative pages a site has, and the ones most likely to have been
hand-tuned. Sampling therefore spreads across the whole list with integer
arithmetic (no float drift), rather than truncating it.

### 3.4 Rebasing is a security property, not a convenience

A sitemap lists absolute canonical production URLs. Pointed at
`http://127.0.0.1:4123`, a naive implementation follows those `<loc>`s and
audits **production** while printing results under a localhost header —
making a local control run impossible and turning every CI or staging run
into Googlebot-shaped traffic at the live site. This was the worst bug the
JavaScript prototype ever had, which is why the invariant is absolute (every
discovered URL, `--urls` included), why the reconstruction refuses anything
that did not stay on the origin (protocol-relative `//evil` escapes
included), and why a rebase is always announced.

The single exception — hreflang targets probed where they live — exists
because a rebased target would prove nothing: it would answer for a URL the
site never advertised, and a multi-domain alternate set (`example.fr` /
`example.de`) is the normal shape of international hreflang. The exception
is bounded and announced rather than silent.

### 3.5 `http` on the loopback, `https` everywhere else

The first example in the tool's own `--help` is
`headless-seo-check localhost:3000 …`, and an assumed `https` made that
exact example fail — no dev server speaks TLS on 3000. Guessing `http` for a
*public* host would be the worse mistake (it silently downgrades a real
site), so the exception stops at the loopback and `.localhost`, which
RFC 6761 reserves. An explicit scheme is never overridden.

### 3.6 `Unclear` is never blocking

The absence of a cache header is an absence of evidence, not evidence of
absence. An undetermined rendering mode is *our* uncertainty, not the site's
fault, and a diagnostic must never fail someone's CI over its own
limitation. This is a general rule for any future check: our uncertainty is
never an error.

### 3.7 Why an audited page's redirects are not followed

Following a redirect silently means auditing a different page than the one
the sitemap named — the answer would be about the wrong URL. A redirect is
therefore a reported outcome, never a detour. Where following *is* the
point (robots.txt, sitemaps, the homepage probe, hreflang target chains),
the following is explicit and hop-bounded so the chain stays observable.

### 3.8 The CJK overcount is chosen

Chinese and Japanese write no whitespace between words, so blank-splitting
counted a healthy CJK paragraph as one token — and behind a bundle, zero
words meant `empty-server-html` and a failed build. That was the tool's one
structural false positive. Counting one word per CJK character is right for
Chinese (~1 word per character) and overcounts Japanese by roughly two (~2
characters per word); Korean is counted by syllable although it does use
spaces. The overcount is deliberate and points in the tool's safe direction:
it would rather miss a thin page than invent a shell (false negatives
tolerated, false positives forbidden). No language detection, no special
mode, no per-language thresholds — the counting itself became correct, and
the 60/200 thresholds never moved, because the thresholds were never what
was wrong.

### 3.9 Why INTERRUPTED exists

An interrupted clean scan wearing a green `PASSED` badge — one line above an
exit code of 130 — is an unearned pass, and a bare `FAILED` on a scan *we*
cut short reads as a finished verdict. The badge is the one line CI and
humans actually read, and the one line `-q` keeps, so it carries the truth
itself: `INTERRUPTED`, with `so far` as the load-bearing clause — nothing is
concluded about pages never reached. The first Ctrl-C still writes the
partial report because work already done (up to hundreds of requests) must
not be discarded; the exit code still refuses to be 0 because a partial
clean scan is not a pass.

### 3.10 One severity computation, three renderings

The JavaScript prototype shipped the divergence this architecture forbids:
`--allow-dynamic` printed findings in red and exited 0 — the report and the
exit code disagreeing about what was blocking. In the Rust tool, checks
produce typed findings and never text; a single function computes effective
severity; the human report, `--plain`, `--json` and the exit code all
consume that one result. There is no code path by which one mode can count
a blocking finding another mode does not, which is why the contract can
promise that the four surfaces always agree.

### 3.11 Exit 2 is not exit 1

A run that could not happen is not a run that found problems: a CI step
testing only for exit 1 goes green against a completely unreachable host.
Hence the distinct code, the fatal report that says in words that nothing
was checked, and the rule that a run whose every page was unreachable is a
fatal 2 rather than a "clean" 1 — nothing was found because nothing was
reached.

### 3.12 A dev server is a note, not a finding

Real runs settled this one: the same Next.js application that produced ten
blocking issues against a production build came back `PASSED`, exit 0, under
`next dev` — the dev server answers `cache-control: no-cache,
must-revalidate` with no prerender marker at all, which is `unclear`, and
`unclear` never blocks (§3.6), so a whole audit evaporated without a word.
It stays a note rather than a finding because auditing a dev server is a
legitimate thing to do and failing the run would punish the user for a
question we simply cannot answer there — the fault to fix is the silence,
not the exit code. The fingerprint requires the Next.js banner *and*
`no-cache` *and* the absence of `x-nextjs-cache` / `x-nextjs-prerender`
together, because a production site is free to send `no-cache` on purpose
and a tool that tells it it is running a dev server has done more damage
than the one that said nothing.

### 3.13 A finite lifetime outranks a prerender marker

A prerender marker says how the page was born; `s-maxage` says when it will
be born again. The rebirth is the thing the reader is asking about, so a
finite lifetime wins — a page rebuilt every sixty seconds is not the same
promise as one frozen until the next deploy, whatever built it. Real runs
forced the question: Next.js self-hosted answers `revalidate = 60` with
`x-nextjs-prerender: 1` *and* `s-maxage=60`, and reading the marker first
labelled the commonest ISR setup there is `static` — the `isr` verdict was
unreachable on that stack. The year-long `s-maxage` stays `static` because
at that scale the number is a build artefact a deploy replaces, not a
revalidation window. Nothing here is a finding either way: both modes are
healthy, and the fix is a label the reader can trust.

### 3.14 `--verbose` shows the evidence a finding could not carry

Classification evidence travels in finding `detail` lines, which is why it
appears in every mode at once — but a page that passed produces no finding,
so its cache headers had nowhere to go and `--verbose` promised evidence it
could never show for the healthy majority of a scan. It now prints those
lines itself, under the page row, and only for the verdicts no finding
speaks for: `dynamic` and `unclear` hand the renderer nothing, so the
headers are shown once or not at all. The evidence stops at the human
report — `--plain` and the JSON document are the stable interfaces, and a
machine stream that grows because a human asked for more is not machine-
readable.

### 3.15 A path on the argument is a discovery hint, not a scope

A base path is an ordinary deployment. `next.config.js` sets
`basePath: "/app"`, a reverse proxy mounts an application under `/docs`, a
company puts three products behind one hostname — and in every one of those,
the URL a person has is `https://example.com/app`, because it is the only URL
that shows them anything. Typing it used to end the run: the path was thrown
away, discovery probed `/robots.txt`, `/sitemap.xml` and `/sitemap_index.xml`
at the origin's root, all three 404'd, and the tool answered `CANNOT RUN`
about a site that was serving perfectly. The advice it printed was worse than
the failure — `--urls /app` audits exactly one page and calls it a site.

The fix separates two things the old message ran together. *Where to look* is
a guess, and the path is the best guess available. *What to check* is a
promise, and it stays origin-wide: a tool that silently audited only `/app/*`
because of how the argument was typed would be reporting on a subset while
printing the site's name, which is the same class of lie as auditing
production while printing localhost (§3.4). Narrowing a run remains `--urls`,
the flag that exists to say so out loud.

robots.txt keeps its place at the head of the candidate list and keeps
looking at the origin's root: that is where the web contract puts it, a site
under a base path still serves it there, and its `Sitemap:` lines usually name
the prefixed location already — the site speaking for itself beats any guess
of ours. After it, the prefix goes before the root, because a `/sitemap.xml`
at the root of such a deployment more often belongs to a different
application than to this one. Everything found is rebased exactly as before;
the prefix survives because it is part of the *path*, and the path is the
half of a URL rebasing keeps.

### 3.16 Said once, and read in the order the eye triages

`hint()` has always printed its guidance once under the verdict instead of
under every occurrence, on the reasoning that a paragraph said once is advice
and the same paragraph said forty times is wallpaper — and `canonical-missing`
turned out to be a finding of exactly that shape, since a template either emits
a canonical or it does not, so a stock Next.js app produced it on fourteen of
twenty-three pages and buried the one warning that differed; past three pages
the human report now says it once too, under the list, while the pages keep
their mark and the counts keep their arithmetic. The findings under a page are
sorted by severity for the same kind of reason: a person triages by how bad a
thing is and reads top-down, whereas a machine matches on the slug and cannot
tell where in the array it sat. Both changes stop at the human report —
`--plain` and `--json` keep one record per finding in production order, the
order §5.2 freezes — because the eye reads severity and the machine reads the
slug, and only one of the two is a stable interface.

## 4. Limits accepted on purpose

- **A thin static page passes.** Above zero words, a page without a bundle
  is never flagged, however thin — the CI compromise (§3.1). A bundled page
  with 200+ words is never flagged either, even if its useful content
  arrives client-side.
- **Residual non-determinism in the verification budget.** Pages are checked
  concurrently and reserve budget as responses arrive, so beyond 12 targets
  per run, two identical runs may verify different targets and differ in
  their `hreflang-dead-target` findings. Making the attribution
  order-independent was evaluated and deliberately not done: it would move
  verification after the page loop, and the progress counter would then
  undercount what the summary reports — and that equality is the stronger
  contract. The spent-budget note makes the case visible instead of silent.
- **System proxies are not read.** Only proxy *environment variables* are
  honoured. The TLS build deliberately excludes the macOS/Windows system
  proxy configuration APIs, so a proxy configured only in System Settings
  requires exporting `HTTP_PROXY`. Also unexercised by the offline suite:
  `HTTPS_PROXY` with a `CONNECT` tunnel, since the fixture server speaks no
  TLS.
- **`--verbose` is a human-mode flag, and the only place a healthy page's
  evidence appears.** The cache headers behind a `static`/`isr`/`cached`
  verdict are printed under the page row in the human report and nowhere
  else: no `--plain` record, no JSON key, no stderr line. A script that wants
  them has to read the response headers itself. The original plan to print
  evidence on stderr in machine modes was dropped — for the verdicts that do
  produce a finding, the finding already carries it.
- **The homepage fallback checks one URL** and claims nothing about the
  rest of the site.
- **Sitemap indexes nest one level.** A child that is itself an index has
  its `<loc>` entries treated as page URLs.
- **Canonical comparison ignores query and fragment**, and compares path
  first, host second — the same-path-other-host case stays Info because it
  is exactly what a preproduction copy looks like, and flagging it as an
  error would make the tool useless where it serves most.
- **hreflang reciprocity is not checked.** `hreflang-one-sided` compares the
  two sources of *one* page, never what the target page declares in return.
- **`.gz` sitemaps are skipped, loudly.** Decompression is out of scope; the
  note exists so a fully-gzipped sitemap set does not impersonate a site
  with no sitemap.
- **No JavaScript rendering, no link crawling, no Core Web Vitals, no
  content quality, no structured data.** Four questions, answered from the
  outside, with no framework knowledge required. A classic crawler already
  does the rest well, and "a crawler already reports this" is a reason to
  reject a new check.

## 5. The prototype

The tool began as a dependency-free JavaScript prototype (`index.mjs`), which
was ported to Rust and then removed from the repository. The prototype's
value survives as the trap list: every parsing shortcut it took — regex HTML
parsing, naive `Link:` splitting, single-header reads, rebasing by string
concatenation, exit-code divergence under `--allow-dynamic` — is now a named
test in the Rust suite. Its conformance audit is archived verbatim at
`docs/history/cli-conformance.md`; the decisions that outlived it are in
`docs/decisions.md`.