# SEO Insight — Install on a production VPS (single server)

Step-by-step to stand up a **prod test** of SEO Insight on **one Ubuntu LTS VPS**,
all-in-one: the Next app + analytics ETL service + GSC worker + self-hosted
Postgres + a **self-hosted Firecrawl** Docker stack, behind **Nginx** (TLS via
certbot). All feature flags ON.

Config files live in [`deploy/`](../../deploy): `deploy/.env.production.example`,
`deploy/systemd/*.service`, `deploy/nginx/*.conf`, `deploy/firecrawl/*`,
`deploy/deploy.sh`.

> Replace **`dash.seoinsight.app`** with your domain. App user **`seoinsight`**,
> app dir **`/opt/seo-insight`**, Firecrawl dir **`/opt/firecrawl`**.

---

## 0. Architecture

```
                    Internet (443/tcp)
                          │  TLS (certbot)
                      ┌───▼────┐
                      │ Nginx  │  fronts ONLY the app
                      └───┬────┘
           proxy_pass     │ 127.0.0.1:3000
                      ┌───▼─────────────┐
                      │ Next.js (bun)   │  systemd: seo-insight-web
                      └─┬──────┬──────┬─┘
        ANALYTICS_URL   │      │      │ FIRECRAWL_API_URL
   http://localhost:8787│      │      │ http://localhost:3002
        ┌───────────────▼─┐    │   ┌──▼──────────────────────────┐
        │ analytics (Bun, │    │   │ Firecrawl stack (Docker)    │
        │ DuckDB) systemd │    │   │ api:3002 ⟂ playwright ⟂     │
        └───────┬─────────┘    │   │ litellm→Anthropic ⟂ redis ⟂ │
                │   DATABASE_URL│   │ rabbitmq ⟂ pg ⟂ searxng     │
            ┌───▼──────────────▼┐  └─────────────────────────────┘
            │   Postgres 16     │  (app DB; Firecrawl has its OWN pg in Docker)
            └───────▲───────────┘
                    │ DATABASE_URL
          ┌─────────┴────────┐
          │ gsc-worker (Rust)│   systemd: seo-insight-gsc-worker
          └──────────────────┘
   External APIs: Cloudflare R2 · Resend · Stripe · Google OAuth · Anthropic
```

Only **443** (+ **22** SSH) is public. Everything else (3000 / 8787 / 5432 /
3002 / the whole Firecrawl backend) stays on localhost / the Docker bridge.

---

## 1. Prerequisites — gather these first

**Server — sized for Firecrawl.** Firecrawl is the heavy tenant: its compose
limits alone are ~**22 GB / 14 vCPU** (api 12 G, playwright 6 G, litellm 1.5 G,
+ infra). With the app (Next build + DuckDB import ≈ 4-8 G) and Postgres on top:

- **32 GB RAM, 16 vCPU, 60 GB disk** (recommended floor). Ubuntu 22.04/24.04 LTS,
  root/sudo.
- A smaller box works **only** if you lower the Firecrawl concurrency knobs
  (`NUM_WORKERS_PER_QUEUE`, `MAX_CONCURRENT_JOBS`, `BROWSER_POOL_SIZE`) in
  `deploy/firecrawl/.env` — or skip self-hosting Firecrawl.
- A **domain** with an **A record → the VPS IP**; DNS must resolve before step 9.

**External accounts / secrets**
- **Resend**: verified sending domain (SPF/DKIM) + API key.
- **Cloudflare R2**: account + bucket (`seo-exports-prod`) + API token.
- **Stripe**: keys + a webhook signing secret (step 5) — **same account** for keys,
  prices, and webhook.
- **Google Cloud**: OAuth 2.0 Web client.
- **Anthropic**: an API key for Firecrawl's LLM (via LiteLLM).

---

## 2. Base system, firewall, toolchains, Docker

> **Disk layout (multi-disk RAID1):** `/` is small (~18 G) and `/data` is large
> (~193 G). All heavy/growing state goes to **`/data`**: Docker images+volumes
> (`/data/docker`), the app Postgres (`/data/postgres`), Firecrawl volumes
> (`/data/firecrawl`), and the app build (`/data/seo-insight`, symlinked at
> `/opt/seo-insight`). Only the toolchains + OS stay on `/`.

**Fast path — `bootstrap.sh`** does this whole section **and** §3 (Postgres),
disk placement included:

```bash
curl -fsSL https://raw.githubusercontent.com/bastienallain/seo-insight/dev/deploy/bootstrap.sh -o /tmp/bootstrap.sh
sudo bash /tmp/bootstrap.sh        # or: sudo DB_PASSWORD='…' bash /tmp/bootstrap.sh
```

It prints the `DATABASE_URL` to paste in step 5, then skip to **§4**. Prefer to
do it by hand (or to understand each step)? The manual equivalent follows.

```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y git curl build-essential pkg-config libssl-dev ufw

# Dedicated non-root app user
sudo adduser --system --group --home /home/seoinsight --shell /bin/bash seoinsight

# Firewall: SSH + HTTP/HTTPS (80 = ACME challenge + http→https redirect)
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw --force enable

# Docker Engine + Compose v2 — official apt repo (docs.docker.com/engine/install/ubuntu)
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker seoinsight     # run compose without sudo
```

> **Firewall × Docker:** Docker-published ports **bypass UFW** (DOCKER chain, not
> INPUT). This stays safe because the only published port — Firecrawl's API — is
> bound to `127.0.0.1:3002`, so nothing Docker-managed is reachable externally.
> UFW (iptables-nft backend) protects the host services (22, 443). Never write
> raw `nft` rules; if you ever expose a container on `0.0.0.0`, filter it in the
> `DOCKER-USER` chain.

Bun + Rust + wasm-pack as the `seoinsight` user (the app build needs them):

```bash
sudo -u seoinsight -i bash <<'EOF'
curl -fsSL https://bun.sh/install | bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"
cargo install wasm-pack
EOF
```

> Bun lands at `/home/seoinsight/.bun/bin/bun` — the absolute path the systemd
> units use.

---

## 3. PostgreSQL (app database — self-hosted, data on `/data`)

> Done by `bootstrap.sh` already. Manual path below — note it moves the cluster
> to `/data/postgres` (the default `/var/lib/postgresql` is on the small `/`).

```bash
sudo apt install -y postgresql postgresql-contrib
PGVER=$(ls /etc/postgresql | sort -V | tail -1)
sudo systemctl stop postgresql
sudo pg_dropcluster "$PGVER" main --stop
sudo pg_createcluster -d /data/postgres/"$PGVER"/main "$PGVER" main
sudo systemctl start postgresql
sudo -u postgres psql <<'SQL'
CREATE ROLE seoinsight LOGIN PASSWORD 'CHANGE_ME_DB_PASSWORD';
CREATE DATABASE seo_insight OWNER seoinsight;
SQL
```

(Firecrawl runs its **own** Postgres inside Docker — don't confuse the two.)

---

## 4. Clone the app

`bootstrap.sh` already created `/opt/seo-insight` → symlink to `/data/seo-insight`
(so the build lands on the big disk). Clone into the real path:

```bash
sudo -u seoinsight git clone https://github.com/bastienallain/seo-insight.git /data/seo-insight
cd /opt/seo-insight && sudo -u seoinsight git checkout dev
```

> Doing it manually (no bootstrap)? First:
> `sudo mkdir -p /data/seo-insight && sudo chown seoinsight:seoinsight /data/seo-insight && sudo ln -sfn /data/seo-insight /opt/seo-insight`

---

## 5. Environment + external services

```bash
sudo -u seoinsight cp deploy/.env.production.example /opt/seo-insight/.env.production
sudo -u seoinsight chmod 600 /opt/seo-insight/.env.production
sudo -u seoinsight nano /opt/seo-insight/.env.production
```

Fill every `CHANGE_ME`. Secrets: `openssl rand -base64 32` (BETTER_AUTH_SECRET),
`openssl rand -hex 32` (ANALYTICS_API_KEY, WORKER_API_KEY). Wire externals:

| Service | Where | Value |
|---|---|---|
| **Google OAuth** | Cloud Console → Credentials → Web client → redirect URIs | `https://dash.seoinsight.app/api/auth/callback/google` |
| **Stripe webhook** | Dashboard → Developers → Webhooks → Add endpoint | `https://dash.seoinsight.app/api/auth/stripe/webhook` → secret → `STRIPE_WEBHOOK_SECRET` |
| **Resend** | Domains → verify → SPF/DKIM | `RESEND_API_KEY`; `EMAIL_FROM="SEO Insight <noreply@dash.seoinsight.app>"` |
| **R2** | `bunx wrangler r2 bucket create seo-exports-prod` + token | `R2_*` |

`FIRECRAWL_API_URL` is already `http://localhost:3002` and `FIRECRAWL_API_KEY`
is `sk-firecrawl-local` — they point at the internal stack from step 6.

> ### ⚠️ Stripe — plans only update via the webhook
> Plan changes go through a Stripe **billing-portal confirm** flow; the new plan
> is written to the DB only when `customer.subscription.updated` arrives. So:
> 1. Keys, prices, and webhook **must be in the SAME Stripe account** (a mismatch
>    silently drops events — the #1 cause of "le forfait ne s'update pas").
> 2. Enable events: `checkout.session.completed`,
>    `customer.subscription.created`, `customer.subscription.updated`,
>    `customer.subscription.deleted`.
> 3. Keep the endpoint on the account's **current** API version (e.g.
>    `2026-03-25.dahli`) — an old pinned version (`2019-02-19`) has no
>    `price.lookup_key`, so the plugin can't resolve the plan.
> 4. The billing page **self-heals** drift on load; for a bulk fix run
>    `bun --env-file=.env.production --conditions=react-server scripts/stripe-resync.ts`.

---

## 6. Firecrawl — internal Docker stack

Copy **your** Firecrawl checkout (it carries `litellm/`, `searxng/`,
`nuq-postgres/`) to the VPS, then drop in the internalized compose from this repo
(Traefik removed, API published only on `127.0.0.1:3002`):

```bash
# From your machine — push the Firecrawl tree to the VPS:
rsync -a ~/Desktop/firecrawl/firecrawl/ seoinsight@<VPS_IP>:/opt/firecrawl/

# On the VPS, as seoinsight:
cd /opt/firecrawl
cp /opt/seo-insight/deploy/firecrawl/docker-compose.yaml ./docker-compose.yaml   # internalized
cp /opt/seo-insight/deploy/firecrawl/.env.example ./.env && nano ./.env          # fill secrets
sudo mkdir -p /data/firecrawl/{redis,rabbitmq,postgres}
sudo chown -R seoinsight:seoinsight /data/firecrawl

docker compose up -d --build      # builds playwright + nuq-postgres (first run is long)
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3002/    # → 200 when ready
```

`./.env` `OPENAI_API_KEY` **must equal** the app's `FIRECRAWL_API_KEY`
(`sk-firecrawl-local` by default). Firecrawl stays private — Nginx never proxies it.

---

## 7. Build the app

```bash
sudo -u seoinsight bash -lc 'cd /opt/seo-insight && ./deploy/deploy.sh'
```

`deploy.sh` = pull → `bun install` (root + analytics) → `build:wasm` → cargo
build the GSC worker → `db:migrate` → `next build` → restart services. On the
**first** run the services don't exist yet (the restart warns) — create them next.

---

## 8. systemd services (app)

```bash
sudo cp /opt/seo-insight/deploy/systemd/seo-insight-* /etc/systemd/system/   # .service + .timer
sudo systemctl daemon-reload
# Long-running services: web + analytics (analytics needs R2 in the env).
sudo systemctl enable --now seo-insight-web seo-insight-analytics
# GSC sync is a ONE-SHOT driven by a timer — enable the TIMER, not the service
# (it would just run once at boot). The timer runs it 10 min after boot, then daily.
sudo systemctl enable --now seo-insight-gsc-worker.timer
systemctl status seo-insight-web seo-insight-analytics --no-pager
systemctl list-timers seo-insight-gsc-worker.timer --no-pager
# Run a GSC sync on demand: sudo systemctl start seo-insight-gsc-worker.service
```

App on `127.0.0.1:3000`, analytics on `127.0.0.1:8787`, Firecrawl on
`127.0.0.1:3002` — all internal.

---

## 9. Nginx + HTTPS

```bash
sudo apt install -y nginx
sudo cp /opt/seo-insight/deploy/nginx/dash.seoinsight.app.conf \
        /etc/nginx/sites-available/dash.seoinsight.app
sudo sed -i 's/dashboard\.seoinsight\.app/YOURDOMAIN/g' \
        /etc/nginx/sites-available/dash.seoinsight.app
sudo ln -sf /etc/nginx/sites-available/dash.seoinsight.app /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

# TLS — certbot rewrites the vhost to HTTPS + adds the 80→443 redirect, and renews.
sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d dash.seoinsight.app
```

Visit `https://dash.seoinsight.app` → the app.

---

## 10. First run

1. Sign up with the email in `SUPER_ADMIN_EMAILS`.
2. Confirm the verification / magic-link email (Resend).
3. Complete onboarding → you have an org + super-admin.

---

## 11. Smoke test

| Feature | Check |
|---|---|
| **Auth/email** | Magic-link login works (email arrives). |
| **Heavy import** | Keywords → Importer un CSV → upload → job « Import terminé » (R2 + analytics). |
| **Billing** | Start a checkout; the Stripe webhook shows **200**; the plan updates. |
| **GSC** | Settings → Intégrations → connect; `journalctl -u seo-insight-gsc-worker -f` polls. |
| **SERP crawl** | Crawl a keyword → competitors enriched (Firecrawl `:3002` reachable). |

---

## 12. Updates, logs, troubleshooting

```bash
# App update
sudo -u seoinsight bash -lc 'cd /opt/seo-insight && ./deploy/deploy.sh'
# Firecrawl update
cd /opt/firecrawl && docker compose pull && docker compose up -d --build

# Logs
journalctl -u seo-insight-web -u seo-insight-analytics -u seo-insight-gsc-worker -f
docker compose -f /opt/firecrawl/docker-compose.yaml logs -f api
sudo journalctl -u nginx -f
```

**Common issues**
- *App won't boot, env error* → the Zod validator names the missing/weak field.
- *Heavy import 502 / job stuck* → analytics service down or `R2_*` wrong.
- *Import 503 « non configuré »* → `analyticsEnabled` false (an `R2_*`/`ANALYTICS_URL` unset).
- *Email never arrives* → `RESEND_API_KEY` + `EMAIL_FROM` set and the sender domain verified.
- *Auth 403 on login* → `BETTER_AUTH_URL` must equal `https://dash.seoinsight.app`.
- *Plan doesn't update after payment* → webhook not delivered (wrong account / endpoint / secret); the billing page self-heals on load, or run `scripts/stripe-resync.ts`.
- *SERP crawl fails* → `docker compose -f /opt/firecrawl/docker-compose.yaml ps`; the `api` healthcheck must be healthy; check `ANTHROPIC_API_KEY` in `/opt/firecrawl/.env`.
- *Firecrawl OOM / host swapping* → lower the concurrency knobs in `/opt/firecrawl/.env` and `docker compose up -d`.
- *DB "column does not exist" post-deploy* → re-run `bun run db:migrate`.

**DB backup (before a risky test)**
```bash
sudo -u postgres pg_dump seo_insight | gzip > ~/seo_insight_$(date +%F).sql.gz
```