berrypod/docs/plans/url-redirects.md

# URL redirects

> Status: Complete (#78–81), #82 deferred until page editor adds configurable links
> Tasks: #78–82 in PROGRESS.md
> Tier: 3 (Compliance & quality — SEO dependency)

## Goal

Preserve link equity and customer experience when product URLs change, products are removed, or collections are renamed. Automatically handle the most common cases, use analytics data to identify what actually matters, and surface anything ambiguous for admin review.

## Why it matters

Product slugs in Berrypod are generated from product titles via `Slug.slugify(title)`. When a provider renames a product, the next sync generates a new slug and the old URL becomes a 404. These old URLs may be:

- Indexed by Google (losing SEO rank)
- Shared on social media, in emails, in newsletters
- Bookmarked by returning customers

Most redirect implementations just provide a manual table. The insight here is that we already have analytics data recording which paths have had real human traffic — so we can separate 404s that matter (broken real URLs) from noise (bot scanners, `/wp-admin` probes, etc.) without any manual work.

## Three layers

### Layer 1: Automatic redirect creation on slug change or deletion

Three triggers, all detected during provider sync:

#### 1a. Product slug change

When a product's title changes during sync, the slug changes, and the old `/products/old-slug` URL breaks. Detected in `upsert_product/2`.

**Hook point:** `lib/berrypod/products.ex` — the `product ->` branch in `upsert_product/2` where `update_product(product, attrs)` is called. At this point we have `product.slug` (old) and can compute the new slug from `attrs[:title]`.

```elixir
product ->
  old_slug = product.slug
  new_slug = Slug.slugify(attrs[:title] || attrs["title"])

  case update_product(product, attrs) do
    {:ok, updated_product} ->
      if old_slug != updated_product.slug do
        Redirects.create_auto(%{
          from_path: "/products/#{old_slug}",
          to_path: "/products/#{updated_product.slug}",
          source: :auto_slug_change
        })
      end
      {:ok, updated_product, :updated}

    error -> error
  end
```

`create_auto/1` uses `on_conflict: :nothing` on the `from_path` unique index — safe to call repeatedly if sync runs multiple times.

#### 1b. Product deletion

When a product is removed during sync, create a redirect to the most specific relevant page. Look up the product's category before deletion and redirect to that collection page. If no category is known, fall back to `/`.

Google's guidance is that a 301 to an irrelevant page (soft 404) is worse than a clean 404, so the redirect target must make sense — the collection page shows related products the customer might want.

```elixir
# In delete_product/1, before the actual deletion
category = product.category
target = if category, do: "/collections/#{Slug.slugify(category)}", else: "/"

Redirects.create_auto(%{
  from_path: "/products/#{product.slug}",
  to_path: target,
  source: :auto_product_deleted
})
```

#### 1c. Collection slug change

Categories come from provider tags. If a tag is renamed, the category slug changes and `/collections/old-slug` breaks. Same detection logic — compare old vs new slug in the category upsert path and create a redirect.

Lower priority than products (collection URLs change less often), but the same mechanism handles it.

### Layer 2: A `redirects` table checked early in the Plug pipeline

One table, one Plug, all redirect types flow through the same path.

**Plug position:** Added to the `:browser` pipeline in `router.ex`, before routing. Checks a path, 301s and halts if a redirect exists, otherwise passes through.

```elixir
# router.ex
pipeline :browser do
  ...
  plug BerrypodWeb.Plugs.Redirects
  ...
end
```

```elixir
defmodule BerrypodWeb.Plugs.Redirects do
  import Plug.Conn
  alias Berrypod.Redirects

  def init(opts), do: opts

  def call(conn, _opts) do
    path = conn.request_path

    # Normalise: trailing slash removal (except root)
    # and lowercase path (not query params)
    normalised = path |> maybe_strip_trailing_slash() |> String.downcase()

    cond do
      # Trailing slash or case mismatch — redirect to canonical form
      normalised != path ->
        location = append_query(normalised, conn.query_string)

        conn
        |> put_resp_header("location", location)
        |> send_resp(301, "")
        |> halt()

      # Check redirect table (ETS-cached)
      match?({:ok, _}, Redirects.lookup(path)) ->
        {:ok, redirect} = Redirects.lookup(path)
        Redirects.increment_hit_count(redirect)
        location = append_query(redirect.to_path, conn.query_string)

        conn
        |> put_resp_header("location", location)
        |> send_resp(redirect.status_code, "")
        |> halt()

      true ->
        conn
    end
  end

  defp maybe_strip_trailing_slash("/"), do: "/"
  defp maybe_strip_trailing_slash(path), do: String.trim_trailing(path, "/")

  defp append_query(path, ""), do: path
  defp append_query(path, qs), do: "#{path}?#{qs}"
end
```

The Plug handles three concerns in one pass:

1. **Trailing slash normalisation** — `/products/foo/` → `/products/foo`. Phoenix generates no-trailing-slash URLs, so this is the canonical form. Prevents duplicate content in Google's index.
2. **Case normalisation** — `/Products/Foo` → `/products/foo`. URLs are technically case-sensitive per RFC 3986, but mixed-case URLs cause duplicate content issues. Shopify lowercases everything. Only applies to the path, not query params (those can be case-sensitive for variant selectors like `?Color=Sand`).
3. **Redirect table lookup** — custom redirects from the `redirects` table.

All three preserve query params. This matters for variant selection URLs (`?Color=Sand&Size=S`) surviving a product slug change redirect.

**Caching:** The redirect lookup is on the hot path for every request. Use ETS for an in-memory cache, populated on app start and invalidated on any redirect create/update/delete.

```elixir
# On app start, load all redirects into ETS
Redirects.warm_cache()

# On redirect change, invalidate
Redirects.invalidate_cache(from_path)
```

The ETS table maps `from_path` (binary) → `{to_path, status_code}`. Cache miss falls through to DB. Given redirects are rare and mostly set-and-forget, the cache hit rate should be near 100% after warmup.

### Layer 3: Analytics-powered 404 monitoring

When a 404 fires, most hits are bots and scanners. The signal that distinguishes a real broken URL from noise is analytics history: if a path appears in `events` with prior real pageviews, it was a genuine product page.

**404 handler hook:** The existing `error.ex` LiveView renders 404s. Add a side-effect: when a 404 fires on a path matching `/products/:slug` or `/collections/:slug`, query analytics and potentially auto-resolve.

```elixir
defp maybe_log_broken_url(path) do
  prior_hits = Analytics.count_pageviews_for_path(path)

  if prior_hits > 0 do
    BrokenUrls.record(%{
      path: path,
      prior_analytics_hits: prior_hits
    })
    attempt_auto_resolve(path, prior_hits)
  end
end
```

**Auto-resolution attempt:**

For `/products/:slug` 404s, extract the slug and run it through the FTS5 search index to find the most likely current product:

```elixir
defp attempt_auto_resolve("/products/" <> old_slug, _hits) do
  query = String.replace(old_slug, "-", " ")

  case Search.search_products(query, limit: 1) do
    [%{score: score, slug: new_slug}] when score > @confidence_threshold ->
      Redirects.create_auto(%{
        from_path: "/products/#{old_slug}",
        to_path: "/products/#{new_slug}",
        source: :analytics_detected,
        confidence: score
      })

    _ ->
      # No confident match - leave in broken_urls for admin review
      :ok
  end
end
```

The `@confidence_threshold` needs tuning — FTS5 BM25 scores are negative (more negative = better match). Start conservative; it's better to leave something for manual review than to auto-redirect to the wrong product.

For **deleted products** with no match, the redirect target defaults to the product's last known category collection page if that's inferable (from the path or broken_url record), otherwise falls back to `/`.

---

## Schemas

### `redirects` table

```elixir
create table(:redirects, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :from_path, :string, null: false     # "/products/old-classic-tee"
  add :to_path, :string, null: false       # "/products/classic-tee-v2" or "/"
  add :status_code, :integer, default: 301 # 301 permanent, 302 temporary
  add :source, :string, null: false        # "auto_slug_change" | "auto_product_deleted" | "analytics_detected" | "admin"
  add :confidence, :float                  # FTS5 match score for analytics_detected, nil otherwise
  add :hit_count, :integer, default: 0    # incremented each time this redirect fires
  timestamps()
end

create unique_index(:redirects, [:from_path])
create index(:redirects, [:source])
```

### `broken_urls` table

```elixir
create table(:broken_urls, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :path, :string, null: false
  add :prior_analytics_hits, :integer, default: 0  # pageviews before the 404 started
  add :recent_404_count, :integer, default: 1       # 404s since it broke
  add :first_seen_at, :utc_datetime, null: false
  add :last_seen_at, :utc_datetime, null: false
  add :status, :string, default: "pending"          # "pending" | "resolved" | "ignored"
  add :resolved_redirect_id, :binary_id             # FK to redirects when resolved
  timestamps()
end

create unique_index(:broken_urls, [:path])
create index(:broken_urls, [:status])
create index(:broken_urls, [:prior_analytics_hits])  # sort by impact
```

---

## Admin UI

**Route:** `/admin/redirects`

### Tab 1: Active redirects

Table of all redirects with columns: from path, to path, source (badge: auto/detected/manual), hit count, created at. Delete button to remove. Edit to change destination.

Sources:
- `auto_slug_change` — created automatically when sync detected a slug change. Trust these.
- `auto_product_deleted` — created automatically when a product was removed. Targets the category collection page or `/`.
- `analytics_detected` — created from analytics + FTS5 match. Show confidence score. Worth reviewing.
- `admin` — manually created.

### Tab 2: Broken URLs (pending review)

Table sorted by `prior_analytics_hits` descending — highest impact broken URLs at the top.

Columns: path, prior traffic (from analytics), 404s since breaking, first seen.

Each row has a quick action: enter a redirect destination and save, or mark as ignored (e.g. it's a legitimate 404 from a product intentionally removed).

Pre-filled suggestion from FTS5 search (same logic as auto-resolution, just surfaced for human confirmation rather than applied automatically).

### Tab 3: Dead links

See below — dead link monitoring surfaces here alongside redirects, since they're two sides of the same problem.

### Tab 4: Create redirect

Simple form: from path, to path, status code (301/302). For manual one-off redirects (external links, social posts, etc.).

---

## Data flow

```
Provider renames product
    ↓
ProductSyncWorker → upsert_product/2
    ↓
old_slug != new_slug detected
    ↓
Redirects.create_auto({from: /products/old, to: /products/new})
    → ETS cache invalidated

    ─────

Provider deletes product
    ↓
delete_product/1
    ↓
Look up product category before deletion
    ↓
Redirects.create_auto({from: /products/slug, to: /collections/category or /})
    → ETS cache invalidated

    ─────

Any request hits the Plug
    ↓
1. Trailing slash? → 301 to canonical (preserving query params)
2. Mixed case path? → 301 to lowercase (preserving query params)
3. Redirect table match? → 301/302 to target (preserving query params)
4. None of the above → pass through to router

    ─────

Customer visits /products/old-slug?Color=Sand
    ↓
BerrypodWeb.Plugs.Redirects checks ETS cache
    ↓ hit
301 → /products/new-slug?Color=Sand
hit_count incremented

    ─────

Bot/customer visits an unknown broken URL
    ↓
Plug: no redirect found → pass through
    ↓
Router: no match → 404 LiveView
    ↓
Analytics.count_pageviews_for_path(path)
    ↓
0 hits → likely a bot, discard silently
> 0 hits → real broken URL
    ↓
BrokenUrls.record(path, prior_hits)
    ↓
Attempt FTS5 auto-resolve
    ↓ confident match
Redirects.create_auto({..., source: :analytics_detected})
    ↓ no match
Left in broken_urls for admin review

    ─────

Admin opens /admin/redirects → broken URLs tab
    ↓
Sees sorted list of broken URLs by prior traffic
    ↓
Enters destination → creates redirect
    ↓
ETS cache warmed → Plug now catches future requests

    ─────

Weekly Oban cron
    ↓
Prune auto redirects with 0 hits older than 90 days
```

---

---

## Dead link monitoring

Redirects fix *incoming* broken URLs. Dead link monitoring fixes *outgoing* broken links in your own content — nav links, footer links, social URLs, announcement bar targets, rich text content, product descriptions. Two sides of the same problem.

### Why Berrypod can do this better than external tools

External link checkers (Ahrefs, Screaming Frog, etc.) crawl your site periodically from the outside. They can't know *why* a link broke or *when* it's about to break. Berrypod knows:

- Exactly which URLs are valid (it owns the router and the DB)
- When products are deleted or renamed (sync events)
- Where every admin-configured link is stored (settings keys)

This means internal links can be validated **instantly and without any HTTP request** — just check the router and DB. External links need an async HTTP HEAD check via Oban.

### Sources of links in Berrypod

| Source | Type | When to check |
|--------|------|---------------|
| Nav/footer links (settings) | Internal or external | On save + when referenced product changes |
| Social links (settings) | External | On save + weekly Oban job |
| Announcement bar target URL (settings) | Internal or external | On save |
| Rich text content (future page editor) | Internal or external | On save + when referenced product changes |
| Product descriptions (synced from providers) | Potentially external | After each sync |
| Contact page email | Not a URL | Format validation only |

**Note:** Links rendered *from DB data* (product cards, collection listings) are safe by construction — you only render a link if the product/collection exists. The risk is entirely in user-entered free-text URLs stored in settings or content.

### Two-phase validation

**Phase 1: Internal links — instant router + DB check**

```elixir
defmodule Berrypod.LinkValidator do
  alias BerrypodWeb.Router.Helpers

  def validate(url) when is_binary(url) do
    uri = URI.parse(url)

    cond do
      # External URL — queue for async check
      uri.host != nil -> {:external, url}

      # Internal — check router match
      true -> validate_internal(uri.path)
    end
  end

  defp validate_internal("/products/" <> slug) do
    case Products.get_product_by_slug(slug) do
      %{visible: true, status: "active"} -> :ok
      %{visible: false} -> {:dead, :product_hidden}
      nil -> {:dead, :product_not_found}
    end
  end

  defp validate_internal("/collections/" <> slug) do
    if Products.category_exists?(slug), do: :ok, else: {:dead, :category_not_found}
  end

  defp validate_internal(path) do
    # Check against router for known static paths
    case Phoenix.Router.route_info(BerrypodWeb.Router, "GET", path, "") do
      :error -> {:dead, :no_route}
      _match -> :ok
    end
  end
end
```

**Phase 2: External links — async Oban job**

```elixir
defmodule Berrypod.Workers.ExternalLinkCheckWorker do
  use Oban.Worker, queue: :default, max_attempts: 2

  def perform(%{args: %{"url" => url, "source_key" => source_key}}) do
    case Req.head(url, receive_timeout: 10_000, redirect: true) do
      {:ok, %{status: status}} when status < 400 -> :ok
      {:ok, %{status: status}} -> record_dead_link(url, source_key, status)
      {:error, _} -> record_dead_link(url, source_key, :unreachable)
    end
  end
end
```

Rate limiting: one check per URL per 24 hours. Don't hammer external servers.

### Event-driven invalidation

The smart part. Rather than only checking periodically, hook into the events that *cause* dead links:

**On product deleted/made invisible:**
```elixir
# After Products.delete_product/1 or hiding a product
DeadLinks.scan_stored_links_for_path("/products/#{old_slug}")
# Finds any nav/footer/content links pointing to that path → flags them
```

**On product slug change:**
The redirect is created automatically (existing plan). Additionally:
```elixir
# Stored links pointing to the old slug are now stale
# Flag them with a "link moved" status + the new destination
DeadLinks.flag_moved_links("/products/#{old_slug}", "/products/#{new_slug}")
# Admin sees: "Your footer links to /products/old-name — this moved to /products/new-name. Update it?"
```

This is more actionable than just "link is broken" — it tells you where it moved to.

**On admin saves any content with URLs:**
Validate immediately. Internal links checked synchronously (fast). External links enqueued for async check.

### Schema

```elixir
create table(:stored_links, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :url, :string, null: false           # the full URL or path
  add :source_key, :string, null: false    # e.g. "settings.footer_link_1", "nav.about"
  add :link_type, :string, null: false     # "internal" or "external"
  add :status, :string, default: "ok"      # "ok" | "dead" | "moved" | "unchecked"
  add :http_status, :integer               # last HTTP status for external links
  add :dead_reason, :string                # "product_not_found", "no_route", "unreachable", etc.
  add :moved_to, :string                   # when status is "moved", the new destination
  add :last_checked_at, :utc_datetime
  timestamps()
end

create unique_index(:stored_links, [:url, :source_key])
create index(:stored_links, [:status])
create index(:stored_links, [:link_type])
```

### Admin UI: Dead links tab

Table of all dead/moved/unchecked stored links, sorted by status (dead first, then moved, then unchecked).

Columns: source (where the link is — "Footer", "Nav", "Announcement bar"), URL, status badge, last checked, action.

Actions:
- **Dead:** "Edit" (opens the relevant settings section pre-focused on that field) — or "Ignore" if intentional
- **Moved:** "Update link" one-click to replace old URL with the new destination in the source setting
- **Unchecked:** "Check now" to trigger immediate validation

Dashboard integration: a small badge on the admin dashboard card ("3 dead links") to draw attention without being annoying. Cleared when all are resolved or ignored.

### Weekly Oban cron job

Re-check all external links stored in `stored_links`. Internal links don't need periodic re-checking — they're validated on demand and on data-change events, which is more efficient.

```elixir
# In Oban crontab
{"0 3 * * 1", Berrypod.Workers.WeeklyExternalLinkCheckWorker}
```

The weekly job enqueues one `ExternalLinkCheckWorker` job per external stored link, with rate limiting.

### What it deliberately doesn't do

- **Doesn't crawl rendered HTML** — too fragile, too slow. We work from structured data (settings keys, content blocks), not parsed HTML.
- **Doesn't check links in transactional emails** — those are templates, not user content.
- **Doesn't validate email addresses** — format check only, not SMTP validation (too invasive).
- **Doesn't check links in product images** — image URLs are managed by the Media pipeline, not free-text.

### Relationship to redirect system

| Problem | Solution |
|---------|----------|
| Visitor hits a broken URL | **Redirect** — 301 to new location |
| Your own content links to a broken URL | **Dead link fix** — update the link in your content |
| Product renamed — old URL works | Redirect created automatically |
| Product renamed — your nav still says old URL | Dead link flagged as "moved" with suggestion |

They complement each other. The redirect preserves SEO and visitor experience for external links you can't control (social posts, other websites linking to you). The dead link monitor fixes links you *can* control — your own navigation, content, and settings.

---

## Auto-pruning

Auto-created redirects with zero hits are pruned after 90 days via a weekly Oban cron job. This prevents unbounded growth if products are renamed repeatedly.

```elixir
# Weekly cron: prune stale auto-redirects
from(r in Redirect,
  where: r.source in ["auto_slug_change", "auto_product_deleted"] and r.hit_count == 0,
  where: r.inserted_at < ago(90, "day")
)
|> Repo.delete_all()
```

Redirects that have been used at least once are kept forever — they're demonstrably serving traffic. Manual (`admin`) and analytics-detected redirects are excluded from auto-pruning; the admin can delete them manually if needed.

---

## Implementation notes

**Slug change detection is safe to add with no behaviour change** for products that don't change slug. The `on_conflict: :nothing` insert ensures idempotency across repeated syncs.

**The FTS5 confidence threshold** should be tuned conservatively at first. An incorrect auto-redirect (wrong product) is worse than no redirect. Admin review catches the gaps.

**ETS cache invalidation** needs to happen on: redirect created, updated, deleted. Simple `GenServer` or `:persistent_term` approach — at the scale of a single-tenant shop, the full redirect table easily fits in memory.

**Redirect chains** (A → B → C) should be detected and flattened on creation. If a new redirect's `to_path` is itself an existing `from_path`, follow it and set the new redirect's `to_path` to the final destination. Avoids multi-hop redirects.

**Status code guidance:**
- `301` Permanent — use for slug changes and deleted products. Tells Google to update its index.
- `302` Temporary — only for sales/temporary campaigns. Tells Google to keep the original URL indexed.

---

## Files to create/modify

- Migration — `redirects` and `broken_urls` tables
- `lib/berrypod/redirects/redirect.ex` — schema
- `lib/berrypod/redirects/broken_url.ex` — schema
- `lib/berrypod/redirects.ex` — context: `lookup/1`, `create_auto/1`, `create_manual/1`, `warm_cache/0`, `invalidate_cache/1`, `increment_hit_count/1`, `list_broken_urls/0`, `record_broken_url/2`
- `lib/berrypod_web/plugs/redirects.ex` — new Plug (redirects + trailing slash + case normalisation)
- `lib/berrypod/products.ex` — slug change detection in `upsert_product/2`, redirect on deletion in `delete_product/1`
- `lib/berrypod_web/live/shop/error.ex` — hook analytics query on 404
- `lib/berrypod_web/live/admin/redirects_live.ex` — new LiveView (3 tabs)
- `lib/berrypod/workers/redirect_pruner_worker.ex` — weekly Oban cron for auto-pruning
- Router — `/admin/redirects` route, ETS cache warm on startup
- Admin nav — new sidebar link

## Tests

- `upsert_product/2` with title change creates redirect automatically
- `upsert_product/2` with no title change does not create redirect
- `delete_product/1` creates redirect to category collection page
- `delete_product/1` with no category creates redirect to `/`
- Redirect Plug: matching path → 301, no match → passthrough
- Redirect Plug: query string preserved on redirect (`?Color=Sand` survives)
- Redirect Plug: trailing slash stripped (`/products/foo/` → `/products/foo`)
- Redirect Plug: mixed case normalised (`/Products/Foo` → `/products/foo`)
- Redirect Plug: root `/` trailing slash not stripped
- Redirect Plug: ETS cache hit (no DB call)
- 404 handler: path with analytics history → broken_url record created
- 404 handler: path with no analytics history → nothing recorded
- FTS5 auto-resolve: confident match → redirect created; no match → broken_url pending
- Redirect chain flattening: A→B, new B→C → stored as A→C
- `hit_count` incremented on each redirect fire
- Auto-pruning: 0-hit auto redirects older than 90 days deleted
- Auto-pruning: manual and analytics-detected redirects excluded
- Auto-pruning: redirects with hits > 0 preserved regardless of age