# Incident — 2026-07-13 ~12:42 UTC: brief public-site downtime during auto-deploy

**Status:** resolved (self-recovered). No code fix required.
**Duration:** < 1 minute (monitor reported "DOWN for ~0 min").
**Impact:** https://gluecron.com briefly unreachable while the app container was replaced.

## Timeline (UTC)

- 12:42:06 — `gluecron-update.service` (minutely auto-deploy timer) picked up commit
  `41d6ea3` ("fix(backup): dump the REAL production DB (Neon), add proven restore drill"),
  rebuilt the image and **replaced** `gluecron-gluecron-1` via
  `docker compose -f docker-compose.standalone.yml up -d --build`.
- 12:42:12 — deploy log: `deploy complete: 41d6ea3… (app healthy)`. Migrations were
  all no-ops ("already applied, skipping").
- ~12:42 — external monitor caught the container-replacement window and fired a
  self-heal alert.
- 12:46:20 — self-heal session started; site already serving HTTP 200 in ~0.1 s
  (5/5 probes). Container healthy, `restarts=0`, `oom=false`, exit code 0.

## Root cause

Not a crash. `docker compose up --build` replaces the app container in-place
(stop-then-start), so every deploy has a short window where Traefik has no healthy
upstream. The monitor's probe landed inside that window. The deploy itself succeeded.

## Actions taken

- Diagnosed only — no restarts, no rollback, no DB operations. The safest fix was no fix.
- Verified stability: repeated probes returned HTTP 200; container health checks passing.
- Ran `bun run typecheck` on the deployed commit: **45 pre-existing TS errors across
  15 files** (top offenders: `src/routes/oauth.tsx` 9, `src/routes/well-known.ts` 5,
  `src/routes/mcp.ts` 5). These pre-date this session; Bun strips types at runtime so
  they don't affect prod, but they should be burned down before `tsc` can gate deploys.
- Build verification: the production Docker image build at 12:42 completed and passed
  the app health gate (no separate `build` script exists; the image build is the build).

## Follow-ups (not done here)

1. **Zero-downtime deploys** — the minutely auto-deploy replaces the container on every
   commit, so every push causes a blip like this one (and can re-trigger the self-heal
   monitor). Consider start-first replacement (bring up the new container, health-check
   it, then swap Traefik routing) or two replicas behind Traefik.
2. **Monitor grace window** — suppress the alert if a deploy is in flight
   (`gluecron-update.service` active) or require 2 consecutive failed probes.
3. **Type-error burn-down** — 45 errors block using `bun run typecheck` as a deploy gate.
4. **Untracked leftovers** — `/root/gluecron/docker-compose.traefik.yml` (unused draft,
   superseded by `docker-compose.override.yml`) and several `.env.bak.*` /
   `docker-compose.standalone.yml.bak.*` files in `/opt/gluecron` should be cleaned up
   or gitignored.