Commit94b5edf
docs(incident): 2026-07-13 deploy-window blip — root cause + follow-ups
docs(incident): 2026-07-13 deploy-window blip — root cause + follow-ups Self-heal alert at ~12:42 UTC was transient downtime caused by the minutely auto-deploy replacing the app container for commit 41d6ea3. The deploy succeeded and the site self-recovered in under a minute; no restart or rollback was needed. Records the timeline, evidence (healthy container, 0 restarts, no OOM), the 45 pre-existing typecheck errors found during verification, and follow-ups (start-first deploys, monitor grace window). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed+49−094b5edfb7a0e8f196eda9b21fe28ea67d95b0c9b
1 changed file+49−0
AddedINCIDENT-2026-07-13-deploy-blip.md+49−0View fileUnifiedSplit
@@ -0,0 +1,49 @@
1# Incident — 2026-07-13 ~12:42 UTC: brief public-site downtime during auto-deploy
2
3**Status:** resolved (self-recovered). No code fix required.
4**Duration:** < 1 minute (monitor reported "DOWN for ~0 min").
5**Impact:** https://gluecron.com briefly unreachable while the app container was replaced.
6
7## Timeline (UTC)
8
9- 12:42:06 — `gluecron-update.service` (minutely auto-deploy timer) picked up commit
10 `41d6ea3` ("fix(backup): dump the REAL production DB (Neon), add proven restore drill"),
11 rebuilt the image and **replaced** `gluecron-gluecron-1` via
12 `docker compose -f docker-compose.standalone.yml up -d --build`.
13- 12:42:12 — deploy log: `deploy complete: 41d6ea3… (app healthy)`. Migrations were
14 all no-ops ("already applied, skipping").
15- ~12:42 — external monitor caught the container-replacement window and fired a
16 self-heal alert.
17- 12:46:20 — self-heal session started; site already serving HTTP 200 in ~0.1 s
18 (5/5 probes). Container healthy, `restarts=0`, `oom=false`, exit code 0.
19
20## Root cause
21
22Not a crash. `docker compose up --build` replaces the app container in-place
23(stop-then-start), so every deploy has a short window where Traefik has no healthy
24upstream. The monitor's probe landed inside that window. The deploy itself succeeded.
25
26## Actions taken
27
28- Diagnosed only — no restarts, no rollback, no DB operations. The safest fix was no fix.
29- Verified stability: repeated probes returned HTTP 200; container health checks passing.
30- Ran `bun run typecheck` on the deployed commit: **45 pre-existing TS errors across
31 15 files** (top offenders: `src/routes/oauth.tsx` 9, `src/routes/well-known.ts` 5,
32 `src/routes/mcp.ts` 5). These pre-date this session; Bun strips types at runtime so
33 they don't affect prod, but they should be burned down before `tsc` can gate deploys.
34- Build verification: the production Docker image build at 12:42 completed and passed
35 the app health gate (no separate `build` script exists; the image build is the build).
36
37## Follow-ups (not done here)
38
391. **Zero-downtime deploys** — the minutely auto-deploy replaces the container on every
40 commit, so every push causes a blip like this one (and can re-trigger the self-heal
41 monitor). Consider start-first replacement (bring up the new container, health-check
42 it, then swap Traefik routing) or two replicas behind Traefik.
432. **Monitor grace window** — suppress the alert if a deploy is in flight
44 (`gluecron-update.service` active) or require 2 consecutive failed probes.
453. **Type-error burn-down** — 45 errors block using `bun run typecheck` as a deploy gate.
464. **Untracked leftovers** — `/root/gluecron/docker-compose.traefik.yml` (unused draft,
47 superseded by `docker-compose.override.yml`) and several `.env.bak.*` /
48 `docker-compose.standalone.yml.bak.*` files in `/opt/gluecron` should be cleaned up
49 or gitignored.
050