CodeIssuesDiscussionsWikiPull RequestsProjectsCommitsActionsReleasesContributorsPulse● GatesSecuritySettingsDeploymentsPipelineInsightsAgents✨ Explain✨ Ask AI✨ Workspace✨ Spec✨ Tests▓ Debt Map✨ NL Search🏛 Archaeology
Blame · Line-by-line history
INCIDENT-2026-07-13-deploy-blip.md
Each line is annotated with the commit that last touched it. Click any SHA to jump to that commit and see the surrounding change.
| 94b5edf | 1 | # Incident — 2026-07-13 ~12:42 UTC: brief public-site downtime during auto-deploy |
| 2 | ||
| 3 | **Status:** resolved (self-recovered). No code fix required. | |
| 4 | **Duration:** < 1 minute (monitor reported "DOWN for ~0 min"). | |
| 5 | **Impact:** https://gluecron.com briefly unreachable while the app container was replaced. | |
| 6 | ||
| 7 | ## Timeline (UTC) | |
| 8 | ||
| 9 | - 12:42:06 — `gluecron-update.service` (minutely auto-deploy timer) picked up commit | |
| 10 | `41d6ea3` ("fix(backup): dump the REAL production DB (Neon), add proven restore drill"), | |
| 11 | rebuilt the image and **replaced** `gluecron-gluecron-1` via | |
| 12 | `docker compose -f docker-compose.standalone.yml up -d --build`. | |
| 13 | - 12:42:12 — deploy log: `deploy complete: 41d6ea3… (app healthy)`. Migrations were | |
| 14 | all no-ops ("already applied, skipping"). | |
| 15 | - ~12:42 — external monitor caught the container-replacement window and fired a | |
| 16 | self-heal alert. | |
| 17 | - 12:46:20 — self-heal session started; site already serving HTTP 200 in ~0.1 s | |
| 18 | (5/5 probes). Container healthy, `restarts=0`, `oom=false`, exit code 0. | |
| 19 | ||
| 20 | ## Root cause | |
| 21 | ||
| 22 | Not a crash. `docker compose up --build` replaces the app container in-place | |
| 23 | (stop-then-start), so every deploy has a short window where Traefik has no healthy | |
| 24 | upstream. The monitor's probe landed inside that window. The deploy itself succeeded. | |
| 25 | ||
| 26 | ## Actions taken | |
| 27 | ||
| 28 | - Diagnosed only — no restarts, no rollback, no DB operations. The safest fix was no fix. | |
| 29 | - Verified stability: repeated probes returned HTTP 200; container health checks passing. | |
| 30 | - Ran `bun run typecheck` on the deployed commit: **45 pre-existing TS errors across | |
| 31 | 15 files** (top offenders: `src/routes/oauth.tsx` 9, `src/routes/well-known.ts` 5, | |
| 32 | `src/routes/mcp.ts` 5). These pre-date this session; Bun strips types at runtime so | |
| 33 | they don't affect prod, but they should be burned down before `tsc` can gate deploys. | |
| 34 | - Build verification: the production Docker image build at 12:42 completed and passed | |
| 35 | the app health gate (no separate `build` script exists; the image build is the build). | |
| 36 | ||
| 37 | ## Follow-ups (not done here) | |
| 38 | ||
| 39 | 1. **Zero-downtime deploys** — the minutely auto-deploy replaces the container on every | |
| 40 | commit, so every push causes a blip like this one (and can re-trigger the self-heal | |
| 41 | monitor). Consider start-first replacement (bring up the new container, health-check | |
| 42 | it, then swap Traefik routing) or two replicas behind Traefik. | |
| 43 | 2. **Monitor grace window** — suppress the alert if a deploy is in flight | |
| 44 | (`gluecron-update.service` active) or require 2 consecutive failed probes. | |
| 45 | 3. **Type-error burn-down** — 45 errors block using `bun run typecheck` as a deploy gate. | |
| 46 | 4. **Untracked leftovers** — `/root/gluecron/docker-compose.traefik.yml` (unused draft, | |
| 47 | superseded by `docker-compose.override.yml`) and several `.env.bak.*` / | |
| 48 | `docker-compose.standalone.yml.bak.*` files in `/opt/gluecron` should be cleaned up | |
| 49 | or gitignored. |