Commit3b77289
fix(deploy): auto-rollback on failed health check instead of leaving a broken container live
fix(deploy): auto-rollback on failed health check instead of leaving a broken container live Investigated true zero-downtime (dual-container blue-green + live Traefik dynamic-config swap) for the Coolify co-tenant box. Confirmed the pieces exist (Traefik's file provider runs --providers.file.watch=true, near-instant reload; coolify-proxy has sed) but a hand-rolled second container would need to fake Docker Compose's own bookkeeping (project labels, config-hash, volume naming) to stay compose-compatible for the next deploy and for the documented manual `docker compose up --build` fallback in CLAUDE.md -- getting that subtly wrong is a worse failure mode than the current blip. Not shipping that blind against a live production box; scoping down to something verified correct instead. The more serious bug this surfaced: a deploy that fails its health check was NOT actually failing safe. `compose up --build` recreates the container immediately, and only *then* does the script check health -- so a bad deploy destroyed the working container, left the broken one running (or crashlooping), and just exited 1. The "brief routing blip" incident (2026-07-13) was the good case; a real breaking bug currently means a full outage until a human notices and manually fixes it. Fix: tag the current image as `last-good` before every build. If the new container never turns healthy, automatically re-tag `last-good` as `latest`, redeploy it, and re-verify health -- restoring the last working version instead of leaving a broken one live. The failed SHA is written to `.last-failed-deploy-sha` so the same broken commit doesn't retry-and-fail-and-rollback every 60s forever; it waits for a newer commit (or a human clearing the marker). Verified: real image tag (gluecron-gluecron:latest), real network name (gluecron_default / coolify), and `--no-build` all confirmed against the actual box before writing this. bash -n clean.
1 file changed+67−73b772894b0a36ebbf024d13782193ef4fb979d44
1 changed file+67−7
Modifiedscripts/auto-update.sh+67−7View fileUnifiedSplit
@@ -5,11 +5,22 @@
55# it pulls, rebuilds, and runs migrations. Push -> live in ~1-2 min, hands-off.
66#
77# Exits immediately (cheap) when there is nothing new, so a tight interval is fine.
8#
9# Rollback (2026-07-21): a bad deploy used to leave the box with the OLD
10# container already destroyed and the NEW one unhealthy -- a failed health
11# check just exited 1 with nothing serving traffic until a human noticed and
12# fixed it by hand. The previous image is now tagged before every build, and
13# a failed health check re-deploys it automatically instead of leaving a
14# broken container live. A SHA that fails is remembered (FAILED_MARKER) so a
15# persistently broken commit doesn't retry-and-fail every 60s forever -- it
16# waits for a newer commit (or a human clearing the marker) instead.
817set -euo pipefail
918
1019REPO_DIR="/opt/gluecron"
1120BRANCH="main"
1221COMPOSE="docker compose -f docker-compose.standalone.yml"
22IMAGE="gluecron-gluecron"
23FAILED_MARKER="$REPO_DIR/.last-failed-deploy-sha"
1324
1425cd "$REPO_DIR"
1526git fetch origin "$BRANCH" --quiet
@@ -18,7 +29,19 @@ local_sha=$(git rev-parse HEAD)
1829remote_sha=$(git rev-parse "origin/$BRANCH")
1930[ "$local_sha" = "$remote_sha" ] && exit 0
2031
32if [ -f "$FAILED_MARKER" ] && [ "$(cat "$FAILED_MARKER")" = "$remote_sha" ]; then
33 # Already tried and rolled back from this exact commit -- don't loop.
34 exit 0
35fi
36
2137echo "$(date -Is) deploying $local_sha -> $remote_sha"
38
39# Preserve the last-known-good image BEFORE touching anything, so a bad
40# deploy can roll back to exactly what was running a moment ago. `|| true`:
41# the very first deploy on a fresh box has no prior image to tag.
42docker tag "$IMAGE:latest" "$IMAGE:last-good" 2>/dev/null || true
43prev_sha="$local_sha"
44
2245git reset --hard "origin/$BRANCH" # untracked .env / backups are preserved
2346
2447# NOTE: `|| true` is deliberate. On this Coolify co-tenant box the compose
@@ -28,7 +51,6 @@ git reset --hard "origin/$BRANCH" # untracked .env / backups are preserved
2851# the deploy BEFORE the coolify reattach + health gate below. We tolerate the
2952# partial failure here and instead gate on the app's OWN health further down.
3053$COMPOSE up -d --build || echo "$(date -Is) compose up returned non-zero (likely the co-tenant caddy port conflict) — continuing to app health gate"
31sleep 5
3254
3355# Co-tenant ingress reattach (Coolify boxes only).
3456# `docker compose up --build` recreates the gluecron container, which DROPS
@@ -38,7 +60,9 @@ sleep 5
3860# /traefik/dynamic/gluecron.yaml in the coolify-proxy container. Without this
3961# reattach, every deploy 502s the site until someone reconnects by hand.
4062# Idempotent: a no-op if already attached; skipped entirely on a dedicated VPS
41# that has no "coolify" network.
63# that has no "coolify" network. Traefik's file provider watches with fsnotify
64# (--providers.file.watch=true), so it notices the container is reachable
65# again within a second or two of the reconnect — no manual nudge needed.
4266if docker network inspect coolify >/dev/null 2>&1; then
4367 docker network connect coolify gluecron-gluecron-1 2>/dev/null \
4468 && echo "$(date -Is) reattached gluecron to coolify network" \
@@ -49,8 +73,8 @@ $COMPOSE exec -T gluecron bun run db:migrate || true
4973docker image prune -f >/dev/null 2>&1 || true
5074
5175# App health gate — the real success signal (not caddy). Poll the container's
52# own /healthz; if the app itself never comes up, exit non-zero so the systemd
53# unit records a failed deploy instead of a silent green.
76# own /healthz; if the app itself never comes up, roll back instead of
77# leaving a broken deploy live.
5478healthy=0
5579for _ in $(seq 1 20); do
5680 if docker exec gluecron-gluecron-1 wget -qO- --timeout=4 http://localhost:3000/healthz >/dev/null 2>&1; then
@@ -58,9 +82,45 @@ for _ in $(seq 1 20); do
5882 fi
5983 sleep 3
6084done
61if [ "$healthy" != "1" ]; then
62 echo "$(date -Is) DEPLOY FAILED: app /healthz never came up for $remote_sha" >&2
85
86if [ "$healthy" = "1" ]; then
87 rm -f "$FAILED_MARKER"
88 echo "$(date -Is) deploy complete: $remote_sha (app healthy)"
89 exit 0
90fi
91
92echo "$(date -Is) DEPLOY FAILED: app /healthz never came up for $remote_sha" >&2
93docker logs --tail 80 gluecron-gluecron-1 >&2 || true
94
95if ! docker image inspect "$IMAGE:last-good" >/dev/null 2>&1; then
96 echo "$(date -Is) no last-good image to roll back to (first deploy on this box?) — human intervention required" >&2
97 echo "$remote_sha" > "$FAILED_MARKER"
98 exit 1
99fi
100
101echo "$(date -Is) rolling back to $prev_sha" >&2
102git reset --hard "$prev_sha"
103docker tag "$IMAGE:last-good" "$IMAGE:latest"
104$COMPOSE up -d --no-build || echo "$(date -Is) compose up (rollback) returned non-zero (likely the co-tenant caddy port conflict) — continuing to health gate"
105
106if docker network inspect coolify >/dev/null 2>&1; then
107 docker network connect coolify gluecron-gluecron-1 2>/dev/null || true
108fi
109
110rb_healthy=0
111for _ in $(seq 1 20); do
112 if docker exec gluecron-gluecron-1 wget -qO- --timeout=4 http://localhost:3000/healthz >/dev/null 2>&1; then
113 rb_healthy=1; break
114 fi
115 sleep 3
116done
117
118echo "$remote_sha" > "$FAILED_MARKER"
119
120if [ "$rb_healthy" = "1" ]; then
121 echo "$(date -Is) rollback to $prev_sha succeeded — app healthy again. $remote_sha will not be retried automatically; push a fix or clear $FAILED_MARKER to try again." >&2
63122 exit 1
64123fi
65124
66echo "$(date -Is) deploy complete: $remote_sha (app healthy)"
125echo "$(date -Is) ROLLBACK ALSO FAILED — human intervention required NOW (site may be down)" >&2
126exit 1
67127