Commit41d6ea3
fix(backup): dump the REAL production DB (Neon), add proven restore drill
fix(backup): dump the REAL production DB (Neon), add proven restore drill
Critical DR fix. The daily backup timer ran scripts/backup.sh which dumped the
LOCAL postgres container (`$COMPOSE exec postgres pg_dump`) — but the app uses
Neon. So every daily backup captured empty/stale data: false confidence, real
production data protected only by Neon's PITR.
backup.sh now dumps Neon:
- Resolves DATABASE_URL from the running app container (always matches what the
app connects to), and cleans it — the box's .env has a MALFORMED DATABASE_URL
(two connstrings mashed on one line, works only because Bun's pg client is
lenient; strict pg_dump chokes). `${URL%%DATABASE_URL=*}` yields the clean
Neon URL and is a no-op once the .env is fixed.
- Uses an ephemeral postgres:17 container for pg_dump (Neon is PG17; the box's
local client is PG16 and refuses a newer server). Custom format, verified
non-empty (>=20 objects) before it's kept. Adds a self-owned Tailscale scp
offsite option (BACKUP_SCP_TARGET) alongside the rclone one.
restore-drill.sh (new): spins up a throwaway postgres:17, restores the latest
dump, and compares users/repos/issues/PRs/oauth_apps counts against live Neon.
Verified on the box today: 4.4M dump, 367 objects, all 5 tables match — the
first proven-restorable backup of production. Zero external dependencies.
Follow-up flagged: the malformed .env DATABASE_URL should be repaired (fragile).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>2 files changed+153−1341d6ea390fb2baf7b92ff3199722167011c46171
2 changed files+153−13
Modifiedscripts/backup.sh+72−13View fileUnifiedSplit
@@ -1,37 +1,96 @@
11
22#
3# Daily Postgres backup for the standalone box. Installed as a systemd timer by
4# scripts/standalone-deploy.sh. Keeps 14 days locally; optionally copies offsite
5# and pings a dead-man's-switch monitor.
3# Daily production-database backup for the standalone/Coolify box.
4# Installed as a systemd timer by scripts/standalone-deploy.sh.
5#
6# WHAT IT BACKS UP (corrected 2026-07-14): the REAL production database, which
7# is **Neon** — reached via the app container's own DATABASE_URL. The compose
8# also runs an unused local `postgres` container; the previous version of this
9# script dumped THAT by mistake, so daily backups were capturing empty/stale
10# data. Do not reintroduce `$COMPOSE exec postgres pg_dump`.
11#
12# HOW: Neon runs Postgres 17, so we dump with an ephemeral `postgres:17`
13# container (the box's local client is 16 and refuses a newer server). The
14# app's DATABASE_URL on this box is malformed (two connstrings mashed on one
15# line — a fragile latent bug), so we take the clean Neon URL up to the second
16# "DATABASE_URL=" marker. `${URL%%DATABASE_URL=*}` is a no-op once the .env is
17# fixed, so this stays correct either way.
18#
19# Output: custom-format (`-Fc`) dumps — compressed + restorable selectively via
20# pg_restore. Compatible with scripts/restore.sh and scripts/backup-restore-drill.sh.
621#
722# Optional env (set in /opt/gluecron/.env):
823# BACKUP_RCLONE_REMOTE e.g. r2:gluecron-backups (needs rclone configured)
9# HEALTHCHECK_PING_URL e.g. https://hc-ping.com/<uuid> (healthchecks.io)
24# BACKUP_SCP_TARGET e.g. user@100.x.y.z:/backups (self-owned offsite over Tailscale)
25# HEALTHCHECK_PING_URL e.g. https://hc-ping.com/<uuid> (dead-man's-switch)
1026set -euo pipefail
1127
1228REPO_DIR="/opt/gluecron"
1329BACKUP_DIR="$REPO_DIR/backups"
1430RETAIN_DAYS=14
15COMPOSE="docker compose -f docker-compose.standalone.yml"
31APP_CONTAINER="gluecron-gluecron-1"
32PG_IMAGE="postgres:17-alpine"
1633
1734cd "$REPO_DIR"
1835mkdir -p "$BACKUP_DIR"
1936ts=$(date +%Y%m%d-%H%M%S)
20out="$BACKUP_DIR/gluecron-$ts.sql.gz"
37out="$BACKUP_DIR/gluecron-db-$ts.dump"
38
39# Resolve the live DATABASE_URL from the running app, then clean it. Reading it
40# from the container (not .env) means the backup always matches what the app
41# actually connects to.
42raw_url="$(docker exec "$APP_CONTAINER" printenv DATABASE_URL)"
43clean_url="${raw_url%%DATABASE_URL=*}"
44if [ -z "$clean_url" ]; then
45 echo "$(date -Is) FATAL: could not resolve DATABASE_URL from $APP_CONTAINER" >&2
46 exit 1
47fi
2148
22$COMPOSE exec -T postgres pg_dump -U gluecron gluecron | gzip > "$out"
49# Dump Neon with a matching-major-version client. Custom format, no owner/ACLs
50# (portable across restore targets). Password + URL travel only in container
51# env; the pg_dump runs via `sh -c` so $NEONURL expands INSIDE the container
52# (host-side expansion would be empty and trip `set -u`).
53docker run --rm -e NEONURL="$clean_url" -e OUTFILE="/backups/$(basename "$out")" \
54 -v "$BACKUP_DIR:/backups" "$PG_IMAGE" \
55 sh -c 'pg_dump "$NEONURL" --format=custom --no-owner --no-privileges -f "$OUTFILE"' \
56 2>/tmp/backup-err.$$ || {
57 echo "$(date -Is) FATAL: pg_dump failed:" >&2
58 cat /tmp/backup-err.$$ >&2
59 rm -f /tmp/backup-err.$$ "$out"
60 exit 1
61 }
62rm -f /tmp/backup-err.$$
2363
24# Retention
25find "$BACKUP_DIR" -name 'gluecron-*.sql.gz' -mtime +$RETAIN_DAYS -delete
64# Sanity: a valid custom-format dump must list objects. Guards against a
65# 0-byte/corrupt file being counted as a good backup.
66obj_count="$(docker run --rm -e F="/backups/$(basename "$out")" -v "$BACKUP_DIR:/backups" "$PG_IMAGE" \
67 sh -c 'pg_restore --list "$F"' 2>/dev/null | grep -cE 'TABLE DATA|TABLE|SEQUENCE' || true)"
68if [ "${obj_count:-0}" -lt 20 ]; then
69 echo "$(date -Is) FATAL: dump looks empty ($obj_count objects) — refusing to keep it" >&2
70 rm -f "$out"
71 exit 1
72fi
73
74size="$(du -h "$out" | cut -f1)"
75
76# Retention — keep RETAIN_DAYS of daily dumps.
77find "$BACKUP_DIR" -name 'gluecron-db-*.dump' -mtime +$RETAIN_DAYS -delete
78
79# Optional self-owned offsite copy over Tailscale (preferred — no third party).
80if [ -n "${BACKUP_SCP_TARGET:-}" ]; then
81 scp -o BatchMode=yes -o StrictHostKeyChecking=accept-new "$out" "$BACKUP_SCP_TARGET/" \
82 && echo "$(date -Is) offsite scp ok -> $BACKUP_SCP_TARGET" \
83 || echo "$(date -Is) WARN: offsite scp failed"
84fi
2685
27# Optional offsite copy
86# Optional rclone offsite copy (if you do use a bucket).
2887if [ -n "${BACKUP_RCLONE_REMOTE:-}" ] && command -v rclone >/dev/null 2>&1; then
29 rclone copy "$out" "$BACKUP_RCLONE_REMOTE" || echo "WARN: offsite copy failed"
88 rclone copy "$out" "$BACKUP_RCLONE_REMOTE" || echo "$(date -Is) WARN: rclone copy failed"
3089fi
3190
32# Optional dead-man's-switch heartbeat (alerts you if a backup is ever missed)
91# Optional dead-man's-switch heartbeat (alerts if a backup is ever missed).
3392if [ -n "${HEALTHCHECK_PING_URL:-}" ]; then
3493 curl -fsS -m 10 "$HEALTHCHECK_PING_URL" >/dev/null 2>&1 || true
3594fi
3695
37echo "$(date -Is) backup written: $out ($(du -h "$out" | cut -f1))"
96echo "$(date -Is) backup written: $out ($size, $obj_count objects)"
Addedscripts/restore-drill.sh+81−0View fileUnifiedSplit
@@ -0,0 +1,81 @@
1
2#
3# Restore drill — proves the latest production (Neon) backup actually restores.
4# "A backup you have never restored is not a backup, it is a hope."
5#
6# Self-contained, zero external dependencies: spins up a THROWAWAY postgres:17
7# container, pg_restores the newest dump into it, then compares key row counts
8# against the live Neon database. Tears the container down unconditionally.
9#
10# Exit 0 = restore verified (counts match). Non-zero = drill FAILED — the
11# backup is NOT trustworthy; investigate before relying on it.
12#
13# Usage: bash scripts/restore-drill.sh
14set -uo pipefail
15
16BACKUP_DIR="/opt/gluecron/backups"
17APP_CONTAINER="gluecron-gluecron-1"
18PG_IMAGE="postgres:17-alpine"
19DRILL="gluecron-restore-drill-$$"
20
21log() { echo "$(date -Is) [drill] $*"; }
22cleanup() { docker rm -f "$DRILL" >/dev/null 2>&1 || true; }
23trap cleanup EXIT
24
25dump="$(ls -t "$BACKUP_DIR"/gluecron-db-*.dump 2>/dev/null | head -1)"
26if [ -z "$dump" ]; then
27 log "FAIL: no backup found in $BACKUP_DIR"
28 exit 1
29fi
30log "restoring: $dump"
31
32raw_url="$(docker exec "$APP_CONTAINER" printenv DATABASE_URL)"
33clean_url="${raw_url%%DATABASE_URL=*}"
34
35# Throwaway target instance.
36docker run -d --name "$DRILL" -e POSTGRES_PASSWORD=drill -e POSTGRES_DB=drill \
37 "$PG_IMAGE" >/dev/null
38for _ in $(seq 1 30); do
39 docker exec "$DRILL" pg_isready -U postgres -d drill >/dev/null 2>&1 && break
40 sleep 1
41done
42
43docker cp "$dump" "$DRILL":/tmp/d.dump
44# Ignore benign restore noise (extensions, comments); we judge by row counts.
45docker exec "$DRILL" pg_restore --no-owner --no-privileges -U postgres -d drill /tmp/d.dump \
46 >/dev/null 2>&1 || true
47
48fails=0
49checked=0
50for tbl in users repositories issues pull_requests oauth_apps; do
51 restored="$(docker exec "$DRILL" psql -At -U postgres -d drill \
52 -c "SELECT count(*) FROM $tbl" 2>/dev/null | tr -d '[:space:]')"
53 # $U/$TBL expand inside the container (host-side would be empty).
54 live="$(docker run --rm -e U="$clean_url" -e TBL="$tbl" "$PG_IMAGE" \
55 sh -c 'psql -At "$U" -c "SELECT count(*) FROM $TBL"' 2>/dev/null | tr -d '[:space:]')"
56 if [ -z "$restored" ] || [ -z "$live" ]; then
57 log "WARN: $tbl — could not read a count (restored='$restored' live='$live')"
58 fails=$((fails + 1))
59 continue
60 fi
61 checked=$((checked + 1))
62 if [ "$restored" = "$live" ]; then
63 log "OK $tbl: $restored rows (matches live)"
64 else
65 log "FAIL $tbl: restored=$restored live=$live (MISMATCH)"
66 fails=$((fails + 1))
67 fi
68done
69
70if [ "$checked" -eq 0 ]; then
71 log "FAIL: could not verify any table"
72 exit 1
73fi
74if [ "$fails" -ne 0 ]; then
75 log "FAIL: $fails check(s) failed — backup is NOT trustworthy"
76 exit 1
77fi
78
79# Record a success marker + metric for monitoring (optional consumers).
80echo "$(date +%s)" > "$BACKUP_DIR/.last-restore-drill-ok"
81log "PASS: restore verified against live Neon ($checked tables). Backup is good."
082