Pre-launch — Gluecron is in final validation. Public signups and git hosting for non-owner users open after launch review.
ccantynz/Gluecron.com
Commit392689d

fix(backup): refuse to report success for a backup missing the repos

fix(backup): refuse to report success for a backup missing the repos

If GIT_REPOS_PATH did not exist the script logged a WARN and carried on.
The tarball was then never created, the `-f` guard on the upload skipped
it silently, retention ran, and the dead-man's-switch got pinged. A green
healthcheck next to a database-only backup — and the DB stores metadata
ONLY, so it cannot reconstruct a single repository. That is precisely the
loss this script exists to prevent, and it would have looked like success
indefinitely.

Not hypothetical. Verified read-only on the live host: the default
/opt/gluecron/repos does NOT exist there — the repos are in a docker
volume — and while GIT_REPOS_PATH is set correctly in /opt/gluecron/.env,
it is absent from /etc/gluecron.env, which is where this script's own
header tells you to configure it and where a systemd unit would read it.
Installing the timer as documented would have produced exactly the silent
DB-only backup described above.

Also found while checking: backup-offsite.sh is not installed as any
systemd unit on the box. The jarvis-backup timer runs an unrelated Jarvis
memory script. So there are currently NO offsite Gluecron backups at all
— which matters because CLAUDE.md keeps GitHub as the emergency source
"until offsite backups are armed and a restore drill has passed". Arming
the timer is an owner action; this commit makes it safe to arm.

Three silent-partial-success paths now fail loudly, each with a
deliberate, default-off escape hatch rather than a hard block:
  - missing repos dir            -> ALLOW_DB_ONLY_BACKUP=1
  - repos dir present but empty  -> ALLOW_EMPTY_REPOS_BACKUP=1
  - no offsite destination       -> ALLOW_LOCAL_ONLY_BACKUP=1

The empty case is the nastier one: tar succeeds on a wrong-but-existing
path and uploads a valid near-empty archive, which reads healthier than
an outright failure. The guard counts bare repos (<owner>/<repo>.git,
falling back to HEAD files) and logs the count alongside the size, so a
shrinking backup is visible rather than inferred.

The upload now keys on an explicit repos_included flag instead of a bare
`-f` test, since a tarball missing after we created it is a bug, not a
reason to skip.

Exercised end-to-end against fixtures rather than reasoned about: missing
path exits 1, empty path exits 1, no-remote exits 1, each escape hatch
proceeds, and a full local-only run reaches "done." with exit 0 and logs
"1 repos".

Suite: 3504 pass, 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ccantynz-alt committed on July 29, 2026Parent: 6ae664d
1 file changed+74−7392689d2414fa0b5a54fe0158576de2b08618d52
1 changed file+74−7
Modifiedscripts/backup-offsite.sh+74−7View fileUnifiedSplit
1818# Required env (set in /etc/gluecron.env):
1919# DATABASE_URL Neon connection string (postgresql://...)
2020# BACKUP_RCLONE_REMOTE e.g. r2:gluecron-backups (rclone must be configured)
21# GIT_REPOS_PATH bare repos dir. REQUIRED IN PRACTICE: the default
22# below (/opt/gluecron/repos) does NOT exist on the
23# live host, where the repos are in a docker volume at
24# /var/lib/docker/volumes/gluecron_git-repos/_data.
25# The app reads this from /opt/gluecron/.env, but a
26# systemd unit for THIS script reads
27# /etc/gluecron.env, so it must be set there too.
2128# Optional env:
22# GIT_REPOS_PATH bare repos dir (default: /opt/gluecron/repos)
2329# BACKUP_DIR local staging dir (default: /opt/gluecron/backups)
2430# RETAIN_DAYS local + offsite retention (default: 14)
2531# HEALTHCHECK_PING_URL https://hc-ping.com/<uuid> (healthchecks.io)
32#
33# Escape hatches — all default OFF, because each one makes the backup weaker
34# than it looks. Set deliberately, never to quiet a failure:
35# ALLOW_DB_ONLY_BACKUP=1 proceed with no repos snapshot
36# ALLOW_EMPTY_REPOS_BACKUP=1 proceed when the repos dir holds no repos
37# ALLOW_LOCAL_ONLY_BACKUP=1 proceed with no offsite destination
2638set -euo pipefail
2739
2840GIT_REPOS_PATH="${GIT_REPOS_PATH:-/opt/gluecron/repos}"
3850
3951mkdir -p "$BACKUP_DIR"
4052ts=$(date +%Y%m%d-%H%M%S)
53repos_included=0
4154repos_out="$BACKUP_DIR/repos-$ts.tar.gz"
4255db_out="$BACKUP_DIR/db-$ts.dump"
4356
4457# 1. Bare git repos — the irreplaceable half. tar the whole tree; bare repos
4558# are self-contained so a plain tar is a valid, restorable snapshot.
46if [ -d "$GIT_REPOS_PATH" ]; then
47 tar -czf "$repos_out" -C "$(dirname "$GIT_REPOS_PATH")" "$(basename "$GIT_REPOS_PATH")"
48 log "repos snapshot: $repos_out ($(du -h "$repos_out" | cut -f1))"
59#
60# A missing or empty repos path is FATAL, not a warning. It used to log a
61# WARN and carry on: the tarball was then never created, the `-f` guard on
62# the upload below skipped it silently, and the run still pinged the
63# dead-man's-switch. The result was a green healthcheck next to a DB-only
64# backup — and the DB stores metadata ONLY, so it cannot reconstruct a
65# single repository. That is the exact failure this script exists to
66# prevent, and it would have looked like success indefinitely.
67#
68# Not hypothetical: the default below is /opt/gluecron/repos, which does
69# not exist on the live host — the repos live in a docker volume. The
70# correct GIT_REPOS_PATH is set in /opt/gluecron/.env, but this script's
71# own header tells you to configure it in /etc/gluecron.env, where it is
72# absent. Installing the timer as documented would have produced exactly
73# the silent DB-only backup described above.
74if [ ! -d "$GIT_REPOS_PATH" ]; then
75 if [ "${ALLOW_DB_ONLY_BACKUP:-0}" = "1" ]; then
76 log "WARN: GIT_REPOS_PATH ($GIT_REPOS_PATH) does not exist — DB-only backup (ALLOW_DB_ONLY_BACKUP=1)"
77 repos_included=0
78 else
79 log "FATAL: GIT_REPOS_PATH ($GIT_REPOS_PATH) does not exist."
80 log " The database alone CANNOT reconstruct the repositories."
81 log " Set GIT_REPOS_PATH, or set ALLOW_DB_ONLY_BACKUP=1 to accept a DB-only backup."
82 exit 1
83 fi
4984else
50 log "WARN: GIT_REPOS_PATH ($GIT_REPOS_PATH) does not exist — skipping repos snapshot"
85 # An existing but WRONG path is the nastier case: tar succeeds and uploads a
86 # valid, near-empty archive, which looks healthier than an outright failure.
87 # Bare repos are laid out as <owner>/<repo>.git, so count those.
88 repo_count=$(find "$GIT_REPOS_PATH" -maxdepth 2 -type d -name '*.git' 2>/dev/null | wc -l | tr -d ' ')
89 if [ "$repo_count" -eq 0 ]; then
90 # Fall back to detecting bare repos by their HEAD file, in case a repo is
91 # stored without the .git suffix.
92 repo_count=$(find "$GIT_REPOS_PATH" -maxdepth 3 -type f -name HEAD 2>/dev/null | wc -l | tr -d ' ')
93 fi
94 if [ "$repo_count" -eq 0 ] && [ "${ALLOW_EMPTY_REPOS_BACKUP:-0}" != "1" ]; then
95 log "FATAL: no bare repositories found under $GIT_REPOS_PATH."
96 log " This is almost always the wrong path rather than a genuinely"
97 log " empty host — on the live box the repos are in a docker volume."
98 log " Set ALLOW_EMPTY_REPOS_BACKUP=1 if this host really has none."
99 exit 1
100 fi
101 tar -czf "$repos_out" -C "$(dirname "$GIT_REPOS_PATH")" "$(basename "$GIT_REPOS_PATH")"
102 log "repos snapshot: $repos_out ($(du -h "$repos_out" | cut -f1), $repo_count repos)"
103 repos_included=1
51104fi
52105
53106# 2. Database — custom format so pg_restore can do selective/parallel restore.
64117 rclone copy "$db_out" "$BACKUP_RCLONE_REMOTE/db/" \
65118 && log "offsite: db -> $BACKUP_RCLONE_REMOTE/db/" \
66119 || { log "FATAL: offsite db copy failed"; exit 1; }
67 if [ -f "$repos_out" ]; then
120 if [ "$repos_included" = "1" ]; then
121 # Explicit flag rather than a bare `-f` test: a missing tarball we EXPECTED
122 # to have is a bug, and the old `-f` guard turned exactly that into a
123 # silent skip that still reported success.
124 if [ ! -f "$repos_out" ]; then
125 log "FATAL: repos snapshot $repos_out is missing after being created"
126 exit 1
127 fi
68128 rclone copy "$repos_out" "$BACKUP_RCLONE_REMOTE/repos/" \
69129 && log "offsite: repos -> $BACKUP_RCLONE_REMOTE/repos/" \
70130 || { log "FATAL: offsite repos copy failed"; exit 1; }
72132 # Prune offsite copies older than retention (best-effort).
73133 rclone delete --min-age "${RETAIN_DAYS}d" "$BACKUP_RCLONE_REMOTE/db/" 2>/dev/null || true
74134 rclone delete --min-age "${RETAIN_DAYS}d" "$BACKUP_RCLONE_REMOTE/repos/" 2>/dev/null || true
135elif [ "${ALLOW_LOCAL_ONLY_BACKUP:-0}" = "1" ]; then
136 log "WARN: BACKUP_RCLONE_REMOTE not set — backup is LOCAL ONLY (ALLOW_LOCAL_ONLY_BACKUP=1)"
75137else
76 log "WARN: BACKUP_RCLONE_REMOTE not set — backup is LOCAL ONLY (not safe as the sole copy once GitHub is gone)"
138 # Same class of lie as the skipped repos snapshot: a backup sitting on the
139 # box it is meant to protect is not a backup, and pinging the
140 # dead-man's-switch for one tells you durability is fine when it is not.
141 log "FATAL: BACKUP_RCLONE_REMOTE is not set — this backup would be LOCAL ONLY."
142 log " Set it, or set ALLOW_LOCAL_ONLY_BACKUP=1 to accept a local-only run."
143 exit 1
77144fi
78145
79146# 4. Local retention.
80147