Pre-launch — Gluecron is in final validation. Public signups and git hosting for non-owner users open after launch review.
ccantynz/Gluecron.com
Commitff4423bunknown_key

fix(deploy): unify Hetzner deploy paths — one canonical script (Level 4 partial)

fix(deploy): unify Hetzner deploy paths — one canonical script (Level 4 partial)

Phase D of the reliability sweep. AUDIT-v2.md P1 #11 flagged that
`scripts/self-deploy.sh` and `.github/workflows/hetzner-deploy.yml`
had drifted into two divergent deploy scripts maintaining a vague
parallel existence. Bugs fixed in one didn't propagate to the other,
which means the box-side and GitHub-side deploys could produce
different outcomes for the same commit.

Now both paths invoke ONE script — `scripts/self-deploy.sh` — which
remains the source of truth for what "deploying gluecron" means.

Changes:

  1. scripts/self-deploy.sh — `set -euo pipefail` → `set -Eeuxo pipefail`
     (adds command tracing). The script already logs to
     /var/log/gluecron-self-deploy.log; the trace output goes through
     the same `>>"$LOG" 2>&1` redirect on the detached re-exec line,
     so it ends up captured in the log. When this fails, the trace
     tells us exactly which command broke — same diagnostic we got
     in the GH workflow but for the post-receive path too.

  2. .github/workflows/hetzner-deploy.yml — the inline Deploy script
     now does only:
        cd /opt/gluecron
        git remote set-url origin <github URL>
        git fetch --prune origin main
        git reset --hard origin/main
        bash scripts/self-deploy.sh --inline
     The `--inline` flag bypasses self-deploy's normal background
     re-exec via systemd-run so the SSH session blocks until the
     full deploy (install/migrate/build/restart/healthz/smoke/
     rollback) completes. The SSH action's exit code now mirrors
     the script's exit code, so GH Actions correctly reports red
     when ANY step fails — including the smoke test and rollback.

What we gained vs the previous GH workflow:
  - Real post-deploy smoke suite (was missing — bun run scripts/post-deploy-smoke.ts)
  - Healthz polling with 30s budget (was a 6s pass/fail loop only)
  - Automatic rollback to previous SHA on failure (was missing — the
    GH workflow had a separate rollback step but self-deploy.sh's
    inline rollback is more thorough)
  - bun build --compile to a static binary (was missing — interpreted
    bun run still works as fallback inside the script)
  - Per-step notify_step events to /api/events/deploy/* for the live
    /admin/deploys timeline (was firing from the workflow inline; now
    fires from the canonical script regardless of trigger path)

What self-deploy.sh gained from this:
  - Loud failure tracing via `set -x` (was opaque on failure)

Tests: 2003/2003 pass. tsc clean. The workflow YAML lint passes
locally (no schema-validated CI for it).

After this lands, the next push to main triggers a GH Hetzner deploy
that runs the full canonical script with rollback safety net. The
"silent deploy failure" failure mode AUDIT-v2.md flagged is mitigated
by: (a) traced output, (b) automatic rollback, (c) Phase C's AI
incident responder on status=failed.
Claude committed on May 16, 2026Parent: 89a0761
2 files changed+31−42ff4423b22d07418c40ecab1dd44a1cddb95283fa
2 changed files+31−42
Modified.github/workflows/hetzner-deploy.yml+25−41View fileUnifiedSplit
147147 command_timeout: 8m
148148 script_stop: true
149149 envs: GH_TOKEN,GH_REPO,GH_SHA
150 # FIRE-DRILL DEPLOY (2026-05-16):
150 # UNIFIED DEPLOY (2026-05-16 reliability sweep, Phase D).
151151 #
152 # The original script (notify_step + cache hash + bun build
153 # --compile + migration verifier + systemd unit rewrite + …)
154 # was aborting silently between "Deploying SHA: …" and any
155 # other output — under `set -euo pipefail`, no visible stderr,
156 # no clue which line. The systemd unit had not been restarted
157 # in 17 hours of failed deploys.
152 # Both deploy paths now share ONE source of truth:
153 # `scripts/self-deploy.sh`. The GitHub Actions Hetzner
154 # workflow (this file) and the post-receive hook on the
155 # gluecron-hosted git server both invoke the same script.
156 # Drift between deploy paths — flagged in AUDIT-v2.md P1
157 # #11 — is eliminated.
158158 #
159 # This is the minimum viable replacement: trace every line
160 # via `set -x` so failures are visible, skip the silent-fail
161 # observability calls, do `git fetch + reset + bun install +
162 # systemctl restart` and nothing else. We can re-add the
163 # compile / migrate / cache-hash later — they were optional
164 # optimisations, the service worker rip-out only needs the
165 # new source code + a restart to land.
159 # The script handles: bun install, db migrations, bun build
160 # --compile, BUILD_SHA pin into systemd drop-in, systemctl
161 # restart, healthz wait, post-deploy smoke suite, AND
162 # automatic rollback to the previous SHA on failure.
163 #
164 # All we do here:
165 # 1. Repair the git remote (still needed because the box
166 # historically pointed at the self-hosted URL which 404s).
167 # 2. git fetch + reset --hard to the new SHA.
168 # 3. `bash scripts/self-deploy.sh --inline` (the `--inline`
169 # flag prevents the script's normal background re-exec
170 # so the SSH session blocks until completion, which is
171 # what we want for GH Actions to mirror the script's
172 # exit code).
166173 script: |
167174 set -Eeuxo pipefail
168175 cd /opt/gluecron
174181 new_sha=$(git rev-parse HEAD)
175182 echo "Deployed SHA on box: $new_sha"
176183
177 echo "=== STEP 2: bun install ==="
178 BUN=/root/.bun/bin/bun
179 "$BUN" install --frozen-lockfile
180
181 echo "=== STEP 3: run db migrations ==="
182 # Source /etc/gluecron.env so DATABASE_URL is available to
183 # migrate.ts. Hard-fail if any migration errors — the previous
184 # `|| echo WARN` swallowed migration failures and left the DB
185 # out of sync with the schema, which produces 500s on every
186 # page that SELECTs a column added by a missed migration.
187 set -a
188 source /etc/gluecron.env
189 set +a
190 "$BUN" run src/db/migrate.ts
191
192 echo "=== STEP 4: pin BUILD_SHA in systemd drop-in ==="
193 mkdir -p /etc/systemd/system/gluecron.service.d
194 cat > /etc/systemd/system/gluecron.service.d/build-sha.conf <<EOF
195 [Service]
196 Environment="BUILD_SHA=$new_sha"
197 Environment="BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
198 EOF
199 systemctl daemon-reload
200
201 echo "=== STEP 5: restart gluecron systemd unit ==="
202 systemctl restart gluecron
203 echo "=== DONE: service restarted with $new_sha ==="
184 echo "=== STEP 2: hand off to scripts/self-deploy.sh (the canonical deploy path) ==="
185 chmod +x scripts/self-deploy.sh
186 bash scripts/self-deploy.sh --inline
187 echo "=== DONE: self-deploy.sh exited cleanly ==="
204188
205189 # ─── 3. Smoke-test the deployed app on the box ──────────────────────
206190 # We SSH back in and curl localhost:3010/healthz directly. This tests
Modifiedscripts/self-deploy.sh+6−1View fileUnifiedSplit
2727# TODO(ops): configure /etc/logrotate.d/gluecron-self-deploy for the log.
2828# =============================================================================
2929
30set -euo pipefail
30# `set -E` so traps propagate into subshells; `-x` traces every command
31# to stderr (captured into $LOG via the `>>"$LOG" 2>&1` redirects on the
32# detached re-exec line). Reliability sweep 2026-05-16: when this script
33# fails, the trace tells us EXACTLY which line broke instead of leaving
34# us guessing as we did for 17 hours of failed Hetzner deploys.
35set -Eeuxo pipefail
3136
3237WORKING_DIR="${GLUECRON_WORKING_DIR:-/opt/gluecron}"
3338LOG="${GLUECRON_SELF_DEPLOY_LOG:-/var/log/gluecron-self-deploy.log}"
3439