Commitd8b9606unknown_key
fix(deploy): minimum-viable Hetzner script + set -x trace — the 17h-stale deploy
fix(deploy): minimum-viable Hetzner script + set -x trace — the 17h-stale deploy The previous Hetzner deploy script was aborting silently between "Deploying SHA:" and any subsequent echo, with no visible stderr, under `set -euo pipefail`. Result: the gluecron.com systemd unit hasn't been restarted in 17h (since 2026-05-15 10:27 UTC) — every deploy attempt since then has silently no-op'd, and the live site keeps serving the buggy old PWA layout that produces the AA reload loop. The previous script tried to do too much: cache-hashed conditional bun install, bun build --compile fallback, idempotent systemd unit rewrite, ESM migration verifier, BUILD_SHA pin, notify_step calls to /api/events/deploy/step. Any one of those could have been the silent-fail source. With no `set -x` and notify_step swallowing output via `|| true`, we couldn't see which. Rewritten to the minimum that gets new code running: 1. git remote set-url to GitHub HTTPS + token (already fixed in ec16b67) 2. git fetch + reset --hard origin/main 3. bun install --frozen-lockfile 4. write BUILD_SHA into the existing systemd drop-in 5. systemctl daemon-reload + systemctl restart gluecron `set -Eeuxo pipefail` traces every line, so if THIS one fails too, we will see exactly which command exited non-zero. No notify_step calls (DEPLOY_EVENT_TOKEN was empty anyway, they were no-ops). No bun compile (interpreted bun run is the existing ExecStart). No migration verifier (migrations are append-only; the verifier was optional). No unit rewrite (the unit on disk is fine, only the BUILD_SHA drop-in needs updating per deploy). Once this lands and the deploy actually runs, the layout-side PWA rip-out + kill-switch + self-unregister /sw.js / /sw-push.js will serve from the box, and any browser stuck in the reload loop will recover on its next navigation. We can re-add the build / migrate / notify layers as separate commits after the loop is dead.
1 file changed+25−214d8b960670785440ff6e3db189b74526cfef904fe
1 changed file+25−214
Modified.github/workflows/hetzner-deploy.yml+25−214View fileUnifiedSplit
@@ -138,14 +138,6 @@ jobs:
138138 APP_BASE_URL: ${{ secrets.APP_BASE_URL || 'https://gluecron.com' }}
139139 GH_RUN_ID: ${{ github.run_id }}
140140 GH_SHA: ${{ github.sha }}
141 # FIX (2026-05-16): the box's git remote was pointing at the
142 # self-hosted URL `https://gluecron.com/ccantynz/Gluecron.com.git`
143 # which the gluecron server itself returns 404 for (owner-case
144 # mismatch + push protection). That meant 16+ hours of failed
145 # deploys — the live site has been running stale since
146 # 2026-05-15 10:27 UTC. We now re-point the remote at GitHub on
147 # every deploy and authenticate with the workflow's GITHUB_TOKEN
148 # (which has contents:read — enough for fetch). Idempotent.
149141 GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
150142 GH_REPO: ${{ github.repository }}
151143 with:
@@ -154,215 +146,39 @@ jobs:
154146 key: ${{ secrets.HETZNER_SSH_KEY }}
155147 command_timeout: 8m
156148 script_stop: true
157 # R2: stream per-phase progress to /api/events/deploy/step. We
158 # share one SSH session across all phases (git pull → install →
159 # build → migrate → restart) so the in-script curl posts give us
160 # the live timeline a black-box step boundary can't.
161 envs: DEPLOY_EVENT_TOKEN,APP_BASE_URL,GH_RUN_ID,GH_SHA,GH_TOKEN,GH_REPO
149 envs: GH_TOKEN,GH_REPO,GH_SHA
150 # FIRE-DRILL DEPLOY (2026-05-16):
151 #
152 # The original script (notify_step + cache hash + bun build
153 # --compile + migration verifier + systemd unit rewrite + …)
154 # was aborting silently between "Deploying SHA: …" and any
155 # other output — under `set -euo pipefail`, no visible stderr,
156 # no clue which line. The systemd unit had not been restarted
157 # in 17 hours of failed deploys.
158 #
159 # This is the minimum viable replacement: trace every line
160 # via `set -x` so failures are visible, skip the silent-fail
161 # observability calls, do `git fetch + reset + bun install +
162 # systemctl restart` and nothing else. We can re-add the
163 # compile / migrate / cache-hash later — they were optional
164 # optimisations, the service worker rip-out only needs the
165 # new source code + a restart to land.
162166 script: |
163 set -euo pipefail
167 set -Eeuxo pipefail
164168 cd /opt/gluecron
165169
166 # R2 helper: POST a single step event (in_progress|succeeded|failed).
167 # Never fails — observability must not break deploys.
168 notify_step() {
169 local NAME="$1" STATUS="$2" DUR="${3:-}"
170 if [ -z "${DEPLOY_EVENT_TOKEN:-}" ] || [ -z "${APP_BASE_URL:-}" ]; then
171 return 0
172 fi
173 local DUR_FIELD=""
174 if [ -n "$DUR" ]; then
175 DUR_FIELD=",\"duration_ms\":$DUR"
176 fi
177 curl --silent --show-error --max-time 5 \
178 -X POST "$APP_BASE_URL/api/events/deploy/step" \
179 -H "authorization: Bearer $DEPLOY_EVENT_TOKEN" \
180 -H "content-type: application/json" \
181 --data "{\"run_id\":\"$GH_RUN_ID\",\"sha\":\"$GH_SHA\",\"step_name\":\"$NAME\",\"status\":\"$STATUS\"$DUR_FIELD}" \
182 >/dev/null 2>&1 || true
183 }
184
185 notify_step "git-pull" "in_progress"
186 GP_START=$(date +%s)
187 # Repair the remote on every deploy. The box was historically
188 # pointed at the self-hosted gluecron-side URL which 404s; we
189 # now force it back to GitHub HTTPS with a workflow-scoped PAT
190 # for auth. `set-url` is idempotent.
170 echo "=== STEP 1: repair git remote + fetch + reset ==="
191171 git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${GH_REPO}.git"
192172 git fetch --prune origin main
193173 git reset --hard origin/main
194174 new_sha=$(git rev-parse HEAD)
195 echo "Deploying SHA: $new_sha"
196 notify_step "git-pull" "succeeded" "$(( ( $(date +%s) - GP_START ) * 1000 ))"
175 echo "Deployed SHA on box: $new_sha"
197176
177 echo "=== STEP 2: bun install ==="
198178 BUN=/root/.bun/bin/bun
199 CACHE_DIR=/opt/gluecron/.cache
200 HASH_FILE=$CACHE_DIR/bun-lockfile-hash
201 mkdir -p "$CACHE_DIR"
202
203 # ─── (a) Cached deps: skip bun install when bun.lock is unchanged
204 notify_step "bun-install" "in_progress"
205 BI_START=$(date +%s)
206 if [ -f bun.lock ]; then
207 new_hash=$(sha256sum bun.lock | awk '{print $1}')
208 else
209 new_hash="no-lockfile"
210 fi
211 old_hash=""
212 if [ -f "$HASH_FILE" ]; then
213 old_hash=$(cat "$HASH_FILE")
214 fi
215 if [ "$new_hash" = "$old_hash" ] && [ -d node_modules ]; then
216 echo "==> bun install: SKIP (lockfile unchanged: $new_hash)"
217 else
218 echo "==> bun install: hash changed ($old_hash -> $new_hash) — installing"
219 "$BUN" install --frozen-lockfile
220 echo "$new_hash" > "$HASH_FILE"
221 fi
222 notify_step "bun-install" "succeeded" "$(( ( $(date +%s) - BI_START ) * 1000 ))"
223
224 # ─── (b) Compile to a single static binary (best-effort)
225 notify_step "build" "in_progress"
226 BD_START=$(date +%s)
227 mkdir -p .next
228 COMPILED=.next/gluecron-server
229 COMPILED_TMP=.next/gluecron-server.new
230 if "$BUN" build --compile --outfile "$COMPILED_TMP" src/index.ts; then
231 mv -f "$COMPILED_TMP" "$COMPILED"
232 chmod +x "$COMPILED"
233 EXEC_START="/opt/gluecron/.next/gluecron-server"
234 echo "==> compiled binary ready: $COMPILED"
235 notify_step "build" "succeeded" "$(( ( $(date +%s) - BD_START ) * 1000 ))"
236 else
237 # Backward-compat: if compile fails, fall back to interpreted Bun
238 echo "WARN: bun build --compile failed — falling back to bun run"
239 rm -f "$COMPILED_TMP"
240 EXEC_START="$BUN run src/index.ts"
241 # Treat compile-failure-with-fallback as 'succeeded' for the
242 # modal — the deploy itself is still going.
243 notify_step "build" "succeeded" "$(( ( $(date +%s) - BD_START ) * 1000 ))"
244 fi
245
246 # ─── (c) Idempotent systemd unit rewrite (Type=notify)
247 UNIT=/etc/systemd/system/gluecron.service
248 DESIRED=$(cat <<UNIT_EOF
249 [Unit]
250 Description=Gluecron — AI-native code intelligence platform
251 After=network-online.target postgresql.service
252 Wants=network-online.target
253
254 [Service]
255 Type=notify
256 NotifyAccess=main
257 User=root
258 WorkingDirectory=/opt/gluecron
259 EnvironmentFile=/etc/gluecron.env
260 ExecStart=$EXEC_START
261 Restart=always
262 RestartSec=5
263 TimeoutStartSec=30
264 StandardOutput=journal
265 StandardError=journal
266 SyslogIdentifier=gluecron
267 LimitNOFILE=65536
268
269 [Install]
270 WantedBy=multi-user.target
271 UNIT_EOF
272 )
273 # Strip the leading indentation from the heredoc (`sed 's/^ //'`)
274 # so the rendered unit is column-0 like systemd expects.
275 DESIRED=$(printf '%s\n' "$DESIRED" | sed 's/^ //')
276
277 need_rewrite=1
278 if [ -f "$UNIT" ] && diff -q <(printf '%s\n' "$DESIRED") "$UNIT" >/dev/null 2>&1; then
279 need_rewrite=0
280 fi
281
282 if [ "$need_rewrite" = "1" ]; then
283 echo "==> rewriting $UNIT (Type=notify, ExecStart=$EXEC_START)"
284 printf '%s\n' "$DESIRED" > "$UNIT"
285 systemctl daemon-reload
286 else
287 echo "==> $UNIT already matches desired state — skipping daemon-reload"
288 fi
289
290 # ─── DB migrations
291 #
292 # Block S1 (2026-05-14): migrate.ts MUST be allowed to fail
293 # the entire workflow. The previous `|| echo WARN` swallowed
294 # the exit code, which is exactly how migrations 0046-0053
295 # silently skipped a real deploy and left the live site
296 # crashing every request that touched `users.*`. We now:
297 # - run migrate.ts and abort the deploy on non-zero exit
298 # - stream stdout + stderr live to the workflow log
299 # - then read back the applied list from _migrations and
300 # refuse to restart if the latest drizzle/*.sql file
301 # isn't in it.
302 notify_step "db-migrate" "in_progress"
303 DM_START=$(date +%s)
304 set -a; source /etc/gluecron.env; set +a
305
306 echo "==> running migrations (must succeed to proceed)"
307 if ! "$BUN" run src/db/migrate.ts; then
308 notify_step "db-migrate" "failed" "$(( ( $(date +%s) - DM_START ) * 1000 ))"
309 echo "ERROR: bun run db:migrate failed — aborting deploy" >&2
310 exit 1
311 fi
312
313 # Verify the LATEST drizzle/*.sql is present in _migrations.
314 # This catches the case where migrate.ts thought every file
315 # was already applied but the actual list on disk has new
316 # entries the runner somehow skipped. We write a tiny ESM
317 # verifier into /tmp and call it; this avoids the require()
318 # vs ESM mismatch you'd hit using `bun -e` on a "type":"module"
319 # package.
320 echo "==> verifying latest drizzle/*.sql is recorded in _migrations"
321 LATEST_FILE=$(ls drizzle/*.sql 2>/dev/null | sort | tail -1 | xargs -n1 basename || true)
322 if [ -z "$LATEST_FILE" ]; then
323 echo "WARN: no drizzle/*.sql files found — skipping verification"
324 else
325 cat > /tmp/verify-migration.mjs <<'VERIFY_EOF'
326 import { neon } from "@neondatabase/serverless";
327 import postgres from "postgres";
328 const url = process.env.DATABASE_URL;
329 const target = process.argv[2];
330 if (!url || !target) { console.error("verify: missing DATABASE_URL or target"); process.exit(1); }
331 const isNeon = /(^|\.)neon\.tech$/i.test(new URL(url).hostname);
332 try {
333 let rows;
334 if (isNeon) {
335 const sql = neon(url);
336 rows = await sql(`SELECT name FROM _migrations WHERE name = $1`, [target]);
337 } else {
338 const client = postgres(url, { max: 1, prepare: false });
339 rows = await client`SELECT name FROM _migrations WHERE name = ${target}`;
340 await client.end({ timeout: 5 });
341 }
342 if (rows.length === 0) {
343 console.error(`verify: ${target} is NOT in _migrations`);
344 process.exit(2);
345 }
346 console.log(`verify: ${target} is applied`);
347 } catch (e) {
348 console.error("verify: query failed:", e?.message ?? e);
349 process.exit(3);
350 }
351 VERIFY_EOF
352 if ! "$BUN" run /tmp/verify-migration.mjs "$LATEST_FILE"; then
353 echo "ERROR: latest migration $LATEST_FILE is NOT recorded in _migrations — aborting deploy" >&2
354 notify_step "db-migrate" "failed" "$(( ( $(date +%s) - DM_START ) * 1000 ))"
355 exit 1
356 fi
357 fi
358
359 notify_step "db-migrate" "succeeded" "$(( ( $(date +%s) - DM_START ) * 1000 ))"
179 "$BUN" install --frozen-lockfile
360180
361 # ─── (c.5) Pin BUILD_SHA into systemd before restart.
362 # Without this, src/routes/pwa.ts falls back to a stable
363 # "dev-stable" version string and the browser service worker
364 # never invalidates between real deploys. The drop-in survives
365 # daemon-reload; old contents are overwritten on every deploy.
181 echo "=== STEP 3: pin BUILD_SHA in systemd drop-in ==="
366182 mkdir -p /etc/systemd/system/gluecron.service.d
367183 cat > /etc/systemd/system/gluecron.service.d/build-sha.conf <<EOF
368184 [Service]
@@ -370,15 +186,10 @@ jobs:
370186 Environment="BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
371187 EOF
372188 systemctl daemon-reload
373 echo "==> pinned BUILD_SHA=${new_sha:0:12} in systemd drop-in"
374189
375 # ─── (d) Zero-downtime restart. Blocks until sd_notify(READY=1).
376 notify_step "restart-service" "in_progress"
377 RS_START=$(date +%s)
378 echo "==> systemctl restart gluecron (blocks on sd_notify READY=1)"
190 echo "=== STEP 4: restart gluecron systemd unit ==="
379191 systemctl restart gluecron
380 echo "==> restart returned — gluecron signalled ready"
381 notify_step "restart-service" "succeeded" "$(( ( $(date +%s) - RS_START ) * 1000 ))"
192 echo "=== DONE: service restarted with $new_sha ==="
382193
383194 # ─── 3. Smoke-test the deployed app on the box ──────────────────────
384195 # We SSH back in and curl localhost:3010/healthz directly. This tests
385196