Commit90cc53d
docs(cutover): step-by-step VPS runbook to drop the GitHub mirror
docs(cutover): step-by-step VPS runbook to drop the GitHub mirror Owner-executed: backup-first sequencing (offsite rclone + verified restore drill), confirm self-deploy is cord-free, push/deploy/rollback drill, then freeze-then-delete GitHub after 7 green days. Defers the creds-gated live verification (MCP matrix, journey, GateTest, import) to a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 file changed+185−090cc53deeac8a9f7801b969b8824ef78329a76cd
1 changed file+185−0
Addeddocs/CUTOVER_RUNBOOK.md+185−0View fileUnifiedSplit
@@ -0,0 +1,185 @@
1# Cutover Runbook — go fully self-sufficient, drop the GitHub mirror
2
3**Goal:** Gluecron hosts, deploys, and backs up itself. GitHub is removed as
4infrastructure. Owner-executed; each step is copy-pasteable on the VPS
5(`66.42.121.161`, working tree `/opt/gluecron`).
6
7**Sequencing rule:** durability (backups) goes in BEFORE anything is removed.
8Do the phases in order. Do not delete the GitHub repo until Phase 4.
9
10The code side is already merged (self-canonical bootstrap, offsite backup +
11restore scripts, docs). What remains is operator config on the box.
12
13---
14
15## Phase 0 — Prerequisites (once)
16
17```bash
18# On the VPS, as the deploy user.
19cd /opt/gluecron
20
21# You need these on PATH:
22which git rclone pg_dump tar curl # install missing: apt-get install -y postgresql-client rclone
23
24# Confirm the live env file exists and holds DATABASE_URL (Neon).
25sudo test -f /etc/gluecron.env && echo "env file present"
26```
27
28Mint a personal access token for pushing to the canonical remote (admin scope).
29Run this OUTSIDE an AI session (it mints live admin creds):
30
31```bash
32EMERGENCY_PAT_USER=ccantynz bun run scripts/emergency-pat.ts
33# copy the glc_... token it prints
34```
35
36---
37
38## Phase 1 — Offsite backup FIRST (the safety net that replaces GitHub)
39
401. Create a bucket (Cloudflare R2 or S3) and configure an rclone remote:
41
42```bash
43rclone config # create a remote, e.g. named "r2", pointing at your bucket
44rclone lsd r2: # verify it lists
45```
46
472. Add backup env to `/etc/gluecron.env`:
48
49```bash
50sudo tee -a /etc/gluecron.env >/dev/null <<'ENV'
51BACKUP_RCLONE_REMOTE=r2:gluecron-backups
52RETAIN_DAYS=14
53# optional dead-man's-switch (create a check at healthchecks.io):
54# HEALTHCHECK_PING_URL=https://hc-ping.com/<uuid>
55ENV
56```
57
583. Run one backup by hand and confirm both artifacts land offsite:
59
60```bash
61set -a; . /etc/gluecron.env; set +a
62bash scripts/backup-offsite.sh
63rclone ls r2:gluecron-backups/repos/ # should show a repos-<ts>.tar.gz
64rclone ls r2:gluecron-backups/db/ # should show a db-<ts>.dump
65```
66
674. **Prove the restore works** (this is what makes it a backup, not a hope):
68
69```bash
70# Pull the newest repos snapshot back and extract to a scratch dir.
71rclone copy r2:gluecron-backups/repos/ /tmp/rb/ --max-age 1d
72bash scripts/restore.sh repos /tmp/rb/repos-<ts>.tar.gz --dest /tmp/verify
73git -C /tmp/verify/repos/ccantynz/Gluecron.com.git log -1 # shows the latest commit
74```
75
765. Install the daily timer:
77
78```bash
79sudo tee /etc/systemd/system/gluecron-backup.service >/dev/null <<'UNIT'
80[Unit]
81Description=Gluecron offsite backup
82[Service]
83Type=oneshot
84EnvironmentFile=/etc/gluecron.env
85WorkingDirectory=/opt/gluecron
86ExecStart=/usr/bin/env bash scripts/backup-offsite.sh
87UNIT
88sudo tee /etc/systemd/system/gluecron-backup.timer >/dev/null <<'UNIT'
89[Unit]
90Description=Daily Gluecron offsite backup
91[Timer]
92OnCalendar=*-*-* 03:30:00
93Persistent=true
94[Install]
95WantedBy=timers.target
96UNIT
97sudo systemctl daemon-reload
98sudo systemctl enable --now gluecron-backup.timer
99systemctl list-timers gluecron-backup.timer
100```
101
102**Do not proceed to Phase 3 until at least one restore drill has passed.**
103
104---
105
106## Phase 2 — Confirm self-deploy is truly cord-free
107
108The one fact that can't be checked off-box: the deploy working-tree must pull
109from the **local canonical repo**, not GitHub.
110
111```bash
112git -C /opt/gluecron remote -v
113# origin should be the LOCAL bare repo (…/repos/ccantynz/Gluecron.com.git) or
114# https://gluecron.com/ccantynz/Gluecron.com.git — NOT github.com.
115# If it points at GitHub, repoint it:
116# git -C /opt/gluecron remote set-url origin /opt/gluecron/repos/ccantynz/Gluecron.com.git
117
118# Ensure the self-deploy trigger is armed:
119grep -q '^SELF_HOST_REPO=' /etc/gluecron.env || \
120 echo 'SELF_HOST_REPO=ccantynz/Gluecron.com' | sudo tee -a /etc/gluecron.env
121```
122
123Repoint your **local dev clone** at the canonical remote too:
124
125```bash
126# On your workstation (this repo):
127git remote set-url origin https://x:<glc_token>@gluecron.com/ccantynz/Gluecron.com.git
128# or keep origin and add a dedicated remote:
129# git remote add gluecron https://x:<glc_token>@gluecron.com/ccantynz/Gluecron.com.git
130```
131
132---
133
134## Phase 3 — Deploy drill (push → deploy → rollback)
135
136```bash
137# 1. Push the already-merged self-sufficiency commits to the canonical remote.
138git push origin main # (or: git push gluecron main)
139
140# 2. Watch it deploy (~25s) — live step stream:
141# https://gluecron.com/admin/deploys
142# tail -f /var/log/gluecron-self-deploy.log
143
144# 3. Rollback drill: deliberately break a deploy (e.g. push a commit that fails
145# the smoke test on a throwaway branch flow) and confirm self-deploy.sh
146# auto-reverts via `git reset --hard $PREV_SHA` + restart, and healthz recovers.
147```
148
149Green criteria: push deploys in ~25s, `/healthz` returns 200, smoke passes,
150and a forced-fail auto-rolls-back to the previous SHA.
151
152---
153
154## Phase 4 — Retire GitHub (only after 7 green days)
155
156Run the platform through Phases 1–3 and let it self-deploy for **7 days** with
157no manual intervention. Then:
158
1591. **Freeze** the GitHub repo `ccantynz-alt/Gluecron.com`: Settings → Archive
160 this repository (makes it read-only). Keep it as a cold archive for 30 days.
1612. Neuter the old CI so a stray GitHub push can't deploy: change
162 `.github/workflows/hetzner-deploy.yml` trigger to `on: workflow_dispatch:`
163 only (do this via a Gluecron push, since GitHub is no longer canonical).
1643. After **30 green days**: delete `ccantynz-alt/Gluecron.com` on GitHub and
165 delete `.github/workflows/hetzner-deploy.yml` from the repo.
166
167---
168
169## Rollback (if the cutover stumbles in week one)
170
171The GitHub repo is still frozen-not-deleted until Phase 4 completes, so it
172remains a last-resort source. But the primary recovery path is now the offsite
173backup: `scripts/restore.sh repos <snapshot>` reconstructs every bare repo, and
174`scripts/restore.sh db <dump> --target-url <db>` restores metadata. Test both in
175Phase 1 so you trust them before you need them.
176
177---
178
179## Post-cutover verification (needs a running instance + PAT — do in a follow-up)
180
181These were deferred from the audit because they need live creds:
182- 60/60 MCP tool matrix against `https://gluecron.com/mcp`.
183- Playwright journey: register → repo → push → issue → PR → merge → webhook.
184- GateTest loop: push → `notifyGateTestOfPush` → results in `gate_runs`.
185- Real GitHub-repo import via `/import` end-to-end (the blessed onboarding flow).
0186