Blame · Line-by-line history

`CUTOVER_RUNBOOK.md`

Each line is annotated with the commit that last touched it. Click any SHA to jump to that commit and see the surrounding change.

CUTOVER_RUNBOOK.mdBlame185 lines · 1 contributor

Normal view Raw

90cc53dccanty labs	1	# Cutover Runbook — go fully self-sufficient, drop the GitHub mirror
	2
	3	Goal: Gluecron hosts, deploys, and backs up itself. GitHub is removed as
	4	infrastructure. Owner-executed; each step is copy-pasteable on the VPS
	5	(`66.42.121.161`, working tree `/opt/gluecron`).
	6
	7	Sequencing rule: durability (backups) goes in BEFORE anything is removed.
	8	Do the phases in order. Do not delete the GitHub repo until Phase 4.
	9
	10	The code side is already merged (self-canonical bootstrap, offsite backup +
	11	restore scripts, docs). What remains is operator config on the box.
	12
	13	---
	14
	15	## Phase 0 — Prerequisites (once)
	16
	17	```bash
	18	# On the VPS, as the deploy user.
	19	cd /opt/gluecron
	20
	21	# You need these on PATH:
	22	which git rclone pg_dump tar curl # install missing: apt-get install -y postgresql-client rclone
	23
	24	# Confirm the live env file exists and holds DATABASE_URL (Neon).
	25	sudo test -f /etc/gluecron.env && echo "env file present"
	26	```
	27
	28	Mint a personal access token for pushing to the canonical remote (admin scope).
	29	Run this OUTSIDE an AI session (it mints live admin creds):
	30
	31	```bash
	32	EMERGENCY_PAT_USER=ccantynz bun run scripts/emergency-pat.ts
	33	# copy the glc_... token it prints
	34	```
	35
	36	---
	37
	38	## Phase 1 — Offsite backup FIRST (the safety net that replaces GitHub)
	39
	40	1. Create a bucket (Cloudflare R2 or S3) and configure an rclone remote:
	41
	42	```bash
	43	rclone config # create a remote, e.g. named "r2", pointing at your bucket
	44	rclone lsd r2: # verify it lists
	45	```
	46
	47	2. Add backup env to `/etc/gluecron.env`:
	48
	49	```bash
	50	sudo tee -a /etc/gluecron.env >/dev/null <<'ENV'
	51	BACKUP_RCLONE_REMOTE=r2:gluecron-backups
	52	RETAIN_DAYS=14
	53	# optional dead-man's-switch (create a check at healthchecks.io):
	54	# HEALTHCHECK_PING_URL=https://hc-ping.com/<uuid>
	55	ENV
	56	```
	57
	58	3. Run one backup by hand and confirm both artifacts land offsite:
	59
	60	```bash
	61	set -a; . /etc/gluecron.env; set +a
	62	bash scripts/backup-offsite.sh
	63	rclone ls r2:gluecron-backups/repos/ # should show a repos-<ts>.tar.gz
	64	rclone ls r2:gluecron-backups/db/ # should show a db-<ts>.dump
	65	```
	66
	67	4. Prove the restore works (this is what makes it a backup, not a hope):
	68
	69	```bash
	70	# Pull the newest repos snapshot back and extract to a scratch dir.
	71	rclone copy r2:gluecron-backups/repos/ /tmp/rb/ --max-age 1d
	72	bash scripts/restore.sh repos /tmp/rb/repos-<ts>.tar.gz --dest /tmp/verify
	73	git -C /tmp/verify/repos/ccantynz/Gluecron.com.git log -1 # shows the latest commit
	74	```
	75
	76	5. Install the daily timer:
	77
	78	```bash
	79	sudo tee /etc/systemd/system/gluecron-backup.service >/dev/null <<'UNIT'
	80	[Unit]
	81	Description=Gluecron offsite backup
	82	[Service]
	83	Type=oneshot
	84	EnvironmentFile=/etc/gluecron.env
	85	WorkingDirectory=/opt/gluecron
	86	ExecStart=/usr/bin/env bash scripts/backup-offsite.sh
	87	UNIT
	88	sudo tee /etc/systemd/system/gluecron-backup.timer >/dev/null <<'UNIT'
	89	[Unit]
	90	Description=Daily Gluecron offsite backup
	91	[Timer]
	92	OnCalendar=--* 03:30:00
	93	Persistent=true
	94	[Install]
	95	WantedBy=timers.target
	96	UNIT
	97	sudo systemctl daemon-reload
	98	sudo systemctl enable --now gluecron-backup.timer
	99	systemctl list-timers gluecron-backup.timer
	100	```
	101
	102	Do not proceed to Phase 3 until at least one restore drill has passed.
	103
	104	---
	105
	106	## Phase 2 — Confirm self-deploy is truly cord-free
	107
	108	The one fact that can't be checked off-box: the deploy working-tree must pull
	109	from the local canonical repo, not GitHub.
	110
	111	```bash
	112	git -C /opt/gluecron remote -v
	113	# origin should be the LOCAL bare repo (…/repos/ccantynz/Gluecron.com.git) or
	114	# https://gluecron.com/ccantynz/Gluecron.com.git — NOT github.com.
	115	# If it points at GitHub, repoint it:
	116	# git -C /opt/gluecron remote set-url origin /opt/gluecron/repos/ccantynz/Gluecron.com.git
	117
	118	# Ensure the self-deploy trigger is armed:
	119	grep -q '^SELF_HOST_REPO=' /etc/gluecron.env \|\| \
	120	echo 'SELF_HOST_REPO=ccantynz/Gluecron.com' \| sudo tee -a /etc/gluecron.env
	121	```
	122
	123	Repoint your local dev clone at the canonical remote too:
	124
	125	```bash
	126	# On your workstation (this repo):
	127	git remote set-url origin https://x:<glc_token>@gluecron.com/ccantynz/Gluecron.com.git
	128	# or keep origin and add a dedicated remote:
	129	# git remote add gluecron https://x:<glc_token>@gluecron.com/ccantynz/Gluecron.com.git
	130	```
	131
	132	---
	133
	134	## Phase 3 — Deploy drill (push → deploy → rollback)
	135
	136	```bash
	137	# 1. Push the already-merged self-sufficiency commits to the canonical remote.
	138	git push origin main # (or: git push gluecron main)
	139
	140	# 2. Watch it deploy (~25s) — live step stream:
	141	# https://gluecron.com/admin/deploys
	142	# tail -f /var/log/gluecron-self-deploy.log
	143
	144	# 3. Rollback drill: deliberately break a deploy (e.g. push a commit that fails
	145	# the smoke test on a throwaway branch flow) and confirm self-deploy.sh
	146	# auto-reverts via `git reset --hard $PREV_SHA` + restart, and healthz recovers.
	147	```
	148
	149	Green criteria: push deploys in ~25s, `/healthz` returns 200, smoke passes,
	150	and a forced-fail auto-rolls-back to the previous SHA.
	151
	152	---
	153
	154	## Phase 4 — Retire GitHub (only after 7 green days)
	155
	156	Run the platform through Phases 1–3 and let it self-deploy for 7 days with
	157	no manual intervention. Then:
	158
	159	1. Freeze the GitHub repo `ccantynz-alt/Gluecron.com`: Settings → Archive
	160	this repository (makes it read-only). Keep it as a cold archive for 30 days.
	161	2. Neuter the old CI so a stray GitHub push can't deploy: change
	162	`.github/workflows/hetzner-deploy.yml` trigger to `on: workflow_dispatch:`
	163	only (do this via a Gluecron push, since GitHub is no longer canonical).
	164	3. After 30 green days: delete `ccantynz-alt/Gluecron.com` on GitHub and
	165	delete `.github/workflows/hetzner-deploy.yml` from the repo.
	166
	167	---
	168
	169	## Rollback (if the cutover stumbles in week one)
	170
	171	The GitHub repo is still frozen-not-deleted until Phase 4 completes, so it
	172	remains a last-resort source. But the primary recovery path is now the offsite
	173	backup: `scripts/restore.sh repos <snapshot>` reconstructs every bare repo, and
	174	`scripts/restore.sh db <dump> --target-url <db>` restores metadata. Test both in
	175	Phase 1 so you trust them before you need them.
	176
	177	---
	178
	179	## Post-cutover verification (needs a running instance + PAT — do in a follow-up)
	180
	181	These were deferred from the audit because they need live creds:
	182	- 60/60 MCP tool matrix against `https://gluecron.com/mcp`.
	183	- Playwright journey: register → repo → push → issue → PR → merge → webhook.
	184	- GateTest loop: push → `notifyGateTestOfPush` → results in `gate_runs`.
	185	- Real GitHub-repo import via `/import` end-to-end (the blessed onboarding flow).