Commit6d2b36e

feat(ops): metal-box deploy checklist + verify script + alert rules

feat(ops): metal-box deploy checklist + verify script + alert rules

Three gluecron polish items from LAUNCH_TODAY.md's open list, landed
together so they can drop in on the live box the moment deploy finishes:

- DEPLOY_CHECKLIST.md — rewritten metal-box-first. Old file was
  Fly-centric ("run flyctl deploy"); new version is the actual path
  we're taking. Fly path preserved as a fallback reference in DEPLOY.md.

- scripts/verify-deploy.sh — runs the post-deploy smoke section of
  DEPLOY_CHECKLIST as a single command. /healthz, /readyz, /status,
  /status.svg, /metrics, and www-alias in one shot. Non-zero exit on
  any failure so it can drop into CI or a post-deploy hook.

- infra/alerts/gluecron.rules.yml + README — Prometheus-format alert
  rules covering availability (GluecronDown/NotReady), errors
  (High5xxRate/Spike5xx/ErrorSinkFailing), latency (HighLatencyP95),
  autopilot heartbeat (AutopilotStalled), and resources (HighMemory/
  DiskFillingFast). Provider-neutral — works with Prom+AM, Mimir,
  VictoriaMetrics, Thanos. README documents wire-up and severity
  routing.

Closes the 🟡 "alerting rules are not [wired]" and
🟡 "/metrics shipping to Grafana/Datadog/Prometheus — pipe not wired"
items from LAUNCH_TODAY.md.

Dictation App committed on April 22, 2026Parent: 5843260

4 files changed+367−206d2b36e49b882096eed826da02c108abc93127fd

4 changed files+367−20

ModifiedDEPLOY_CHECKLIST.md+81−20View fileUnified Split

@@ -1,45 +1,106 @@

11# Deploy checklist

3Scannable go-live runbook. For detailed rationale + per-variable docs, see `DEPLOY.md`.

3Scannable go-live runbook for the metal-box deploy at `45.76.171.37`.

4For the compose + Caddy setup, see `DEPLOY_METAL.md`. For the (deferred)

5Fly.io path, see `DEPLOY.md`.

57## Pre-deploy

79- [ ] `DATABASE_URL` set to a live Neon PostgreSQL connection string (hard-required)

8- [ ] `GIT_REPOS_PATH` points at a persistent volume (Fly: `/app/repos` via `gluecron_repos`)

9- [ ] Secrets reviewed in `.env.example`; optional keys (`ANTHROPIC_API_KEY`, `RESEND_API_KEY`, `GATETEST_API_KEY`, `VOYAGE_API_KEY`) set or consciously skipped

10- [ ] `ERROR_WEBHOOK_URL` **or** `SENTRY_DSN` set as a Fly secret (without this, errors log to stderr only)

10- [ ] `.env` populated on the box from `.env.example`; optional keys

11 (`ANTHROPIC_API_KEY`, `RESEND_API_KEY`, `GATETEST_API_KEY`, `VOYAGE_API_KEY`)

12 set or consciously skipped

13- [ ] `ERROR_WEBHOOK_URL` **or** `SENTRY_DSN` set (without this, errors log to stderr only)

1114- [ ] `AUTOPILOT_DISABLED` decision made (default: enabled)

1215- [ ] `DEMO_SEED_ON_BOOT` decision made (default: off)

13- [ ] Run `bun run preflight` locally — green before shipping

14- [ ] `bun test` clean on the deploy commit

16- [ ] `APP_BASE_URL=https://gluecron.com`

17- [ ] Local: `bun run preflight` green

18- [ ] Local: `bun test` clean on the deploy commit

1519- [ ] `CHANGELOG.md` has an `[Unreleased]` entry for this deploy

20- [ ] Cloudflare DNS confirmed: `gluecron.com` + `www.gluecron.com`

21 A-records point at `45.76.171.37`, proxy status "DNS only"

1622

17## Deploy

23## Deploy (metal box, `45.76.171.37`)

1824

19- [ ] `flyctl deploy` from the repo root

20- [ ] Release command runs `bun run db:migrate` automatically (configured in `fly.toml`); confirm it succeeded in the release logs

21- [ ] Fly machine reaches healthy state; no boot-loop

22- [ ] Persistent volume mounted and writable at the configured `GIT_REPOS_PATH`

25```sh

26ssh root@45.76.171.37

27git clone https://github.com/ccantynz-alt/Gluecron.com.git /opt/gluecron # first time

28cd /opt/gluecron && git pull

29cp .env.example .env && vim .env # first time only

30docker compose up -d --build

31docker compose logs -f caddy # watch for cert issuance

32```

34- [ ] Repo cloned / updated at `/opt/gluecron`

35- [ ] `.env` present and populated

36- [ ] `docker compose up -d --build` exits clean

37- [ ] Caddy logs show `certificate obtained successfully` for

38 `gluecron.com` AND `www.gluecron.com`

39- [ ] `docker compose ps` shows both services `Up (healthy)`

41## First-run DB migration

43```sh

44docker compose exec gluecron bun run db:migrate

45```

47- [ ] Exits with code 0

48- [ ] No "column already exists" / "missing relation" warnings

2349

2450## Post-deploy smoke

2551

26- [ ] `GET /healthz` → 200

27- [ ] `GET /readyz` → 200 (DB + git reachable)

28- [ ] `GET /status` renders; `GET /status.svg` returns a shields badge

52Run the one-command verifier:

54```sh

55bash scripts/verify-deploy.sh https://gluecron.com

56```

58Or tick manually:

60- [ ] `GET https://gluecron.com/healthz` → 200 `ok`

61- [ ] `GET https://gluecron.com/readyz` → 200 (DB + git reachable)

62- [ ] `GET https://gluecron.com/status` renders the HTML status page

63- [ ] `GET https://gluecron.com/status.svg` returns a shields-style badge

64- [ ] `GET https://gluecron.com/metrics` returns Prometheus-format metrics

65- [ ] `GET https://www.gluecron.com/healthz` → 200 (no cert error)

2966- [ ] Register a new user via `/register`

30- [ ] First user auto-promoted to site admin (check `/admin`)

67- [ ] First user auto-promoted to site admin (`/admin` loads for that account)

3168- [ ] Create a repo via `/new`

32- [ ] `git clone` over HTTPS succeeds

69- [ ] `git clone` over HTTPS succeeds against the new repo

3370- [ ] `git push` succeeds; post-receive pipeline runs (GateTest callback + webhooks)

3471- [ ] AI review path exercised if `ANTHROPIC_API_KEY` is set

35- [ ] Sentry/webhook sink receives a forced test error

72- [ ] Sentry / error-webhook sink receives a forced test error

3673

3774## First-day operations

3875

39- [ ] Admin bootstrap: oldest row in `users` is the intended admin — register that account **first** (see `src/routes/admin.ts` bootstrap rule)

76- [ ] Admin bootstrap: oldest row in `users` is the intended admin —

77 register that account **first** (see `src/routes/admin.ts` bootstrap rule)

4078- [ ] Site banner / motd configured in `/admin` if needed for launch

4179- [ ] Billing plans seeded (free/pro/team/enterprise) — verify in `/admin`

42- [ ] Autopilot ticker heartbeat visible (unless `AUTOPILOT_DISABLED=1`)

80- [ ] Autopilot ticker heartbeat visible in logs (unless `AUTOPILOT_DISABLED=1`)

4381- [ ] `docs/LAUNCH_ANNOUNCEMENT.md` queued for Show HN / social

44- [ ] Add a dated entry to `CHANGELOG.md` and tag a release

82- [ ] Point Alertmanager at `infra/alerts/gluecron.rules.yml` (see `infra/alerts/README.md`)

83- [ ] Dated entry added to `CHANGELOG.md`; tag a release

4584- [ ] Monitor `/metrics`, `/healthz`, and the error sink for the first hour

86## Redeploy

88```sh

89ssh root@45.76.171.37

90cd /opt/gluecron && git pull && docker compose up -d --build

91docker compose exec gluecron bun run db:migrate # if migrations changed

92bash scripts/verify-deploy.sh https://gluecron.com

93```

95## Rollback

97```sh

98ssh root@45.76.171.37

99cd /opt/gluecron

100git log --oneline -10 # find last known-good sha

101git checkout <prev-sha>

102docker compose up -d --build

103```

104

105DNS is unchanged, so rollback = ~60s of rebuild time with no

106Cloudflare/registrar changes required.

Addedinfra/alerts/README.md+85−0View fileUnified Split

@@ -0,0 +1,85 @@

1# Alerting rules

3Prometheus-format alert rules for gluecron. Provider-neutral — works

4with Prometheus + Alertmanager, Grafana Mimir + Grafana OnCall,

5VictoriaMetrics + vmalert, or Thanos Ruler.

7## File

9- `gluecron.rules.yml` — five groups: availability, errors, latency,

10 autopilot, resources. Three severity levels: `critical`, `warning`, `info`.

12## Wire-up

14### Prometheus + Alertmanager

161. Scrape gluecron:

17 ```yaml

18 # prometheus.yml

19 scrape_configs:

20 - job_name: gluecron

21 metrics_path: /metrics

22 static_configs:

23 - targets: ['gluecron.com:443']

24 scheme: https

25 ```

262. Include the rules file:

27 ```yaml

28 # prometheus.yml

29 rule_files:

30 - /etc/prometheus/gluecron.rules.yml

31 ```

323. Point Alertmanager at Slack / PagerDuty / email. Route `severity=critical`

33 to the pager channel, `severity=warning` to Slack-warning.

35### Blackbox probes for /readyz

37One rule (`GluecronNotReady`) assumes you also run the Prometheus

38[blackbox_exporter](https://github.com/prometheus/blackbox_exporter) probing

39`https://gluecron.com/readyz`. If you skip blackbox, that rule will never

40fire — safe, just redundant with `GluecronDown`.

42Minimal blackbox scrape:

43```yaml

44- job_name: gluecron-blackbox

45 metrics_path: /probe

46 params:

47 module: [http_2xx]

48 static_configs:

49 - targets:

50 - https://gluecron.com/readyz

51 relabel_configs:

52 - source_labels: [__address__]

53 target_label: __param_target

54 - source_labels: [__param_target]

55 target_label: instance

56 - target_label: __address__

57 replacement: blackbox-exporter:9115

58```

60## Severity routing

62| Severity | Where it goes | Examples |

63|------------|------------------------|-----------------------------------------|

64| `critical` | page (SMS / phone) | `GluecronDown`, `GluecronSpike5xx` |

65| `warning` | Slack / email | `GluecronHigh5xxRate`, `GluecronHighLatencyP95` |

66| `info` | dashboard only | (reserved; none yet) |

68## Tuning

70- `GluecronHigh5xxRate` fires at >5% for 10m. Tighten once baseline is known.

71- `GluecronHighLatencyP95` fires at >2s for 15m. Tune after the first

72 24 hours of real traffic.

73- `GluecronAutopilotStalled` expects `autopilot_last_run_timestamp_seconds`

74 metric. If the autopilot ticker doesn't emit this gauge yet, either

75 add it to `src/lib/autopilot.ts` (one `gauge.set(Date.now()/1000)`

76 per tick) or drop this rule until the metric exists.

78## Validation

80Before rolling out:

81```sh

82promtool check rules infra/alerts/gluecron.rules.yml

83```

85No output = valid.

Addedinfra/alerts/gluecron.rules.yml+133−0View fileUnified Split

@@ -0,0 +1,133 @@

1# Prometheus alerting rules for gluecron.com.

3# Drop-in for any Prometheus-compatible system: Prometheus itself,

4# Grafana Mimir, VictoriaMetrics, Thanos. Point Alertmanager at the

5# rules file or include it via `rule_files:` in prometheus.yml.

7# Scrape target assumed: /metrics on the gluecron container (Prom format).

8# Label `job="gluecron"` assumed — edit if your scrape config uses a

9# different job name.

10#

11# Severity convention:

12# critical = page (SMS / phone / Slack-critical channel)

13# warning = notify (Slack-warning channel, email)

14# info = dashboard only

16groups:

17 - name: gluecron-availability

18 interval: 30s

19 rules:

20 - alert: GluecronDown

21 expr: up{job="gluecron"} == 0

22 for: 2m

23 labels:

24 severity: critical

25 service: gluecron

26 annotations:

27 summary: "gluecron.com is down"

28 description: "Prometheus cannot scrape /metrics on {{ $labels.instance }}. Likely the container exited or Caddy can't reach it. Check `docker compose ps` and `docker compose logs gluecron` on 45.76.171.37."

29 runbook: "https://github.com/ccantynz-alt/Gluecron.com/blob/main/DEPLOY_CHECKLIST.md#rollback"

31 - alert: GluecronNotReady

32 expr: probe_success{instance=~".*/readyz"} == 0

33 for: 5m

34 labels:

35 severity: critical

36 service: gluecron

37 annotations:

38 summary: "gluecron /readyz failing for 5m"

39 description: "App is running but /readyz is returning non-200. Usually means the DB (Neon) is unreachable or the git repos volume is unmounted. Check `docker compose exec gluecron bun run src/scripts/db-check.ts` or equivalent."

41 - name: gluecron-errors

42 interval: 30s

43 rules:

44 - alert: GluecronHigh5xxRate

45 expr: |

46 (

47 sum(rate(http_requests_total{job="gluecron",status=~"5.."}[5m]))

48 /

49 sum(rate(http_requests_total{job="gluecron"}[5m]))

50 ) > 0.05

51 for: 10m

52 labels:

53 severity: warning

54 service: gluecron

55 annotations:

56 summary: ">5% 5xx rate over 10m"

57 description: "Current 5xx rate is {{ $value | humanizePercentage }}. Look at the error-webhook/Sentry sink and recent deploys."

59 - alert: GluecronSpike5xx

60 expr: |

61 sum(rate(http_requests_total{job="gluecron",status=~"5.."}[1m])) > 1

62 for: 2m

63 labels:

64 severity: critical

65 service: gluecron

66 annotations:

67 summary: "5xx spike (>1/s for 2m)"

68 description: "Absolute 5xx rate exceeded 1/s. A post-deploy regression or downstream outage."

70 - alert: GluecronErrorSinkFailing

71 expr: increase(error_sink_failures_total{job="gluecron"}[15m]) > 0

72 for: 15m

73 labels:

74 severity: warning

75 service: gluecron

76 annotations:

77 summary: "Error webhook / Sentry sink failing"

78 description: "Errors are being generated but the sink (ERROR_WEBHOOK_URL / SENTRY_DSN) is rejecting them. Check the credentials and the sink's own status page."

80 - name: gluecron-latency

81 interval: 30s

82 rules:

83 - alert: GluecronHighLatencyP95

84 expr: |

85 histogram_quantile(0.95,

86 sum by (le) (rate(http_request_duration_seconds_bucket{job="gluecron"}[5m]))

87 ) > 2

88 for: 15m

89 labels:

90 severity: warning

91 service: gluecron

92 annotations:

93 summary: "p95 latency >2s for 15m"

94 description: "Current p95 is {{ $value }}s. Check DB connection pool, Neon status, and recent slow queries."

96 - name: gluecron-autopilot

97 interval: 1m

98 rules:

99 - alert: GluecronAutopilotStalled

100 expr: time() - autopilot_last_run_timestamp_seconds{job="gluecron"} > 900

101 for: 5m

102 labels:

103 severity: warning

104 service: gluecron

105 annotations:

106 summary: "Autopilot ticker stalled >15m"

107 description: "Autopilot runs every 5m; last tick was {{ $value | humanizeDuration }} ago. If AUTOPILOT_DISABLED=1 this alert should be muted at the receiver. Otherwise investigate the ticker in `src/lib/autopilot.ts`."

108

109 - name: gluecron-resources

110 interval: 30s

111 rules:

112 - alert: GluecronHighMemory

113 expr: |

114 (container_memory_usage_bytes{name=~".*gluecron.*"}

115 / container_spec_memory_limit_bytes{name=~".*gluecron.*"}) > 0.9

116 for: 15m

117 labels:

118 severity: warning

119 service: gluecron

120 annotations:

121 summary: "Container memory >90% of limit"

122 description: "Near the OOM cliff. Either raise the compose memory limit or investigate a leak."

123

124 - alert: GluecronDiskFillingFast

125 expr: |

126 predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 24*3600) < 0

127 for: 30m

128 labels:

129 severity: warning

130 service: gluecron

131 annotations:

132 summary: "Disk will fill within 24h at current rate"

133 description: "Usually the git repos volume growing unchecked. Check `/data/repos` size and retention policy."

Addedscripts/verify-deploy.sh+68−0View fileUnified Split

@@ -0,0 +1,68 @@

1#!/usr/bin/env bash

2# Post-deploy smoke verifier. Runs the DEPLOY_CHECKLIST.md post-deploy

3# smoke tests as a single command. Exits non-zero on any failure so it

4# can be dropped into CI or a post-deploy hook.

6# Usage:

7# bash scripts/verify-deploy.sh # defaults to https://gluecron.com

8# bash scripts/verify-deploy.sh https://staging.example # override base URL

9# BASE_URL=https://gluecron.com bash scripts/verify-deploy.sh

11set -uo pipefail

13BASE_URL="${1:-${BASE_URL:-https://gluecron.com}}"

14BASE_URL="${BASE_URL%/}"

16fail=0

18check() {

19 local label="$1"

20 local url="$2"

21 local expect_status="${3:-200}"

22 local match="${4:-}"

24 local tmp status body

25 tmp="$(mktemp)"

26 status="$(curl -sS -o "$tmp" -w '%{http_code}' --max-time 10 "$url" || echo 000)"

27 body="$(cat "$tmp")"

28 rm -f "$tmp"

30 if [[ "$status" != "$expect_status" ]]; then

31 printf ' \033[31mFAIL\033[0m %-32s %s (got %s, want %s)\n' \

32 "$label" "$url" "$status" "$expect_status"

33 fail=$((fail + 1))

34 return

35 fi

37 if [[ -n "$match" && "$body" != *"$match"* ]]; then

38 printf ' \033[31mFAIL\033[0m %-32s %s (body missing %q)\n' \

39 "$label" "$url" "$match"

40 fail=$((fail + 1))

41 return

42 fi

44 printf ' \033[32mOK\033[0m %-32s %s (%s)\n' "$label" "$url" "$status"

45}

47echo "Verifying $BASE_URL ..."

49check "healthz" "$BASE_URL/healthz" 200

50check "readyz" "$BASE_URL/readyz" 200

51check "status (html)" "$BASE_URL/status" 200 "status"

52check "status.svg" "$BASE_URL/status.svg" 200 "<svg"

53check "metrics" "$BASE_URL/metrics" 200 "# HELP"

54check "landing (/)" "$BASE_URL/" 200

56# www alias — only runs when the base URL has no www and is HTTPS

57if [[ "$BASE_URL" =~ ^https://[^/]+$ ]] && [[ ! "$BASE_URL" =~ https://www\. ]]; then

58 www_url="${BASE_URL/https:\/\//https:\/\/www.}"

59 check "www alias /healthz" "$www_url/healthz" 200

60fi

62if [[ "$fail" -eq 0 ]]; then

63 printf '\n\033[32mAll checks passed.\033[0m\n'

64 exit 0

65fi

67printf '\n\033[31m%d check(s) failed.\033[0m\n' "$fail"

68exit 1

069