CodeIssuesDiscussionsWikiPull RequestsProjectsCommitsActionsReleasesContributorsPulse● GatesSecuritySettingsDeploymentsPipelineInsightsAgents✨ Explain✨ Ask AI✨ Workspace✨ Spec✨ Tests▓ Debt Map✨ NL Search🏛 Archaeology
Blame · Line-by-line history
DEPLOY.md
Each line is annotated with the commit that last touched it. Click any SHA to jump to that commit and see the surrounding change.
| 988380a | 1 | # Deploying Gluecron |
| 8ade77b | 2 | |
| 90fa787 | 3 | Gluecron is a standalone product. It runs anywhere Bun runs. The repo ships a `fly.toml` for Fly.io as the documented primary target, and a `Dockerfile` for any other container host. |
| 3644238 | 4 | |
| 9dd96b9 | 5 | > **Already deployed?** Day-to-day operations (enable AI auto-merge, trigger a deploy, rollback) all live at [`/admin/ops`](https://gluecron.com/admin/ops). Live deploy progress streams to [`/admin/deploys`](https://gluecron.com/admin/deploys). No SSH required — see §6 below. This document covers first-time bootstrap and the terminal fallbacks. |
| 6 | ||
| 988380a | 7 | --- |
| 3644238 | 8 | |
| 988380a | 9 | ## 1. Prerequisites |
| 3644238 | 10 | |
| 988380a | 11 | 1. **Neon Postgres** — create a project at https://neon.tech and copy the pooled connection string. This becomes `DATABASE_URL`. |
| 90fa787 | 12 | 2. **Fly.io account** (or any Docker-compatible host) — `flyctl` installed locally if you're using Fly. |
| 988380a | 13 | 3. **(Recommended) Anthropic API key** — https://console.anthropic.com. Everything AI-flavoured degrades to safe fallbacks without it, but you'll want this for the differentiator features. |
| 3644238 | 14 | |
| 988380a | 15 | --- |
| 8ade77b | 16 | |
| 90fa787 | 17 | ## 2. Deploy to Fly.io |
| 8ade77b | 18 | |
| 90fa787 | 19 | The repo includes a ready `fly.toml`. First-time setup: |
| 8ade77b | 20 | |
| 90fa787 | 21 | ```bash |
| 22 | fly launch # adopts the existing fly.toml; pick an app name and region | |
| 23 | fly deploy # builds via the in-repo Dockerfile and releases | |
| 24 | ``` | |
| 25 | ||
| 26 | The shipped `fly.toml` already wires up: | |
| 27 | ||
| 28 | - **Release command:** `bun run db:migrate` (runs before each deploy cuts over) | |
| 29 | - **Persistent volume:** `gluecron_repos` mounted at `/app/repos` (bare git repos live on disk) | |
| 30 | - **HTTP service** on port 3000 with forced HTTPS | |
| 8ade77b | 31 | |
| 90fa787 | 32 | Set secrets before the first deploy: |
| 8ade77b | 33 | |
| 90fa787 | 34 | ```bash |
| 35 | fly secrets set DATABASE_URL="..." APP_BASE_URL="https://your-app.fly.dev" ANTHROPIC_API_KEY="..." | |
| 36 | ``` | |
| 37 | ||
| 38 | Route a custom domain via `fly certs add your-domain.com` if you want something other than `*.fly.dev`. | |
| 39 | ||
| 40 | ### Other Docker hosts | |
| 41 | ||
| 42 | Any platform that runs the in-repo `Dockerfile` works. Required wiring: | |
| 43 | ||
| 44 | - **Build:** `docker build .` | |
| 45 | - **Release (pre-start):** `bun run db:migrate` | |
| 46 | - **Start:** `bun run src/index.ts` | |
| 47 | - **Port:** `3000` | |
| 48 | - **Persistent volume:** mount the path set by `GIT_REPOS_PATH` (default `/app/repos`) | |
| 8ade77b | 49 | |
| 988380a | 50 | --- |
| 8ade77b | 51 | |
| 988380a | 52 | ## 3. Environment variables |
| 8ade77b | 53 | |
| 988380a | 54 | Full reference is in [`.env.example`](./.env.example); cross-reference BUILD_BIBLE §5.2 for anything not listed here. |
| 8ade77b | 55 | |
| 988380a | 56 | ### Required |
| 57 | | Variable | Purpose | | |
| 58 | |---|---| | |
| 59 | | `DATABASE_URL` | Neon Postgres connection string (pooled). | | |
| 60 | | `GIT_REPOS_PATH` | Where bare repos live on disk (e.g. `/data/repos`). | | |
| 61 | | `PORT` | HTTP port. Default `3000`. | | |
| 62 | | `APP_BASE_URL` | Canonical public URL, used when composing outbound emails and webhooks. | | |
| 8ade77b | 63 | |
| 988380a | 64 | ### Strongly recommended |
| 65 | | Variable | Purpose | | |
| 66 | |---|---| | |
| 67 | | `ANTHROPIC_API_KEY` | Unlocks AI review, triage, chat, incident responder, auto-repair, completions. | | |
| 68 | | `EMAIL_PROVIDER` | `log` (default, writes to stderr) or `resend` (production). | | |
| 69 | | `EMAIL_FROM` | Sender identity on outbound mail. | | |
| 70 | | `RESEND_API_KEY` | Required when `EMAIL_PROVIDER=resend`. | | |
| 71 | | `VOYAGE_API_KEY` | Upgrades semantic search to Voyage `voyage-code-3` embeddings; without it, a deterministic hashing embedder is used. | | |
| 72 | ||
| 90fa787 | 73 | ### Integrations (all optional) |
| 988380a | 74 | | Variable | Purpose | |
| 75 | |---|---| | |
| 90fa787 | 76 | | `GATETEST_URL` | Outbound push webhook to the GateTest third-party security scanner. Default `https://gatetest.ai/api/events/push`. | |
| 988380a | 77 | | `GATETEST_API_KEY` | Bearer sent on outbound GateTest posts. | |
| 78 | | `GATETEST_CALLBACK_SECRET` | Inbound bearer GateTest must present on result callbacks. See `GATETEST_HOOK.md`. | | |
| 79 | | `GATETEST_HMAC_SECRET` | HMAC secret for inbound GateTest callbacks (alternative to bearer). | | |
| 90fa787 | 80 | | `CRONTECH_DEPLOY_URL` | Optional outbound deploy webhook. When set, pushes to the default branch POST here. | |
| 81 | | `GLUECRON_WEBHOOK_SECRET` | Bearer sent on the outbound deploy webhook above. | | |
| 82 | | `CRONTECH_EVENT_TOKEN` | Bearer an external deploy service must present on `deploy.succeeded` / `deploy.failed` callbacks to `POST /api/events/deploy`. | | |
| 83 | | `CRONTECH_STATUS_URL` / `GLUECRON_STATUS_URL` / `GATETEST_STATUS_URL` | Third-party platform-status endpoints surfaced in the `/admin/platform` widget. | | |
| 8ade77b | 84 | |
| 988380a | 85 | ### Operational flags |
| 86 | | Variable | Purpose | | |
| 87 | |---|---| | |
| 88 | | `AUTOPILOT_DISABLED=1` | Opt out of the 5-minute autopilot ticker (mirror sync, merge-queue processing, weekly digests, advisory rescans). Default: enabled. | | |
| 89 | | `DEMO_SEED_ON_BOOT=1` | Idempotently create a `demo` user plus three public sample repos (`hello-python`, `todo-api`, `design-docs`) on server start. Safe to leave enabled — a second run is a near-instant no-op. Site admins can also trigger a reseed manually from `/admin`. | | |
| 90 | ||
| 534f04a | 91 | ### Web Push / PWA (Block M2) |
| 92 | | Variable | Purpose | | |
| 93 | |---|---| | |
| 94 | | `VAPID_PUBLIC_KEY` | Base64url-encoded uncompressed P-256 public key (65 bytes). Sent to browsers as the `applicationServerKey` for `pushManager.subscribe()`. **If unset, a fresh keypair is generated in-memory at first use and every restart invalidates all existing subscriptions — production must set this.** | | |
| 95 | | `VAPID_PRIVATE_KEY` | Base64url-encoded raw 32-byte private scalar paired with `VAPID_PUBLIC_KEY`. Used to ES256-sign the per-delivery JWT. Treat as a secret. | | |
| 96 | | `VAPID_SUBJECT` | `mailto:` or `https:` URL identifying the app, included in every VAPID JWT. Defaults to `mailto:ops@gluecron.com`. | | |
| 97 | ||
| 98 | To generate a fresh pair locally, run a one-off Bun snippet using Web Crypto's `subtle.generateKey({ name: "ECDSA", namedCurve: "P-256" }, true, ["sign", "verify"])` and base64url-encode the raw public key + the JWK private scalar (`d`). The boot log emits the generated public key on first push send when env vars are absent. | |
| 99 | ||
| 988380a | 100 | --- |
| 101 | ||
| 102 | ## 4. Database migrations | |
| 103 | ||
| 104 | Migrations are checked into `drizzle/` and run via: | |
| 8ade77b | 105 | |
| 106 | ```bash | |
| 988380a | 107 | bun run db:migrate |
| 8ade77b | 108 | ``` |
| 109 | ||
| 90fa787 | 110 | Hook this into your host's release phase so every deploy self-migrates. On Fly.io the in-repo `fly.toml` already wires this up via `[deploy].release_command`. First-time bootstrap: the release command will pick up `0000_init.sql` through the latest migration in order. |
| 8ade77b | 111 | |
| 988380a | 112 | --- |
| 8ade77b | 113 | |
| 988380a | 114 | ## 5. Post-deploy checklist |
| 8ade77b | 115 | |
| 988380a | 116 | Verify these in order after the first deploy: |
| 8ade77b | 117 | |
| 90fa787 | 118 | - [ ] **Release command ran successfully** — `bun run db:migrate` reported success in the release logs. |
| 988380a | 119 | - [ ] **`/healthz` is green** — returns `{"ok": true, ...}` with a 200. |
| 120 | - [ ] **`/readyz` is green** — returns `{"ok": true}`, confirming DB connectivity. | |
| 121 | - [ ] **`/metrics` responds** — basic process snapshot is emitted. | |
| 122 | - [ ] **First admin bootstrap** — register the first account at `/register`. Per `src/lib/admin.ts`, while `site_admins` is empty the **oldest user** is treated as site admin. Register the intended admin account first so the bootstrap rule applies; subsequent admins can then be granted from `/admin/users`. | |
| 123 | - [ ] **Smart HTTP works** — create a repo at `/new`, then `git clone https://<your-host>/<user>/<repo>.git` round-trips. | |
| 90fa787 | 124 | - [ ] **Push triggers the pipeline** — a `git push` fires the secret scanner, webhook fan-out, and (if configured) the outbound GateTest post (check `/:owner/:repo/gates` and `/:owner/:repo/settings/audit`). |
| 988380a | 125 | - [ ] **Custom domain TLS** — certificate issued and `APP_BASE_URL` reflects the canonical host. |
| 126 | - [ ] **Autopilot state** — if you want the background ticker off, confirm `AUTOPILOT_DISABLED=1` is set; otherwise expect mirror syncs and weekly digests to fire on schedule. | |
| 8ade77b | 127 | |
| 988380a | 128 | --- |
| 8ade77b | 129 | |
| 9dd96b9 | 130 | ## 6. Day-to-day operations |
| 131 | ||
| 132 | Every routine action is a button click on [`/admin/ops`](https://gluecron.com/admin/ops): | |
| 133 | ||
| 134 | - **Enable AI auto-merge on main** — flips the per-repo opt-in covered in §8 below. | |
| 135 | - **Trigger a deploy** — fires the same path as a push to the default branch. | |
| 136 | - **Rollback to the previous successful release** — selects from the history shown on `/admin/deploys`. | |
| 137 | ||
| 138 | Live deploy progress streams to [`/admin/deploys`](https://gluecron.com/admin/deploys) while the workflow runs. No SSH required for any of this. | |
| 8ade77b | 139 | |
| b1be050 | 140 | **Post-deploy verification is automatic.** Every deploy runs a 15-endpoint smoke suite (`scripts/post-deploy-smoke.ts`) after `systemctl restart`. If any endpoint returns the wrong status or shape — including the case where migrations didn't apply and `/login` now 500s — the workflow auto-rolls back to the previous successful SHA and marks the deploy failed. You'll see this on `/admin/deploys` with a red status pill and a "ROLLED BACK" reason header. The same step also reads `_migrations` from the live DB and refuses to mark the deploy successful if the latest `drizzle/*.sql` file isn't recorded — closing the silent-migration-failure gap that broke gluecron.com for hours on 2026-05-13. |
| 141 | ||
| 988380a | 142 | ### Logs |
| 9dd96b9 | 143 | Every request carries an `X-Request-Id` header. Grep `/admin/deploys` (per-deploy log panel) or your host's log stream for that ID when tracing a report. |
| 988380a | 144 | |
| 145 | ### Restarts | |
| 9dd96b9 | 146 | Trigger from `/admin/ops` ("Trigger a deploy" — restart-equivalent) or your host's dashboard. Rate-limit counters are in-memory and reset on restart — that is intentional. |
| 988380a | 147 | |
| 148 | ### Rollbacks | |
| 9dd96b9 | 149 | Use the "Rollback" button on `/admin/ops`. Database migrations are additive; rolling back the service does not roll back the schema. If a migration needs reverting, write a new forward-migration. |
| 988380a | 150 | |
| 151 | ### Backups | |
| 90fa787 | 152 | Neon handles PITR + branch snapshots — configure retention in the Neon console. The bare repos on the persistent volume must be backed up separately (filesystem snapshot of the mount at `GIT_REPOS_PATH`; on Fly.io, snapshot the `gluecron_repos` volume). See BUILD_BIBLE §2.6 for what still needs wiring on the observability side. |
| 988380a | 153 | |
| 9dd96b9 | 154 | <details> |
| 155 | <summary>Manual fallback (terminal)</summary> | |
| 156 | ||
| 157 | Only needed for first-time box bootstrap or if `/admin/ops` is itself broken. | |
| 158 | ||
| 159 | ```bash | |
| 160 | # Logs (Fly.io) | |
| 161 | fly logs | |
| 162 | ||
| 163 | # Restart (Fly.io) | |
| 164 | fly apps restart | |
| 165 | ||
| 166 | # Rollback (Fly.io) | |
| 167 | fly releases | |
| 168 | fly deploy --image <previous> | |
| 169 | ||
| 170 | # Restart on Hetzner (matches scripts/bootstrap-hetzner.sh) | |
| 171 | ssh root@gluecron.com 'systemctl restart gluecron' | |
| 172 | ||
| 173 | # Rollback on Hetzner | |
| 174 | ssh root@gluecron.com 'cd /opt/gluecron && git checkout <previous-sha> && systemctl restart gluecron' | |
| 175 | ``` | |
| 176 | ||
| 177 | </details> | |
| 178 | ||
| 988380a | 179 | --- |
| 8ade77b | 180 | |
| 988380a | 181 | ## 7. Graceful-degradation matrix |
| 8ade77b | 182 | |
| 988380a | 183 | | Missing secret | Effect | |
| 184 | |---|---| | |
| 185 | | `DATABASE_URL` | App boots, `/healthz` returns 200, any DB-backed route returns 500. Do not deploy without it. | | |
| 186 | | `ANTHROPIC_API_KEY` | AI review, chat, triage, incident, completions, explain, test-gen, merge resolver all return deterministic fallback strings. Site remains fully usable as a plain git host. | | |
| 187 | | `GATETEST_API_KEY` | Outbound GateTest integration silently skipped. Local secret scanner + gate runner still execute. | | |
| 188 | | `RESEND_API_KEY` (with `EMAIL_PROVIDER=resend`) | `sendEmail()` still never throws; emails are logged instead of delivered. | | |
| 189 | | `VOYAGE_API_KEY` | Semantic search falls back to the deterministic hashing embedder. Quality drops; feature still works. | | |
| 8ade77b | 190 | |
| 988380a | 191 | Every missing integration follows the same rule: **never break the primary request path**. See BUILD_BIBLE §4.9 for the full invariant list. |
| f764c07 | 192 | |
| 193 | --- | |
| 194 | ||
| 9dd96b9 | 195 | ## 8. Lightning-fast deploys (Block N1) |
| f764c07 | 196 | |
| 9dd96b9 | 197 | Once migration 0040 has been applied (the release command does this automatically), the operator can flip a single repo into "auto-merge mode" with a single click. From "feature description → live in ~6 minutes, zero clicks" goes from possible to the default. |
| f764c07 | 198 | |
| 199 | ### What it does | |
| 200 | ||
| 201 | When `branch_protection.enable_auto_merge=true` on the matching rule, the K3 autopilot sweep (`auto-merge-sweep`, fires every 5 minutes) will auto-merge any PR that: | |
| 202 | ||
| 203 | - Targets a branch matching the rule's `pattern` | |
| 204 | - Is not a draft | |
| 205 | - Passes every gate the manual-merge path enforces (green gates, AI approval, required checks) | |
| 206 | - Is not flagged as `critical` by the M3 PR risk scorer | |
| 207 | ||
| 208 | The merge is performed by the autopilot, not by any human click. Default-deny: this only fires on rules where the operator has explicitly opted in. | |
| 209 | ||
| 9dd96b9 | 210 | ### Web flow (primary) |
| 211 | ||
| 212 | 1. Go to [`/admin/ops`](https://gluecron.com/admin/ops). | |
| 213 | 2. Click **"Enable AI auto-merge"**, select the repo (e.g. `ccantynz/Gluecron.com`) and pattern (defaults to `main`). | |
| 214 | 3. The page runs the readiness check inline (migration 0040 applied, `ANTHROPIC_API_KEY` set, autopilot running, `auto-merge-sweep` registered) and shows red lines if anything is wrong. | |
| 215 | 4. Confirm. The page writes the `enable_auto_merge=true` flip and surfaces the resulting audit-log row. | |
| 216 | ||
| 217 | Within 5 minutes (the autopilot sweep cadence), any qualifying PR on that branch will auto-merge with zero human clicks. Use the same screen's **"Disable"** toggle to revert. | |
| f764c07 | 218 | |
| 9dd96b9 | 219 | <details> |
| 220 | <summary>Manual fallback (terminal)</summary> | |
| 221 | ||
| 222 | Only needed if `/admin/ops` is broken. Otherwise prefer the web flow above. | |
| 223 | ||
| 224 | #### Step 1 — Readiness check | |
| f764c07 | 225 | |
| 226 | ```bash | |
| 227 | # Fly.io | |
| 228 | fly ssh console -C "bun run /opt/gluecron/scripts/check-auto-merge-readiness.ts" | |
| 229 | ||
| 230 | # Hetzner (matches scripts/bootstrap-hetzner.sh) | |
| 231 | ssh root@gluecron.com "cd /opt/gluecron && bun run scripts/check-auto-merge-readiness.ts" | |
| 232 | ``` | |
| 233 | ||
| 234 | The check verifies: | |
| 235 | - Migration 0040 has been applied (`branch_protection.enable_auto_merge` column exists) | |
| 236 | - `ANTHROPIC_API_KEY` is set (otherwise the AI approval gate blocks every candidate) | |
| 237 | - The autopilot is running (`AUTOPILOT_DISABLED != 1`) | |
| 238 | - The K3 `auto-merge-sweep` task is in `defaultTasks()` | |
| 239 | ||
| 240 | Exit code 0 if all green; 1 if any check fails. Fix any red lines before continuing. | |
| 241 | ||
| 9dd96b9 | 242 | #### Step 2 — Flip the switch for a single repo |
| f764c07 | 243 | |
| 244 | ```bash | |
| 245 | # Fly.io | |
| 246 | fly ssh console -C "bun run /opt/gluecron/scripts/enable-auto-merge.ts ccantynz/Gluecron.com" | |
| 247 | ||
| 248 | # Hetzner | |
| 249 | ssh root@gluecron.com "cd /opt/gluecron && bun run scripts/enable-auto-merge.ts ccantynz/Gluecron.com" | |
| 250 | ``` | |
| 251 | ||
| 252 | By default this targets the `main` branch. Pass a second arg to target a different pattern, e.g. `release/*`: | |
| 253 | ||
| 254 | ```bash | |
| 255 | bun run scripts/enable-auto-merge.ts ccantynz/Gluecron.com release/* | |
| 256 | ``` | |
| 257 | ||
| 258 | The script is idempotent: | |
| 259 | - If a `branch_protection` row already exists for the (repo, pattern), it flips `enable_auto_merge=true` and preserves every other field. | |
| 260 | - If no row exists, it inserts a fresh one with the documented safety defaults (`require_green_gates=true`, `require_ai_approval=true`, `require_human_review=false`, `required_approvals=0`). | |
| 261 | - Running it twice in a row produces a "no-op" — no duplicate audit entry. | |
| 262 | ||
| 263 | It prints a before / after diff so you see exactly what changed, and writes an `auto_merge.enabled_on_main` row to `audit_log` for traceability. | |
| 264 | ||
| 9dd96b9 | 265 | #### Revert |
| f764c07 | 266 | |
| 267 | ```bash | |
| 268 | bun run scripts/enable-auto-merge.ts ccantynz/Gluecron.com --off | |
| 269 | ``` | |
| 270 | ||
| 271 | Or manually: `UPDATE branch_protection SET enable_auto_merge = false WHERE repository_id = ... AND pattern = 'main';`. The autopilot sweep is default-deny — it will stop firing the moment the column flips back to `false`. | |
| 272 | ||
| 9dd96b9 | 273 | #### Don'ts |
| f764c07 | 274 | |
| 275 | - Do not run `enable-auto-merge.ts` without first running the readiness check. The AI approval gate silently fails closed when `ANTHROPIC_API_KEY` is missing, and you'll spend half an hour wondering why nothing auto-merges. | |
| 276 | - The script intentionally does NOT auto-discover repos and flip them all. Explicit per-repo opt-in is the whole point — the operator decides which repos go on autopilot. | |
| 9dd96b9 | 277 | |
| 278 | </details> |