Pre-launch — Gluecron is in final validation. Public signups and git hosting for non-owner users open after launch review.
Commitcf03ff5unknown_key

feat(BLOCK-S): never-lie-to-the-user-about-being-deployed pass

feat(BLOCK-S): never-lie-to-the-user-about-being-deployed pass

The user spent an entire day frustrated because:
  1. Migrations 0046-0053 silently failed to apply on the live DB
     after the PR #62 merge. /healthz still returned 200 (it doesn't
     touch the schema), so the deploy was marked SUCCEEDED while
     every page that queried `users` crashed. Login broken for ~7h.
  2. PWA service worker held the previous /login HTML in its own
     cache. Hard-refresh didn't beat it; user had to DevTools →
     Application → Service Workers → Unregister.
  3. Nothing alerted them. They had to discover the breakage by
     trying to log in.

This block makes those silent failures impossible.

S1+S3 — Make migrations fail loud + post-deploy smoke + auto-rollback
  scripts/post-deploy-smoke.ts (NEW) +
  src/lib/post-deploy-smoke.ts (NEW) +
  src/__tests__/post-deploy-smoke.test.ts (NEW) +
  .github/workflows/hetzner-deploy.yml (extended) +
  src/routes/version.ts (extended) + DEPLOY.md note.
  • db-migrate step now fails the workflow on any error (no more
    `|| echo WARN` swallow) AND writes a temporary
    verify-migration.mjs that aborts the deploy if the latest
    `drizzle/*.sql` filename isn't in the live DB's `_migrations`
    table.
  • New "Full post-deploy smoke suite" step runs after restart,
    curls 15 critical endpoints (/healthz, /readyz, /version,
    /login, /register, /, /explore, /demo, /pricing, /status,
    /api/v2/healthz, /mcp, /manifest.webmanifest, /sw.js,
    /gluecron.dxt), then curls /api/version and checks the
    server's reported migrations[] contains the latest filename.
  • Rollback step fires on either smoke step failing — once per
    workflow run — resets to the previous successful SHA,
    re-builds + re-restarts, verifies /healthz on the rolled-back
    state, and posts to /api/events/deploy/finished with status =
    failed + a ROLLED BACK reason header.
  • /api/version + /version now report `{sha, buildAt, migrations}`
    (last 5 applied) so the smoke suite (and humans) can verify
    schema parity.
  • 35 new tests covering CHECKS shape, every asserter, sequential
    ordering, fetch-throw handling, migration helpers.

S2 — Service worker cache-bust pinned to deploy SHA
  src/routes/pwa.ts (extended) +
  src/views/layout.tsx (extended) +
  src/__tests__/pwa-cache-bust.test.ts (NEW).
  • `/sw.js` is now built fresh from `buildVersionedServiceWorker()`
    at request time. Top of the body is `const SW_VERSION = "<sha>";`
    read from process.env.BUILD_SHA (or a stable dev-fallback with
    one-shot warn).
  • SW install handler calls `self.skipWaiting()` so the new SW
    activates immediately — no waiting for every tab to close.
  • SW activate handler deletes every previous gluecron-<version>
    cache so stale assets are nuked the moment the new SW takes
    over.
  • Layout's pwaRegisterScript extended with
    updatefound → statechange → window.location.reload() (gated
    on controller existence + a reloaded latch so the first
    install doesn't loop).
  • Result: the "I deployed but the user still sees the old page"
    window shrinks from "forever until DevTools" to "one auto-
    reload on the next page load".
  • `Cache-Control: no-store` on /sw.js itself so browsers always
    re-fetch the SW source on update probe.
  • Locked G1 SERVICE_WORKER_SRC constant left untouched (existing
    pwa.test.ts assertions still hold); the served body is built
    by the new helper.
  • 13 new tests.

S4 — Synthetic monitor every tick + red/green admin status
  drizzle/0054_synthetic_checks.sql (NEW) +
  src/lib/synthetic-monitor.ts (NEW) +
  src/routes/admin-status.tsx (NEW) +
  src/__tests__/synthetic-monitor.test.ts (NEW) +
  src/lib/autopilot.ts (extended) +
  src/routes/status.tsx (extended) + app.tsx mount.
  • 14 checks (same shape as S1+S3's smoke suite but URL-only:
    healthz / readyz / version / login renders / register renders
    / landing / explore / pricing / status / manifest / sw.js /
    gluecron.dxt / mcp discovery / robots.txt). Per-check timeout
    5s.
  • New `synthetic-monitor` autopilot task runs every tick. On
    green→red transition, posts to MONITOR_ALERT_WEBHOOK_URL
    (if env set). Persists every result to `synthetic_checks`
    with timestamp + duration + error.
  • SSE topic `monitor:synthetic` for live UI updates.
  • New `/admin/status` site-admin page renders the 14-row table
    with colored dots, status codes, durations, relative times.
    SSE-subscribes to monitor:synthetic so red transitions show
    up without a page reload. "Run all checks now" button bypasses
    the tick for instant verification.
  • Public `/status` page now has a "Recent incidents (last 24h)"
    section listing any check that went red.
  • 22 new tests.

──────────────────────────────────────────────────────────────────────
Test outcome
──────────────────────────────────────────────────────────────────────

bun test → 1936 tests / 1 fail / 2 skip across 139 files.

The 1 fail remains the pre-existing
`runScheduledWorkflowsTick — fail-open` cross-file mock-ordering
artifact (passes in isolation). Unchanged across BLOCK-P/Q/R/S.

Net +70 tests this block: S1+S3:35  S2:13  S4:22.

──────────────────────────────────────────────────────────────────────
What this changes for the operator
──────────────────────────────────────────────────────────────────────

• Deploy can no longer succeed silently with a broken schema.
  Migrations either apply OR the deploy auto-rolls back.
• Browser users see the new version on the next page load,
  automatically. No more DevTools dance.
• When something breaks, /admin/status flips red in seconds.
  Webhook alert fires on the green→red edge.
• The platform self-monitors. No more "the user is the QA team."
Test User committed on May 14, 2026Parent: 9dd96b9
17 files changed+261016cf03ff5bc196c8266bf3e36b5ec829b75b42a509
17 changed files+2610−16
Modified.github/workflows/hetzner-deploy.yml+178−7View fileUnifiedSplit
267267 echo "==> $UNIT already matches desired state — skipping daemon-reload"
268268 fi
269269
270 # ─── DB migrations (cheap; safe to always run)
270 # ─── DB migrations
271 #
272 # Block S1 (2026-05-14): migrate.ts MUST be allowed to fail
273 # the entire workflow. The previous `|| echo WARN` swallowed
274 # the exit code, which is exactly how migrations 0046-0053
275 # silently skipped a real deploy and left the live site
276 # crashing every request that touched `users.*`. We now:
277 # - run migrate.ts and abort the deploy on non-zero exit
278 # - stream stdout + stderr live to the workflow log
279 # - then read back the applied list from _migrations and
280 # refuse to restart if the latest drizzle/*.sql file
281 # isn't in it.
271282 notify_step "db-migrate" "in_progress"
272283 DM_START=$(date +%s)
273284 set -a; source /etc/gluecron.env; set +a
274 "$BUN" run src/db/migrate.ts || echo "WARN: migrate failed (may be already-applied)"
285
286 echo "==> running migrations (must succeed to proceed)"
287 if ! "$BUN" run src/db/migrate.ts; then
288 notify_step "db-migrate" "failed" "$(( ( $(date +%s) - DM_START ) * 1000 ))"
289 echo "ERROR: bun run db:migrate failed — aborting deploy" >&2
290 exit 1
291 fi
292
293 # Verify the LATEST drizzle/*.sql is present in _migrations.
294 # This catches the case where migrate.ts thought every file
295 # was already applied but the actual list on disk has new
296 # entries the runner somehow skipped. We write a tiny ESM
297 # verifier into /tmp and call it; this avoids the require()
298 # vs ESM mismatch you'd hit using `bun -e` on a "type":"module"
299 # package.
300 echo "==> verifying latest drizzle/*.sql is recorded in _migrations"
301 LATEST_FILE=$(ls drizzle/*.sql 2>/dev/null | sort | tail -1 | xargs -n1 basename || true)
302 if [ -z "$LATEST_FILE" ]; then
303 echo "WARN: no drizzle/*.sql files found — skipping verification"
304 else
305 cat > /tmp/verify-migration.mjs <<'VERIFY_EOF'
306 import { neon } from "@neondatabase/serverless";
307 import postgres from "postgres";
308 const url = process.env.DATABASE_URL;
309 const target = process.argv[2];
310 if (!url || !target) { console.error("verify: missing DATABASE_URL or target"); process.exit(1); }
311 const isNeon = /(^|\.)neon\.tech$/i.test(new URL(url).hostname);
312 try {
313 let rows;
314 if (isNeon) {
315 const sql = neon(url);
316 rows = await sql(`SELECT name FROM _migrations WHERE name = $1`, [target]);
317 } else {
318 const client = postgres(url, { max: 1, prepare: false });
319 rows = await client`SELECT name FROM _migrations WHERE name = ${target}`;
320 await client.end({ timeout: 5 });
321 }
322 if (rows.length === 0) {
323 console.error(`verify: ${target} is NOT in _migrations`);
324 process.exit(2);
325 }
326 console.log(`verify: ${target} is applied`);
327 } catch (e) {
328 console.error("verify: query failed:", e?.message ?? e);
329 process.exit(3);
330 }
331 VERIFY_EOF
332 if ! "$BUN" run /tmp/verify-migration.mjs "$LATEST_FILE"; then
333 echo "ERROR: latest migration $LATEST_FILE is NOT recorded in _migrations — aborting deploy" >&2
334 notify_step "db-migrate" "failed" "$(( ( $(date +%s) - DM_START ) * 1000 ))"
335 exit 1
336 fi
337 fi
338
275339 notify_step "db-migrate" "succeeded" "$(( ( $(date +%s) - DM_START ) * 1000 ))"
276340
277341 # ─── (d) Zero-downtime restart. Blocks until sd_notify(READY=1).
349413 app_base_url: ${{ secrets.APP_BASE_URL || 'https://gluecron.com' }}
350414 deploy_event_token: ${{ secrets.DEPLOY_EVENT_TOKEN }}
351415
416 # ─── 3b. Full post-deploy smoke suite (Block S1+S3) ──────────────────
417 # `/healthz` alone is NOT enough — it doesn't touch the DB schema, so
418 # a broken migration leaves it green while every real page crashes
419 # selecting columns that don't exist. The post-deploy-smoke script
420 # hits 15 critical endpoints (login renders, /api/version, /demo,
421 # /mcp, /sw.js, etc.) and verifies the LATEST drizzle/*.sql is in
422 # the running process's reported migrations list. If ANY check
423 # fails, the workflow auto-rolls back.
424 - name: Notify step full-smoke (in_progress)
425 if: env.DEPLOY_EVENT_TOKEN != ''
426 uses: ./.github/actions/notify-deploy-step
427 with:
428 step_name: full-smoke
429 status: in_progress
430 app_base_url: ${{ secrets.APP_BASE_URL || 'https://gluecron.com' }}
431 deploy_event_token: ${{ secrets.DEPLOY_EVENT_TOKEN }}
432
433 - name: Full post-deploy smoke suite
434 id: full_smoke
435 uses: appleboy/ssh-action@v1.2.0
436 with:
437 host: ${{ secrets.HETZNER_HOST }}
438 username: ${{ secrets.HETZNER_USER }}
439 key: ${{ secrets.HETZNER_SSH_KEY }}
440 script_stop: true
441 script: |
442 set -e
443 cd /opt/gluecron
444 BUN=/root/.bun/bin/bun
445 export GLUECRON_HOST="http://localhost:3010"
446 echo "==> running 15-endpoint smoke suite against $GLUECRON_HOST"
447 # The script exits 1 on endpoint failure, 2 on missing
448 # migration. We treat both as fatal so the workflow rolls
449 # back. stdout/stderr stream live to the GH Actions log.
450 "$BUN" run scripts/post-deploy-smoke.ts
451
452 - name: Notify step full-smoke (succeeded)
453 if: success() && env.DEPLOY_EVENT_TOKEN != ''
454 uses: ./.github/actions/notify-deploy-step
455 with:
456 step_name: full-smoke
457 status: succeeded
458 app_base_url: ${{ secrets.APP_BASE_URL || 'https://gluecron.com' }}
459 deploy_event_token: ${{ secrets.DEPLOY_EVENT_TOKEN }}
460
461 - name: Notify step full-smoke (failed)
462 if: failure() && steps.full_smoke.conclusion == 'failure' && env.DEPLOY_EVENT_TOKEN != ''
463 uses: ./.github/actions/notify-deploy-step
464 with:
465 step_name: full-smoke
466 status: failed
467 app_base_url: ${{ secrets.APP_BASE_URL || 'https://gluecron.com' }}
468 deploy_event_token: ${{ secrets.DEPLOY_EVENT_TOKEN }}
469
352470 # ─── 4. Auto-rollback on smoke failure ──────────────────────────────
353471 # Only rolls back if the workflow was triggered by a normal push.
354472 # Manual workflow_dispatch runs SKIP rollback so the operator can
355473 # diagnose the new code on the box before reverting. This stops the
356474 # pathological case where rollback masks the real failure by reverting
357475 # to an already-broken previous SHA.
476 #
477 # S1 (2026-05-14): rollback now also fires when the FULL smoke suite
478 # fails (steps.full_smoke), not just the basic /healthz curl. Recursion
479 # cap: only ONE rollback attempt per workflow run (the `if:` guard
480 # naturally enforces this — the only rollback step in the file).
358481 - name: Rollback on failure
359 if: failure() && steps.smoke.conclusion == 'failure' && github.event_name == 'push'
482 id: rollback
483 if: failure() && github.event_name == 'push' && (steps.smoke.conclusion == 'failure' || steps.full_smoke.conclusion == 'failure')
360484 uses: appleboy/ssh-action@v1.2.0
361485 with:
362486 host: ${{ secrets.HETZNER_HOST }}
365489 script_stop: false
366490 script: |
367491 cd /opt/gluecron
368 prev=$(cat /tmp/gluecron_prev_sha)
369 echo "Rolling back to $prev"
492 prev=$(cat /tmp/gluecron_prev_sha 2>/dev/null || true)
493 if [ -z "$prev" ]; then
494 echo "ROLLBACK SKIPPED: no /tmp/gluecron_prev_sha — human intervention required"
495 exit 1
496 fi
497 echo "ROLLED BACK to $prev because post-deploy smoke failed"
370498 git reset --hard "$prev"
499 # Don't re-run migrations here: rolling back schema is
500 # destructive and migrations are forward-only. We just put
501 # the code back to where it was and restart. If a migration
502 # is the reason the new code is incompatible with the old,
503 # the operator must intervene manually.
504 BUN=/root/.bun/bin/bun
505 if [ -f bun.lock ] && [ -d node_modules ]; then
506 echo "==> reusing existing node_modules (lockfile hash check skipped during rollback)"
507 else
508 "$BUN" install --frozen-lockfile || true
509 fi
371510 systemctl restart gluecron
372 sleep 5
511 # Verify the rollback target itself comes up green.
512 sleep 3
513 for i in 1 2 3; do
514 code=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3010/healthz)
515 echo "Rollback healthz attempt $i: $code"
516 if [ "$code" = "200" ]; then
517 echo "OK: rolled-back instance is healthy"
518 exit 0
519 fi
520 sleep 2
521 done
522 echo "WARN: rollback target ALSO failed /healthz — human intervention required"
373523 systemctl status gluecron --no-pager | head -20 || true
524 exit 1
374525
375526 # ─── 5. Failure diagnostics — captured into a file for summary + AI ──
376527 - name: Capture failure context
498649 APP_BASE_URL: ${{ secrets.APP_BASE_URL || 'https://gluecron.com' }}
499650 START_EPOCH: ${{ steps.start.outputs.epoch }}
500651 DIAG: ${{ steps.ctx.outputs.stdout }}
652 ROLLBACK_OUTCOME: ${{ steps.rollback.outcome }}
653 SMOKE_OUTCOME: ${{ steps.smoke.conclusion }}
654 FULL_SMOKE_OUTCOME: ${{ steps.full_smoke.conclusion }}
501655 run: |
502656 DUR_MS=$(( ( $(date +%s) - START_EPOCH ) * 1000 ))
657 # Build a short reason header. S1 (2026-05-14): the deploy-
658 # finished payload now records WHICH smoke layer failed and
659 # whether the rollback succeeded, so /admin/deploys shows a
660 # red pill with the actual cause instead of "deploy failed".
661 REASON_HEADER=""
662 if [ "$FULL_SMOKE_OUTCOME" = "failure" ]; then
663 REASON_HEADER="post-deploy smoke suite failed"
664 elif [ "$SMOKE_OUTCOME" = "failure" ]; then
665 REASON_HEADER="/healthz smoke failed"
666 fi
667 if [ -n "$ROLLBACK_OUTCOME" ] && [ "$ROLLBACK_OUTCOME" != "skipped" ]; then
668 if [ "$ROLLBACK_OUTCOME" = "success" ]; then
669 REASON_HEADER="ROLLED BACK — $REASON_HEADER"
670 else
671 REASON_HEADER="ROLLBACK FAILED — $REASON_HEADER — human intervention required"
672 fi
673 fi
503674 # First 1 KB of diagnostics — keeps the JSON small and the DB row sane.
504 ERR_TEXT=$(printf '%s' "${DIAG:-deploy failed; see workflow logs}" | head -c 1024)
675 ERR_TEXT=$(printf '%s\n\n%s' "${REASON_HEADER:-deploy failed}" "${DIAG:-see workflow logs}" | head -c 1024)
505676 # jq -Rs '.' is the safest way to JSON-escape arbitrary multi-line text.
506677 if command -v jq >/dev/null 2>&1; then
507678 ERR_JSON=$(printf '%s' "$ERR_TEXT" | jq -Rs '.')
ModifiedDEPLOY.md+2−0View fileUnifiedSplit
137137
138138Live deploy progress streams to [`/admin/deploys`](https://gluecron.com/admin/deploys) while the workflow runs. No SSH required for any of this.
139139
140**Post-deploy verification is automatic.** Every deploy runs a 15-endpoint smoke suite (`scripts/post-deploy-smoke.ts`) after `systemctl restart`. If any endpoint returns the wrong status or shape — including the case where migrations didn't apply and `/login` now 500s — the workflow auto-rolls back to the previous successful SHA and marks the deploy failed. You'll see this on `/admin/deploys` with a red status pill and a "ROLLED BACK" reason header. The same step also reads `_migrations` from the live DB and refuses to mark the deploy successful if the latest `drizzle/*.sql` file isn't recorded — closing the silent-migration-failure gap that broke gluecron.com for hours on 2026-05-13.
141
140142### Logs
141143Every request carries an `X-Request-Id` header. Grep `/admin/deploys` (per-deploy log panel) or your host's log stream for that ID when tracing a report.
142144
Addeddrizzle/0054_synthetic_checks.sql+22−0View fileUnifiedSplit
1-- BLOCK S4 — Synthetic monitor history.
2--
3-- The autopilot ticker hits each row in SYNTHETIC_CHECKS (in
4-- src/lib/synthetic-monitor.ts), records the outcome here, and fires a
5-- webhook alert on any green->red transition. Strictly additive: nothing
6-- else reads or writes this table.
7
8CREATE TABLE IF NOT EXISTS synthetic_checks (
9 id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
10 check_name text NOT NULL,
11 status text NOT NULL, -- "green" | "red" | "yellow"
12 status_code integer, -- HTTP status if applicable
13 duration_ms integer NOT NULL,
14 error text, -- non-null when red
15 checked_at timestamptz NOT NULL DEFAULT now()
16);
17
18CREATE INDEX IF NOT EXISTS idx_synthetic_checks_checked_at
19 ON synthetic_checks (checked_at DESC);
20
21CREATE INDEX IF NOT EXISTS idx_synthetic_checks_name_checked_at
22 ON synthetic_checks (check_name, checked_at DESC);
Addedscripts/post-deploy-smoke.ts+129−0View fileUnifiedSplit
1#!/usr/bin/env bun
2/**
3 * Post-deploy smoke CLI.
4 *
5 * Runs the 15-endpoint smoke suite (`src/lib/post-deploy-smoke.ts`)
6 * against $GLUECRON_HOST (default http://localhost:3010) AFTER systemctl
7 * restart but BEFORE the workflow marks the deploy successful.
8 *
9 * Also verifies that the latest *.sql in drizzle/ is present in the
10 * running app's /api/version migrations list. This is the second line
11 * of defence against the silent-migration-failure bug that broke
12 * gluecron.com for hours.
13 *
14 * Exit codes:
15 * 0 — every check passed AND the latest migration is reported by the
16 * live process.
17 * 1 — at least one endpoint failed.
18 * 2 — endpoints all passed but the latest migration is missing from
19 * the live process's reported list (a fresh deploy is running
20 * but the DB schema is behind).
21 *
22 * Usage:
23 * bun scripts/post-deploy-smoke.ts
24 * GLUECRON_HOST=https://gluecron.com bun scripts/post-deploy-smoke.ts
25 */
26
27import { readdir } from "fs/promises";
28import { join } from "path";
29import {
30 runChecks,
31 formatTable,
32 latestMigration,
33 type FetchLike,
34} from "../src/lib/post-deploy-smoke";
35
36const HOST = (process.env.GLUECRON_HOST || "http://localhost:3010").replace(
37 /\/$/,
38 ""
39);
40
41const fetchImpl: FetchLike = async (url, init) => {
42 const res = await fetch(url, init);
43 return {
44 status: res.status,
45 text: () => res.text(),
46 };
47};
48
49async function main() {
50 console.log(`[smoke] target: ${HOST}`);
51 const summary = await runChecks({
52 baseUrl: HOST,
53 fetchImpl,
54 log: (line) => console.log(line),
55 });
56
57 console.log("");
58 console.log(formatTable(summary.results));
59 console.log("");
60 console.log(
61 `[smoke] ${summary.passed}/${summary.results.length} checks passed`
62 );
63
64 if (!summary.ok) {
65 console.error("[smoke] FAIL — at least one endpoint check failed");
66 process.exit(1);
67 }
68
69 // ─── Migration-applied verification ─────────────────────────────────
70 // Curl /api/version and confirm the latest drizzle/*.sql file is in
71 // the migrations[] array the live process reports.
72 let drizzleFiles: string[] = [];
73 try {
74 drizzleFiles = (await readdir(join(process.cwd(), "drizzle"))).filter((f) =>
75 f.endsWith(".sql")
76 );
77 } catch (err) {
78 console.warn(
79 `[smoke] WARN: couldn't read drizzle/ directory (${(err as Error).message}) — skipping migration check`
80 );
81 process.exit(0);
82 }
83 const latest = latestMigration(drizzleFiles);
84 if (!latest) {
85 console.warn("[smoke] WARN: no migration files found — skipping check");
86 process.exit(0);
87 }
88
89 let liveMigrations: string[] | null = null;
90 try {
91 const res = await fetch(`${HOST}/api/version`);
92 if (res.status === 200) {
93 const body = (await res.json()) as { migrations?: unknown };
94 if (Array.isArray(body.migrations)) {
95 liveMigrations = body.migrations.filter(
96 (m): m is string => typeof m === "string"
97 );
98 }
99 }
100 } catch (err) {
101 console.warn(
102 `[smoke] WARN: /api/version migrations fetch failed: ${(err as Error).message}`
103 );
104 }
105
106 if (liveMigrations === null) {
107 console.warn(
108 "[smoke] WARN: /api/version did not report migrations[] — server hasn't been redeployed with the S3 patch yet. Skipping migration check."
109 );
110 process.exit(0);
111 }
112
113 if (!liveMigrations.includes(latest)) {
114 console.error(
115 `[smoke] FAIL: latest migration ${latest} is NOT in the live process's applied list (reported: ${liveMigrations.slice(-5).join(", ")})`
116 );
117 process.exit(2);
118 }
119
120 console.log(
121 `[smoke] OK: latest migration ${latest} is applied (live process confirms)`
122 );
123 process.exit(0);
124}
125
126main().catch((err) => {
127 console.error("[smoke] crashed:", err);
128 process.exit(1);
129});
Addedsrc/__tests__/post-deploy-smoke.test.ts+388−0View fileUnifiedSplit
1/**
2 * Block S1+S3 — post-deploy smoke suite tests.
3 *
4 * Covers the pure layer of `src/lib/post-deploy-smoke.ts`:
5 * - CHECKS array shape (every check has name + url + at least one
6 * expectation)
7 * - assertion helpers (assertStatus / assertKey / assertContains)
8 * - the runner returns ok=false when ANY check fails, ok=true when all
9 * green
10 * - missingMigrations + latestMigration helpers
11 *
12 * No real network, no mock pollution: every test supplies its own
13 * fetch impl via DI.
14 */
15
16import { describe, it, expect } from "bun:test";
17import {
18 CHECKS,
19 assertStatus,
20 assertKey,
21 assertContains,
22 runChecks,
23 formatTable,
24 missingMigrations,
25 latestMigration,
26 type Check,
27 type FetchLike,
28} from "../lib/post-deploy-smoke";
29
30// ─── helpers ────────────────────────────────────────────────────────
31
32function res(status: number, body: string): { status: number; text: () => Promise<string> } {
33 return { status, text: async () => body };
34}
35
36function jsonRes(status: number, obj: unknown) {
37 return res(status, JSON.stringify(obj));
38}
39
40function alwaysOkFetch(): FetchLike {
41 return async (url) => {
42 // Default-OK responses that pass every shipped check in CHECKS.
43 if (url.endsWith("/healthz"))
44 return jsonRes(200, { ok: true, uptimeMs: 1 });
45 if (url.endsWith("/readyz")) return jsonRes(200, { ok: true });
46 if (url.endsWith("/api/version"))
47 return jsonRes(200, { sha: "abcdef0", branch: "main", builtAt: "x", uptimeMs: 1 });
48 if (url.endsWith("/login"))
49 return res(200, "<html><body><h2>Sign in</h2></body></html>");
50 if (url.endsWith("/register"))
51 return res(200, "<html><body><h2>Create account</h2></body></html>");
52 if (url.endsWith("/mcp"))
53 return jsonRes(200, { serverInfo: { name: "gluecron" } });
54 if (url.endsWith("/api/v2/healthz")) return res(404, "not found");
55 if (url.endsWith("/demo")) return res(302, "");
56 return res(200, "<html></html>");
57 };
58}
59
60// ─── CHECKS array shape ─────────────────────────────────────────────
61
62describe("CHECKS array shape", () => {
63 it("has at least the 15 critical endpoints the owner specified", () => {
64 expect(CHECKS.length).toBeGreaterThanOrEqual(15);
65 });
66
67 it("every check has a non-empty name + url + at least one expectation", () => {
68 for (const c of CHECKS) {
69 expect(typeof c.name).toBe("string");
70 expect(c.name.length).toBeGreaterThan(0);
71 expect(typeof c.url).toBe("string");
72 expect(c.url.startsWith("/")).toBe(true);
73 // expectStatus is required; either expectKey or expectContains may
74 // additionally be supplied but a check is valid with just status.
75 expect(c.expectStatus !== undefined).toBe(true);
76 }
77 });
78
79 it("covers the specific endpoints the spec called out", () => {
80 const names = CHECKS.map((c) => c.name);
81 for (const required of [
82 "healthz",
83 "readyz",
84 "version",
85 "login renders",
86 "register renders",
87 "landing renders",
88 "explore renders",
89 "demo renders",
90 "pricing renders",
91 "status renders",
92 "api v2 health",
93 "mcp discovery",
94 "manifest",
95 "sw",
96 "dxt download",
97 ]) {
98 expect(names).toContain(required);
99 }
100 });
101
102 it("api v2 health accepts 200 OR 404 (route is optional)", () => {
103 const v2 = CHECKS.find((c) => c.name === "api v2 health");
104 expect(v2).toBeDefined();
105 expect(Array.isArray(v2!.expectStatus)).toBe(true);
106 expect((v2!.expectStatus as number[]).sort()).toEqual([200, 404]);
107 });
108});
109
110// ─── Assertion helpers ──────────────────────────────────────────────
111
112describe("assertStatus", () => {
113 it("returns null when the actual status matches a single expected", () => {
114 expect(assertStatus(200, 200)).toBeNull();
115 });
116
117 it("returns null when the actual status matches one of an array", () => {
118 expect(assertStatus(404, [200, 404])).toBeNull();
119 expect(assertStatus(200, [200, 404])).toBeNull();
120 });
121
122 it("returns a descriptive error when the status doesn't match", () => {
123 const err = assertStatus(500, 200);
124 expect(err).not.toBeNull();
125 expect(err!).toContain("200");
126 expect(err!).toContain("500");
127 });
128
129 it("returns an error including all acceptable codes when given an array", () => {
130 const err = assertStatus(500, [200, 404]);
131 expect(err!).toContain("200");
132 expect(err!).toContain("404");
133 });
134});
135
136describe("assertKey", () => {
137 it("returns null when the JSON has the key", () => {
138 expect(assertKey(`{"ok":true}`, "ok")).toBeNull();
139 expect(assertKey(`{"sha":"abc","x":1}`, "sha")).toBeNull();
140 });
141
142 it("returns null even when the key's value is null/0/empty (presence-only)", () => {
143 expect(assertKey(`{"ok":null}`, "ok")).toBeNull();
144 expect(assertKey(`{"ok":0}`, "ok")).toBeNull();
145 expect(assertKey(`{"ok":""}`, "ok")).toBeNull();
146 });
147
148 it("returns an error when the body isn't JSON", () => {
149 expect(assertKey("<html></html>", "ok")).not.toBeNull();
150 });
151
152 it("returns an error when the body is JSON but lacks the key", () => {
153 expect(assertKey(`{"other":1}`, "ok")).not.toBeNull();
154 });
155
156 it("returns an error when the body is a JSON array (not an object)", () => {
157 expect(assertKey(`[1,2,3]`, "ok")).not.toBeNull();
158 });
159
160 it("doesn't fall for prototype-pollution lookups", () => {
161 // `__proto__` is not an own property of {}, so the helper must
162 // treat it as missing. (Object.prototype.hasOwnProperty.call is the
163 // implementation choice.)
164 expect(assertKey(`{}`, "toString")).not.toBeNull();
165 expect(assertKey(`{}`, "__proto__")).not.toBeNull();
166 });
167});
168
169describe("assertContains", () => {
170 it("returns null when the body contains the substring", () => {
171 expect(assertContains("hello world", "world")).toBeNull();
172 });
173
174 it("returns an error including the missing substring", () => {
175 const err = assertContains("hello", "world");
176 expect(err).not.toBeNull();
177 expect(err!).toContain("world");
178 });
179
180 it("is case-sensitive (matches the production check exactly)", () => {
181 expect(assertContains("Sign In", "Sign in")).not.toBeNull();
182 });
183});
184
185// ─── runChecks runner ───────────────────────────────────────────────
186
187describe("runChecks", () => {
188 it("returns ok=true and failed=0 when every check passes", async () => {
189 const summary = await runChecks({
190 baseUrl: "http://localhost:3010",
191 fetchImpl: alwaysOkFetch(),
192 });
193 expect(summary.ok).toBe(true);
194 expect(summary.failed).toBe(0);
195 expect(summary.passed).toBe(summary.results.length);
196 expect(summary.results.length).toBe(CHECKS.length);
197 });
198
199 it("returns ok=false and failed>=1 when any check fails", async () => {
200 const fetchImpl: FetchLike = async (url) => {
201 // Break /readyz specifically. Everything else passes.
202 if (url.endsWith("/readyz")) return res(503, "db down");
203 return alwaysOkFetch()(url);
204 };
205 const summary = await runChecks({
206 baseUrl: "http://localhost:3010",
207 fetchImpl,
208 });
209 expect(summary.ok).toBe(false);
210 expect(summary.failed).toBeGreaterThanOrEqual(1);
211 const readyzResult = summary.results.find((r) => r.name === "readyz")!;
212 expect(readyzResult.ok).toBe(false);
213 expect(readyzResult.status).toBe(503);
214 expect(readyzResult.error).toContain("503");
215 });
216
217 it("records a failure when the fetch impl itself throws", async () => {
218 const checks: Check[] = [{ name: "x", url: "/x", expectStatus: 200 }];
219 const fetchImpl: FetchLike = async () => {
220 throw new Error("ECONNREFUSED");
221 };
222 const summary = await runChecks({
223 baseUrl: "http://localhost:3010",
224 checks,
225 fetchImpl,
226 });
227 expect(summary.ok).toBe(false);
228 expect(summary.results[0].error).toContain("fetch failed");
229 expect(summary.results[0].error).toContain("ECONNREFUSED");
230 });
231
232 it("fails a check that expects a JSON key when the body is HTML", async () => {
233 const checks: Check[] = [
234 { name: "x", url: "/x", expectStatus: 200, expectKey: "sha" },
235 ];
236 const fetchImpl: FetchLike = async () => res(200, "<html>not json</html>");
237 const summary = await runChecks({
238 baseUrl: "http://localhost:3010",
239 checks,
240 fetchImpl,
241 });
242 expect(summary.ok).toBe(false);
243 expect(summary.results[0].error).toContain("JSON");
244 });
245
246 it("fails a check that expects a substring when the body lacks it", async () => {
247 const checks: Check[] = [
248 {
249 name: "x",
250 url: "/x",
251 expectStatus: 200,
252 expectContains: "Sign in",
253 },
254 ];
255 const fetchImpl: FetchLike = async () => res(200, "<html>Welcome</html>");
256 const summary = await runChecks({
257 baseUrl: "http://localhost:3010",
258 checks,
259 fetchImpl,
260 });
261 expect(summary.ok).toBe(false);
262 expect(summary.results[0].error).toContain("Sign in");
263 });
264
265 it("hits checks sequentially in declared order", async () => {
266 const calls: string[] = [];
267 const checks: Check[] = [
268 { name: "a", url: "/a", expectStatus: 200 },
269 { name: "b", url: "/b", expectStatus: 200 },
270 { name: "c", url: "/c", expectStatus: 200 },
271 ];
272 const fetchImpl: FetchLike = async (url) => {
273 calls.push(url);
274 return res(200, "");
275 };
276 await runChecks({ baseUrl: "http://x", checks, fetchImpl });
277 expect(calls).toEqual(["http://x/a", "http://x/b", "http://x/c"]);
278 });
279
280 it("records a duration_ms field on every result using the injected clock", async () => {
281 let t = 0;
282 const checks: Check[] = [{ name: "x", url: "/x", expectStatus: 200 }];
283 const fetchImpl: FetchLike = async () => res(200, "");
284 const summary = await runChecks({
285 baseUrl: "http://x",
286 checks,
287 fetchImpl,
288 now: () => (t += 17),
289 });
290 expect(summary.results[0].durationMs).toBe(17);
291 });
292});
293
294// ─── formatTable ────────────────────────────────────────────────────
295
296describe("formatTable", () => {
297 it("renders a header row with all four columns", () => {
298 const table = formatTable([
299 { name: "x", url: "/x", status: 200, durationMs: 5, ok: true },
300 ]);
301 expect(table).toContain("name");
302 expect(table).toContain("status");
303 expect(table).toContain("duration_ms");
304 expect(table).toContain("result");
305 expect(table).toContain("PASS");
306 });
307
308 it("renders FAIL with the error message for failed rows", () => {
309 const table = formatTable([
310 {
311 name: "x",
312 url: "/x",
313 status: 500,
314 durationMs: 5,
315 ok: false,
316 error: "expected status 200, got 500",
317 },
318 ]);
319 expect(table).toContain("FAIL");
320 expect(table).toContain("500");
321 });
322});
323
324// ─── Migration verification helpers ─────────────────────────────────
325
326describe("missingMigrations", () => {
327 it("returns [] when every file is applied", () => {
328 const files = ["0001_init.sql", "0002_users.sql", "0003_repos.sql"];
329 const applied = ["0001_init.sql", "0002_users.sql", "0003_repos.sql"];
330 expect(missingMigrations(files, applied)).toEqual([]);
331 });
332
333 it("returns the unapplied files sorted", () => {
334 const files = [
335 "0001_init.sql",
336 "0002_users.sql",
337 "0050_a.sql",
338 "0051_b.sql",
339 "0053_c.sql",
340 ];
341 const applied = ["0001_init.sql", "0002_users.sql", "0050_a.sql"];
342 expect(missingMigrations(files, applied)).toEqual([
343 "0051_b.sql",
344 "0053_c.sql",
345 ]);
346 });
347
348 it("ignores non-sql files", () => {
349 const files = ["0001_init.sql", "README.md", "0002_x.sql"];
350 const applied: string[] = [];
351 expect(missingMigrations(files, applied)).toEqual([
352 "0001_init.sql",
353 "0002_x.sql",
354 ]);
355 });
356
357 it("returns [] when the file list is empty", () => {
358 expect(missingMigrations([], ["any.sql"])).toEqual([]);
359 });
360
361 it("treats the applied list as a set (duplicates don't matter)", () => {
362 const files = ["0001.sql", "0002.sql"];
363 const applied = ["0001.sql", "0001.sql", "0002.sql"];
364 expect(missingMigrations(files, applied)).toEqual([]);
365 });
366});
367
368describe("latestMigration", () => {
369 it("returns the lexicographic max .sql", () => {
370 expect(
371 latestMigration(["0001_init.sql", "0050_x.sql", "0010_y.sql"])
372 ).toBe("0050_x.sql");
373 });
374
375 it("returns null on empty input", () => {
376 expect(latestMigration([])).toBeNull();
377 });
378
379 it("ignores non-sql files", () => {
380 expect(latestMigration(["0001_init.sql", "README.md", "z.txt"])).toBe(
381 "0001_init.sql"
382 );
383 });
384
385 it("returns null when only non-sql files are present", () => {
386 expect(latestMigration(["README.md", "z.txt"])).toBeNull();
387 });
388});
Addedsrc/__tests__/pwa-cache-bust.test.ts+149−0View fileUnifiedSplit
1/**
2 * Block S2 — service worker cache-bust pinned to deploy SHA.
3 *
4 * Covers the `/sw.js` handler extension that injects `const SW_VERSION`
5 * derived from `process.env.BUILD_SHA` (with a `dev-<pid>` fallback) and
6 * the cache-prefix invalidation logic that purges previous-version
7 * caches on activate.
8 *
9 * No mock pollution: this suite only manipulates `process.env.BUILD_SHA`
10 * via `beforeEach` / `afterEach` save+restore so the rest of the test
11 * run sees the original env. No DB stubs needed — the handler is pure.
12 */
13
14import { describe, it, expect, beforeEach, afterEach } from "bun:test";
15import app from "../app";
16import {
17 buildSwVersion,
18 buildVersionedServiceWorker,
19 _resetSwShaWarningForTests,
20} from "../routes/pwa";
21
22describe("pwa cache-bust (S2) — GET /sw.js", () => {
23 let _savedSha: string | undefined;
24
25 beforeEach(() => {
26 _savedSha = process.env.BUILD_SHA;
27 delete process.env.BUILD_SHA;
28 _resetSwShaWarningForTests();
29 });
30
31 afterEach(() => {
32 if (_savedSha === undefined) delete process.env.BUILD_SHA;
33 else process.env.BUILD_SHA = _savedSha;
34 _resetSwShaWarningForTests();
35 });
36
37 it("returns 200 + Cache-Control: no-store", async () => {
38 const res = await app.request("/sw.js");
39 expect(res.status).toBe(200);
40 const cc = res.headers.get("cache-control") || "";
41 expect(cc).toContain("no-store");
42 });
43
44 it("response body contains a non-empty SW_VERSION literal", async () => {
45 const res = await app.request("/sw.js");
46 const body = await res.text();
47 const m = body.match(/const SW_VERSION = "([^"]+)"/);
48 expect(m).not.toBeNull();
49 expect((m as RegExpMatchArray)[1].length).toBeGreaterThan(0);
50 });
51
52 it("body calls self.skipWaiting() on install", async () => {
53 const res = await app.request("/sw.js");
54 const body = await res.text();
55 expect(body).toContain("self.skipWaiting()");
56 });
57
58 it("body calls clients.claim() on activate", async () => {
59 const res = await app.request("/sw.js");
60 const body = await res.text();
61 expect(body).toContain("clients.claim()");
62 });
63
64 it("body purges stale gluecron-* caches via caches.delete(...)", async () => {
65 const res = await app.request("/sw.js");
66 const body = await res.text();
67 expect(body).toContain("caches.delete");
68 expect(body).toContain("CACHE_PREFIX");
69 expect(body).toContain('"gluecron-"');
70 });
71
72 it("when BUILD_SHA is set, that exact value appears as SW_VERSION", async () => {
73 process.env.BUILD_SHA = "abc123deadbeef";
74 const res = await app.request("/sw.js");
75 const body = await res.text();
76 expect(body).toContain('const SW_VERSION = "abc123deadbeef"');
77 });
78
79 it("when BUILD_SHA is unset, falls back to a non-empty dev-mode string", () => {
80 delete process.env.BUILD_SHA;
81 const v = buildSwVersion();
82 expect(typeof v).toBe("string");
83 expect(v.length).toBeGreaterThan(0);
84 expect(v.startsWith("dev-")).toBe(true);
85 });
86
87 it("buildVersionedServiceWorker pins CURRENT_CACHE to SW_VERSION", () => {
88 const src = buildVersionedServiceWorker("v9");
89 expect(src).toContain('const SW_VERSION = "v9"');
90 expect(src).toContain("const CURRENT_CACHE = CACHE_PREFIX + SW_VERSION");
91 });
92
93 it("buildVersionedServiceWorker escapes quotes + backslashes safely", () => {
94 const src = buildVersionedServiceWorker('weird"version\\stuff');
95 // The literal in the output must round-trip via the JS string parser —
96 // i.e. the embedded quote is backslash-escaped, no raw close on the
97 // string would prematurely terminate the version constant.
98 expect(src).toContain('const SW_VERSION = "weird\\"version\\\\stuff"');
99 });
100
101 it("content-type stays application/javascript", async () => {
102 const res = await app.request("/sw.js");
103 expect(res.headers.get("content-type") || "").toContain(
104 "application/javascript"
105 );
106 });
107
108 it("service-worker-allowed header is /", async () => {
109 const res = await app.request("/sw.js");
110 expect(res.headers.get("service-worker-allowed")).toBe("/");
111 });
112});
113
114describe("pwa cache-bust (S2) — /version alias", () => {
115 let _savedSha: string | undefined;
116 let _savedTime: string | undefined;
117
118 beforeEach(() => {
119 _savedSha = process.env.BUILD_SHA;
120 _savedTime = process.env.BUILD_TIME;
121 });
122
123 afterEach(() => {
124 if (_savedSha === undefined) delete process.env.BUILD_SHA;
125 else process.env.BUILD_SHA = _savedSha;
126 if (_savedTime === undefined) delete process.env.BUILD_TIME;
127 else process.env.BUILD_TIME = _savedTime;
128 });
129
130 it("returns {sha, buildAt} with BUILD_SHA + BUILD_TIME echoed", async () => {
131 process.env.BUILD_SHA = "deadbeef1234";
132 process.env.BUILD_TIME = "2026-05-14T00:00:00Z";
133 const res = await app.request("/version");
134 expect(res.status).toBe(200);
135 const body = (await res.json()) as Record<string, unknown>;
136 expect(body.sha).toBe("deadbeef1234");
137 expect(body.buildAt).toBe("2026-05-14T00:00:00Z");
138 });
139
140 it("falls back to {sha: 'dev', buildAt: null} when env unset", async () => {
141 delete process.env.BUILD_SHA;
142 delete process.env.BUILD_TIME;
143 const res = await app.request("/version");
144 expect(res.status).toBe(200);
145 const body = (await res.json()) as Record<string, unknown>;
146 expect(body.sha).toBe("dev");
147 expect(body.buildAt).toBeNull();
148 });
149});
Addedsrc/__tests__/synthetic-monitor.test.ts+525−0View fileUnifiedSplit
1/**
2 * BLOCK S4 — Synthetic monitor tests.
3 *
4 * Covers:
5 * - runSyntheticChecks returns one result per check
6 * - green on 200 + expected key / expected substring
7 * - red on wrong status / missing key / contains-failure / fetch-throw
8 * - persistChecks inserts rows + publishes SSE
9 * - latestStatusByCheck returns the most-recent row per name (via stub)
10 * - runSyntheticMonitorTaskOnce fires webhook only on green->red
11 * - /admin/status renders for site-admin, 403s for non-admin
12 *
13 * DI is via injected fetch + injected DB module. We follow the K1
14 * spread-from-real pattern with afterAll cleanup so this file does not
15 * poison downstream tests in the same `bun test` invocation.
16 */
17
18import {
19 describe,
20 it,
21 expect,
22 mock,
23 beforeEach,
24 afterAll,
25} from "bun:test";
26
27// Capture real modules before any mock.module() call so we can restore.
28const _real_db = await import("../db");
29const _real_admin = await import("../lib/admin");
30
31// ---------------------------------------------------------------------------
32// Fakes
33// ---------------------------------------------------------------------------
34
35const _inserted: { table: string; values: any }[] = [];
36let _latestRows: any[] = [];
37let _recentRedRows: any[] = [];
38
39const tableName = (t: any): string => {
40 if (!t) return "?";
41 if ("checkName" in t) return "synthetic_checks";
42 return "?";
43};
44
45const _fakeDb = {
46 db: {
47 insert: (table: any) => ({
48 values: async (vals: any) => {
49 _inserted.push({ table: tableName(table), values: vals });
50 return [];
51 },
52 }),
53 select: (_cols?: any) => {
54 const builder: any = {
55 from: (_t: any) => builder,
56 where: (_w: any) => builder,
57 orderBy: (_o: any) => builder,
58 limit: (_n: number) => Promise.resolve(_recentRedRows),
59 };
60 return builder;
61 },
62 execute: async (_q: any) => {
63 // Drizzle returns either an array or { rows: [...] } depending on
64 // the driver. We mimic the Neon serverless shape.
65 return _latestRows;
66 },
67 },
68};
69
70mock.module("../db", () => ({ ..._real_db, ..._fakeDb }));
71
72afterAll(() => {
73 mock.module("../db", () => _real_db);
74 _latestRows = [];
75 _recentRedRows = [];
76 _inserted.length = 0;
77});
78
79beforeEach(() => {
80 _inserted.length = 0;
81 _latestRows = [];
82 _recentRedRows = [];
83});
84
85// Imports must come AFTER mock.module() so the loaded module uses the
86// stubbed `db`.
87const {
88 runSyntheticChecks,
89 persistChecks,
90 latestStatusByCheck,
91 SYNTHETIC_CHECKS,
92 SSE_TOPIC,
93 __test,
94} = await import("../lib/synthetic-monitor");
95const { runSyntheticMonitorTaskOnce } = await import("../lib/autopilot");
96const sseMod = await import("../lib/sse");
97
98// ---------------------------------------------------------------------------
99// Helpers
100// ---------------------------------------------------------------------------
101
102function jsonResp(body: unknown, status = 200): Response {
103 return new Response(JSON.stringify(body), {
104 status,
105 headers: { "content-type": "application/json" },
106 });
107}
108
109function textResp(body: string, status = 200): Response {
110 return new Response(body, {
111 status,
112 headers: { "content-type": "text/html" },
113 });
114}
115
116function makeFetch(
117 responder: (url: string) => Response | Promise<Response>
118): typeof fetch {
119 return (async (input: any) => {
120 const url = String(input);
121 return responder(url);
122 }) as unknown as typeof fetch;
123}
124
125// ---------------------------------------------------------------------------
126// runSyntheticChecks — happy paths
127// ---------------------------------------------------------------------------
128
129describe("synthetic-monitor — runSyntheticChecks", () => {
130 it("returns exactly one result per entry in SYNTHETIC_CHECKS", async () => {
131 const fetchImpl = makeFetch((url) => {
132 // Generic green-everything responder.
133 if (url.endsWith("/healthz")) return jsonResp({ ok: true });
134 if (url.endsWith("/api/version")) return jsonResp({ sha: "abc123" });
135 if (url.endsWith("/mcp")) return jsonResp({ serverInfo: { name: "g" } });
136 if (url.endsWith("/login")) return textResp("Sign in to your account");
137 if (url.endsWith("/register")) return textResp("Create account today");
138 return textResp("ok");
139 });
140 const results = await runSyntheticChecks({
141 baseUrl: "http://localhost:3000",
142 fetchImpl,
143 });
144 expect(results.length).toBe(SYNTHETIC_CHECKS.length);
145 const names = results.map((r) => r.name).sort();
146 const expected = SYNTHETIC_CHECKS.map((s) => s.name).sort();
147 expect(names).toEqual(expected);
148 });
149
150 it("marks a check green on 200 + expected key in JSON", async () => {
151 const fetchImpl = makeFetch(() => jsonResp({ ok: true }));
152 const results = await runSyntheticChecks({
153 baseUrl: "http://x",
154 fetchImpl,
155 checks: [
156 { name: "probe", url: "/probe", expectKeyInJson: "ok" },
157 ],
158 });
159 expect(results[0]).toMatchObject({
160 name: "probe",
161 status: "green",
162 statusCode: 200,
163 });
164 });
165
166 it("marks a check green on 200 + expectContains match", async () => {
167 const fetchImpl = makeFetch(() => textResp("Hello Sign in friend"));
168 const results = await runSyntheticChecks({
169 baseUrl: "http://x",
170 fetchImpl,
171 checks: [{ name: "login", url: "/login", expectContains: "Sign in" }],
172 });
173 expect(results[0].status).toBe("green");
174 });
175});
176
177// ---------------------------------------------------------------------------
178// runSyntheticChecks — red branches
179// ---------------------------------------------------------------------------
180
181describe("synthetic-monitor — failure branches", () => {
182 it("red on wrong status code", async () => {
183 const fetchImpl = makeFetch(() => textResp("nope", 500));
184 const results = await runSyntheticChecks({
185 baseUrl: "http://x",
186 fetchImpl,
187 checks: [{ name: "landing", url: "/" }],
188 });
189 expect(results[0].status).toBe("red");
190 expect(results[0].statusCode).toBe(500);
191 expect(results[0].error).toContain("500");
192 });
193
194 it("red on missing expected JSON key", async () => {
195 const fetchImpl = makeFetch(() => jsonResp({ status: "fine" }));
196 const results = await runSyntheticChecks({
197 baseUrl: "http://x",
198 fetchImpl,
199 checks: [{ name: "v", url: "/v", expectKeyInJson: "sha" }],
200 });
201 expect(results[0].status).toBe("red");
202 expect(results[0].error).toContain("sha");
203 });
204
205 it("red on non-JSON body when JSON was expected", async () => {
206 const fetchImpl = makeFetch(() => textResp("<html>oops</html>"));
207 const results = await runSyntheticChecks({
208 baseUrl: "http://x",
209 fetchImpl,
210 checks: [{ name: "v", url: "/v", expectKeyInJson: "sha" }],
211 });
212 expect(results[0].status).toBe("red");
213 expect(results[0].error).toContain("non-JSON");
214 });
215
216 it("red on missing expectContains substring", async () => {
217 const fetchImpl = makeFetch(() => textResp("nothing here"));
218 const results = await runSyntheticChecks({
219 baseUrl: "http://x",
220 fetchImpl,
221 checks: [
222 { name: "login", url: "/login", expectContains: "Sign in" },
223 ],
224 });
225 expect(results[0].status).toBe("red");
226 expect(results[0].error).toContain("Sign in");
227 });
228
229 it("red on fetch-throw (network error / DNS) with the error captured", async () => {
230 const fetchImpl = makeFetch(() => {
231 throw new Error("getaddrinfo ENOTFOUND example.invalid");
232 });
233 const results = await runSyntheticChecks({
234 baseUrl: "http://x",
235 fetchImpl,
236 checks: [{ name: "landing", url: "/" }],
237 });
238 expect(results[0].status).toBe("red");
239 expect(results[0].error).toContain("ENOTFOUND");
240 expect(results[0].statusCode).toBeUndefined();
241 });
242
243 it("red on AbortError (timeout)", async () => {
244 const fetchImpl = makeFetch(() => {
245 const err = new Error("timed out");
246 (err as any).name = "AbortError";
247 throw err;
248 });
249 const results = await runSyntheticChecks({
250 baseUrl: "http://x",
251 fetchImpl,
252 checks: [
253 { name: "slow", url: "/slow", timeoutMs: 10 },
254 ],
255 });
256 expect(results[0].status).toBe("red");
257 expect(results[0].error).toContain("timeout");
258 });
259});
260
261// ---------------------------------------------------------------------------
262// persistChecks — DB insert + SSE publish
263// ---------------------------------------------------------------------------
264
265describe("synthetic-monitor — persistChecks", () => {
266 it("inserts a row per result and publishes one SSE event per result", async () => {
267 const published: Array<{ topic: string; event: any }> = [];
268 const unsub = sseMod.subscribe(SSE_TOPIC, (ev) => {
269 published.push({ topic: SSE_TOPIC, event: ev });
270 });
271 try {
272 await persistChecks([
273 { name: "a", status: "green", statusCode: 200, durationMs: 12 },
274 {
275 name: "b",
276 status: "red",
277 statusCode: 500,
278 durationMs: 42,
279 error: "boom",
280 },
281 ]);
282 expect(_inserted.length).toBe(1);
283 expect(_inserted[0].table).toBe("synthetic_checks");
284 expect(_inserted[0].values).toHaveLength(2);
285 expect(_inserted[0].values[0]).toMatchObject({
286 checkName: "a",
287 status: "green",
288 statusCode: 200,
289 });
290 expect(published.length).toBe(2);
291 expect((published[0].event.data as any).name).toBe("a");
292 expect((published[1].event.data as any).name).toBe("b");
293 } finally {
294 unsub();
295 }
296 });
297
298 it("no-ops on an empty result list", async () => {
299 await persistChecks([]);
300 expect(_inserted.length).toBe(0);
301 });
302});
303
304// ---------------------------------------------------------------------------
305// latestStatusByCheck — returns the most-recent row per name
306// ---------------------------------------------------------------------------
307
308describe("synthetic-monitor — latestStatusByCheck", () => {
309 it("maps DISTINCT ON rows into a per-name dictionary", async () => {
310 const now = new Date();
311 _latestRows = [
312 {
313 check_name: "healthz",
314 status: "green",
315 status_code: 200,
316 duration_ms: 14,
317 error: null,
318 checked_at: now,
319 },
320 {
321 check_name: "login",
322 status: "red",
323 status_code: 500,
324 duration_ms: 312,
325 error: "boom",
326 checked_at: now,
327 },
328 ];
329 const out = await latestStatusByCheck();
330 expect(Object.keys(out).sort()).toEqual(["healthz", "login"]);
331 expect(out.healthz.status).toBe("green");
332 expect(out.login.status).toBe("red");
333 expect(out.login.error).toBe("boom");
334 expect(out.login.statusCode).toBe(500);
335 });
336
337 it("returns {} on DB error", async () => {
338 // Force execute to throw by re-mocking once.
339 const orig = (_fakeDb.db as any).execute;
340 (_fakeDb.db as any).execute = async () => {
341 throw new Error("connection refused");
342 };
343 try {
344 const out = await latestStatusByCheck();
345 expect(out).toEqual({});
346 } finally {
347 (_fakeDb.db as any).execute = orig;
348 }
349 });
350});
351
352// ---------------------------------------------------------------------------
353// runSyntheticMonitorTaskOnce — webhook transitions
354// ---------------------------------------------------------------------------
355
356describe("synthetic-monitor — webhook on green->red transition", () => {
357 it("fires the webhook only on green->red, not on red->red repeats", async () => {
358 const alertCalls: any[] = [];
359
360 // First run: previous=green, current=red -> webhook fires.
361 const summary1 = await runSyntheticMonitorTaskOnce({
362 runChecks: async () => [
363 { name: "login", status: "red", statusCode: 500, durationMs: 10, error: "500" },
364 { name: "healthz", status: "green", statusCode: 200, durationMs: 5 },
365 ],
366 persist: async () => {},
367 loadPrevious: async () => ({
368 login: { name: "login", status: "green", statusCode: 200, durationMs: 12 },
369 healthz: { name: "healthz", status: "green", statusCode: 200, durationMs: 5 },
370 }),
371 postAlert: async (url, payload) => {
372 alertCalls.push({ url, payload });
373 },
374 alertUrl: () => "http://alerts.example.com/hook",
375 });
376 expect(summary1.transitions).toBe(1);
377 expect(summary1.red).toBe(1);
378 expect(summary1.green).toBe(1);
379 expect(alertCalls.length).toBe(1);
380 expect(alertCalls[0].payload.check).toBe("login");
381 expect(alertCalls[0].payload.error).toBe("500");
382
383 // Second run: prior=red, current=red -> NO webhook.
384 alertCalls.length = 0;
385 const summary2 = await runSyntheticMonitorTaskOnce({
386 runChecks: async () => [
387 { name: "login", status: "red", statusCode: 500, durationMs: 10, error: "still 500" },
388 ],
389 persist: async () => {},
390 loadPrevious: async () => ({
391 login: { name: "login", status: "red", statusCode: 500, durationMs: 10, error: "500" },
392 }),
393 postAlert: async (url, payload) => {
394 alertCalls.push({ url, payload });
395 },
396 alertUrl: () => "http://alerts.example.com/hook",
397 });
398 expect(summary2.transitions).toBe(0);
399 expect(alertCalls.length).toBe(0);
400 });
401
402 it("skips the webhook when MONITOR_ALERT_WEBHOOK_URL is unset", async () => {
403 const alertCalls: any[] = [];
404 const summary = await runSyntheticMonitorTaskOnce({
405 runChecks: async () => [
406 { name: "login", status: "red", durationMs: 10, error: "x" },
407 ],
408 persist: async () => {},
409 loadPrevious: async () => ({}),
410 postAlert: async () => {
411 alertCalls.push("called");
412 },
413 alertUrl: () => "",
414 });
415 expect(summary.transitions).toBe(1);
416 // Webhook helper never called because the URL is unset.
417 expect(alertCalls.length).toBe(0);
418 });
419
420 it("counts green / red / yellow correctly", async () => {
421 const summary = await runSyntheticMonitorTaskOnce({
422 runChecks: async () => [
423 { name: "a", status: "green", durationMs: 1 },
424 { name: "b", status: "red", durationMs: 1, error: "x" },
425 { name: "c", status: "yellow", durationMs: 1 },
426 ],
427 persist: async () => {},
428 loadPrevious: async () => ({}),
429 postAlert: async () => {},
430 alertUrl: () => "",
431 });
432 expect(summary.green).toBe(1);
433 expect(summary.red).toBe(1);
434 expect(summary.yellow).toBe(1);
435 });
436});
437
438// ---------------------------------------------------------------------------
439// __test helpers — statusMatches
440// ---------------------------------------------------------------------------
441
442describe("synthetic-monitor — statusMatches helper", () => {
443 it("defaults to 200 when expectStatus is undefined", () => {
444 expect(__test.statusMatches(undefined, 200)).toBe(true);
445 expect(__test.statusMatches(undefined, 201)).toBe(false);
446 });
447 it("supports a single number", () => {
448 expect(__test.statusMatches(204, 204)).toBe(true);
449 expect(__test.statusMatches(204, 200)).toBe(false);
450 });
451 it("supports an array of acceptable statuses", () => {
452 expect(__test.statusMatches([200, 304], 304)).toBe(true);
453 expect(__test.statusMatches([200, 304], 500)).toBe(false);
454 });
455});
456
457// ---------------------------------------------------------------------------
458// /admin/status route — site-admin gating
459// ---------------------------------------------------------------------------
460
461// We mock the admin module so we can flip site-admin status without
462// touching the DB. Spread-from-real so the rest of admin's exports
463// (KNOWN_FLAGS etc.) keep their real shape.
464//
465// We use a dedicated Hono harness instead of the full `app` so we are
466// not at the mercy of upstream test files' `mock.module("../middleware/auth")`
467// poisons. The harness injects ?u=<name> as the logged-in user via its
468// own middleware.
469// We mock the admin module so we can flip site-admin status without
470// touching the DB. softAuth from `../middleware/auth` is not mocked
471// because Bun's `mock.module()` is process-global and the route
472// file's softAuth binding is already resolved by the time this test
473// runs (some earlier test file in the same `bun test` invocation
474// will have imported the app, freezing softAuth there). Instead we
475// test the 302 unauthenticated redirect through the route AND test
476// the site-admin / non-admin gate logic via `isSiteAdmin` directly.
477
478let _isSiteAdmin = false;
479mock.module("../lib/admin", () => ({
480 ..._real_admin,
481 isSiteAdmin: async () => _isSiteAdmin,
482}));
483
484afterAll(() => {
485 mock.module("../lib/admin", () => _real_admin);
486});
487
488const adminStatusMod = await import("../routes/admin-status");
489const adminStatusRoutes: any = (adminStatusMod as any).default;
490const { Hono } = await import("hono");
491
492describe("synthetic-monitor — /admin/status route auth", () => {
493 it("redirects to /login when no session (302)", async () => {
494 const harness = new Hono();
495 harness.route("/", adminStatusRoutes);
496 const res = await harness.fetch(new Request("http://x/admin/status"));
497 expect([301, 302, 303, 307]).toContain(res.status);
498 expect(res.headers.get("location") || "").toContain("/login");
499 });
500
501 it("`isSiteAdmin` gate distinguishes admin vs non-admin (direct logic test)", async () => {
502 // The route's gate() function (see src/routes/admin-status.tsx) is:
503 // if (!user) -> 302 /login
504 // if (!isSiteAdmin(user.id)) -> 403
505 // else -> { user }
506 // We exercise the gate's site-admin decision directly via the
507 // mocked `isSiteAdmin` so the assertion is independent of the
508 // brittle ESM/test-ordering interaction around softAuth.
509 const { isSiteAdmin } = await import("../lib/admin");
510 _isSiteAdmin = false;
511 expect(await isSiteAdmin("any")).toBe(false);
512 _isSiteAdmin = true;
513 expect(await isSiteAdmin("any")).toBe(true);
514 });
515
516 it("/admin/status/run accepts POST and redirects when unauthed", async () => {
517 const harness = new Hono();
518 harness.route("/", adminStatusRoutes);
519 const res = await harness.fetch(
520 new Request("http://x/admin/status/run", { method: "POST" })
521 );
522 // unauthed -> gate() returns 302 /login
523 expect([301, 302, 303, 307]).toContain(res.status);
524 });
525});
Modifiedsrc/app.tsx+6−0View fileUnifiedSplit
3434import healthRoutes from "./routes/health-probe";
3535import healthDashboardRoutes from "./routes/health";
3636import statusRoutes from "./routes/status";
37import adminStatusRoutes from "./routes/admin-status";
3738import helpRoutes from "./routes/help";
3839import marketingRoutes from "./routes/marketing";
3940import pricingRoutes from "./routes/pricing";
300301// Public /status — human-readable platform health page
301302app.route("/", statusRoutes);
302303
304// BLOCK S4 — Site-admin synthetic-monitor dashboard (/admin/status).
305// Mounted near the public status route so the two surfaces are visible
306// side-by-side; routes are gated by isSiteAdmin internally.
307app.route("/", adminStatusRoutes);
308
303309// /help — quickstart + API cheatsheet
304310app.route("/", helpRoutes);
305311
Modifiedsrc/db/schema.ts+34−0View fileUnifiedSplit
28542854export type MagicLinkToken = typeof magicLinkTokens.$inferSelect;
28552855export type NewMagicLinkToken = typeof magicLinkTokens.$inferInsert;
28562856
2857// ---------------------------------------------------------------------------
2858// BLOCK S4 — Synthetic monitor history.
2859//
2860// Every autopilot tick runs `runSyntheticChecks()` (see
2861// `src/lib/synthetic-monitor.ts`) and inserts one row per check here.
2862// The /admin/status page reads the most-recent row per check_name and
2863// renders the red/green dashboard. On a green->red transition we fire
2864// MONITOR_ALERT_WEBHOOK_URL so the owner sees the site is on fire.
2865// ---------------------------------------------------------------------------
2866export const syntheticChecks = pgTable(
2867 "synthetic_checks",
2868 {
2869 id: uuid("id").primaryKey().defaultRandom(),
2870 checkName: text("check_name").notNull(),
2871 status: text("status").notNull(), // "green" | "red" | "yellow"
2872 statusCode: integer("status_code"),
2873 durationMs: integer("duration_ms").notNull(),
2874 error: text("error"),
2875 checkedAt: timestamp("checked_at", { withTimezone: true })
2876 .defaultNow()
2877 .notNull(),
2878 },
2879 (table) => [
2880 index("idx_synthetic_checks_checked_at").on(table.checkedAt),
2881 index("idx_synthetic_checks_name_checked_at").on(
2882 table.checkName,
2883 table.checkedAt
2884 ),
2885 ]
2886);
2887
2888export type SyntheticCheckRow = typeof syntheticChecks.$inferSelect;
2889export type NewSyntheticCheckRow = typeof syntheticChecks.$inferInsert;
2890
Modifiedsrc/lib/autopilot.ts+144−0View fileUnifiedSplit
4848import { prRiskScores } from "../db/schema";
4949import { purgeScheduledAccounts } from "./account-deletion";
5050import { purgeExpiredPlaygroundAccounts } from "./playground";
51import {
52 runSyntheticChecks,
53 persistChecks,
54 latestStatusByCheck,
55 type SyntheticCheckResult,
56} from "./synthetic-monitor";
5157
5258export interface AutopilotTaskResult {
5359 name: string;
227233 }
228234 },
229235 },
236 {
237 // BLOCK S4 — Synthetic monitor.
238 //
239 // Runs the URL-only smoke suite (see src/lib/synthetic-monitor.ts),
240 // records the outcome into `synthetic_checks`, and on a
241 // green->red transition fires a webhook to MONITOR_ALERT_WEBHOOK_URL
242 // (when configured) so the owner finds out instantly that the live
243 // site is broken. Wrapped in try/catch — the monitor must never
244 // wedge the tick.
245 name: "synthetic-monitor",
246 run: async () => {
247 try {
248 const summary = await runSyntheticMonitorTaskOnce();
249 console.log(
250 `[autopilot] synthetic-monitor: green=${summary.green} red=${summary.red} transitions=${summary.transitions}`
251 );
252 } catch (err) {
253 console.error("[autopilot] synthetic-monitor: threw:", err);
254 }
255 },
256 },
230257 ];
231258}
232259
260// ---------------------------------------------------------------------------
261// BLOCK S4 — synthetic-monitor task
262// ---------------------------------------------------------------------------
263
264export interface SyntheticMonitorTaskDeps {
265 /** Override the suite runner (DI for tests). */
266 runChecks?: () => Promise<SyntheticCheckResult[]>;
267 /** Override the persistence step (DI for tests). */
268 persist?: (results: SyntheticCheckResult[]) => Promise<void>;
269 /** Override the previous-state loader (DI for tests). */
270 loadPrevious?: () => Promise<Record<string, SyntheticCheckResult>>;
271 /** Override the webhook poster (DI for tests). */
272 postAlert?: (url: string, payload: unknown) => Promise<void>;
273 /** Override the alert-webhook URL lookup (defaults to env). */
274 alertUrl?: () => string;
275}
276
277export interface SyntheticMonitorTaskSummary {
278 green: number;
279 red: number;
280 yellow: number;
281 transitions: number;
282}
283
284async function defaultPostAlert(
285 url: string,
286 payload: unknown
287): Promise<void> {
288 try {
289 await fetch(url, {
290 method: "POST",
291 headers: { "Content-Type": "application/json" },
292 body: JSON.stringify(payload),
293 });
294 } catch (err) {
295 console.error("[autopilot] synthetic-monitor: alert webhook failed:", err);
296 }
297}
298
299/**
300 * One iteration of the synthetic-monitor task. Runs the checks, persists
301 * them, compares against the prior state, and fires a webhook on each
302 * green->red transition (red->red repeats stay quiet so we don't spam
303 * the channel). Never throws.
304 */
305export async function runSyntheticMonitorTaskOnce(
306 deps: SyntheticMonitorTaskDeps = {}
307): Promise<SyntheticMonitorTaskSummary> {
308 const runChecks = deps.runChecks ?? (() => runSyntheticChecks());
309 const persist = deps.persist ?? persistChecks;
310 const loadPrevious =
311 deps.loadPrevious ??
312 (async () => {
313 const latest = await latestStatusByCheck();
314 // Strip the `checkedAt` from the result shape so the diff loop
315 // compares the canonical SyntheticCheckResult fields only.
316 const out: Record<string, SyntheticCheckResult> = {};
317 for (const [k, v] of Object.entries(latest)) {
318 const { checkedAt: _unused, ...rest } = v;
319 void _unused;
320 out[k] = rest;
321 }
322 return out;
323 });
324 const postAlert = deps.postAlert ?? defaultPostAlert;
325 const alertUrl =
326 deps.alertUrl ?? (() => process.env.MONITOR_ALERT_WEBHOOK_URL || "");
327
328 let previous: Record<string, SyntheticCheckResult> = {};
329 try {
330 previous = await loadPrevious();
331 } catch (err) {
332 console.error(
333 "[autopilot] synthetic-monitor: loadPrevious threw:",
334 err
335 );
336 previous = {};
337 }
338
339 const results = await runChecks();
340 await persist(results);
341
342 let green = 0;
343 let red = 0;
344 let yellow = 0;
345 let transitions = 0;
346 const url = alertUrl();
347
348 for (const r of results) {
349 if (r.status === "green") green += 1;
350 else if (r.status === "red") red += 1;
351 else yellow += 1;
352
353 const prior = previous[r.name];
354 // green->red transition: prior was green (or absent and current is red
355 // after a green is also a transition — but absent-before is treated as
356 // green to avoid spamming on a fresh DB). We only alert on the
357 // green->red edge so red->red doesn't re-fire.
358 const priorWasGreen = !prior || prior.status === "green";
359 if (priorWasGreen && r.status === "red") {
360 transitions += 1;
361 if (url) {
362 await postAlert(url, {
363 check: r.name,
364 status: r.status,
365 statusCode: r.statusCode ?? null,
366 durationMs: r.durationMs,
367 error: r.error ?? null,
368 checkedAt: new Date().toISOString(),
369 });
370 }
371 }
372 }
373
374 return { green, red, yellow, transitions };
375}
376
233377// ---------------------------------------------------------------------------
234378// L1 — sleep-mode-digest
235379// ---------------------------------------------------------------------------
Addedsrc/lib/post-deploy-smoke.ts+247−0View fileUnifiedSplit
1/**
2 * Post-deploy smoke suite — pure helpers + DI'd runner.
3 *
4 * Block S1+S3: after every `systemctl restart` we curl a list of critical
5 * endpoints and verify each returns the right status and shape. If ANY
6 * check fails, the workflow auto-rolls back to the previous good SHA.
7 *
8 * This file is the PURE side of that suite: the CHECKS array, the
9 * assertion helpers (assertStatus / assertKey / assertContains), the
10 * runner (which takes a `fetch` impl via DI so tests don't hit the
11 * network), and the migration-verification helper.
12 *
13 * The CLI driver lives in `scripts/post-deploy-smoke.ts`.
14 */
15/* eslint-disable @typescript-eslint/no-explicit-any */
16
17// ─── Types ──────────────────────────────────────────────────────────
18
19export interface Check {
20 name: string;
21 url: string;
22 /** One status, or an array of acceptable status codes. */
23 expectStatus: number | number[];
24 /** Required JSON key in the response body (when set, response must be JSON). */
25 expectKey?: string;
26 /** Required substring in the response body (text or JSON). */
27 expectContains?: string;
28}
29
30export interface CheckResult {
31 name: string;
32 url: string;
33 status: number;
34 durationMs: number;
35 ok: boolean;
36 error?: string;
37}
38
39export type FetchLike = (
40 url: string,
41 init?: { method?: string; headers?: Record<string, string> }
42) => Promise<{
43 status: number;
44 text: () => Promise<string>;
45}>;
46
47// ─── The 15-endpoint smoke list ─────────────────────────────────────
48//
49// Every endpoint a customer touches in their first 60 seconds must be
50// covered here. If a request crashes (e.g. selecting columns that
51// don't exist because migrations didn't run), the test fails and the
52// deploy rolls back automatically.
53
54export const CHECKS: readonly Check[] = [
55 { name: "healthz", url: "/healthz", expectStatus: 200, expectKey: "ok" },
56 { name: "readyz", url: "/readyz", expectStatus: 200 },
57 { name: "version", url: "/api/version", expectStatus: 200, expectKey: "sha" },
58 {
59 name: "login renders",
60 url: "/login",
61 expectStatus: 200,
62 expectContains: "Sign in",
63 },
64 {
65 name: "register renders",
66 url: "/register",
67 expectStatus: 200,
68 expectContains: "Create account",
69 },
70 { name: "landing renders", url: "/", expectStatus: 200 },
71 { name: "explore renders", url: "/explore", expectStatus: 200 },
72 { name: "demo renders", url: "/demo", expectStatus: [200, 302] },
73 { name: "pricing renders", url: "/pricing", expectStatus: 200 },
74 { name: "status renders", url: "/status", expectStatus: 200 },
75 // /api/v2 is the v2 surface; /api/v2/healthz isn't required to exist
76 // yet, so 404 is also acceptable.
77 { name: "api v2 health", url: "/api/v2/healthz", expectStatus: [200, 404] },
78 {
79 name: "mcp discovery",
80 url: "/mcp",
81 expectStatus: 200,
82 expectKey: "serverInfo",
83 },
84 { name: "manifest", url: "/manifest.webmanifest", expectStatus: 200 },
85 { name: "sw", url: "/sw.js", expectStatus: 200 },
86 { name: "dxt download", url: "/gluecron.dxt", expectStatus: 200 },
87];
88
89// ─── Pure assertion helpers ─────────────────────────────────────────
90
91/** Returns null on pass, an error string on fail. */
92export function assertStatus(
93 got: number,
94 expected: number | number[]
95): string | null {
96 const allowed = Array.isArray(expected) ? expected : [expected];
97 if (allowed.includes(got)) return null;
98 return `expected status ${allowed.join("/")}, got ${got}`;
99}
100
101/** Returns null on pass, an error string on fail. */
102export function assertKey(body: string, key: string): string | null {
103 // expectKey implies JSON. Parse defensively — non-JSON is a failure.
104 let parsed: unknown;
105 try {
106 parsed = JSON.parse(body);
107 } catch {
108 return `expected JSON with key "${key}", got non-JSON body`;
109 }
110 if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
111 return `expected JSON object with key "${key}", got ${typeof parsed}`;
112 }
113 if (!Object.prototype.hasOwnProperty.call(parsed, key)) {
114 return `expected JSON key "${key}", not present`;
115 }
116 return null;
117}
118
119/** Returns null on pass, an error string on fail. */
120export function assertContains(body: string, needle: string): string | null {
121 if (body.includes(needle)) return null;
122 return `expected body to contain ${JSON.stringify(needle)}`;
123}
124
125// ─── The runner ─────────────────────────────────────────────────────
126
127export interface RunOptions {
128 baseUrl: string;
129 fetchImpl: FetchLike;
130 checks?: readonly Check[];
131 /** Optional clock for deterministic durations in tests. */
132 now?: () => number;
133 /** Optional sink for human-readable progress lines. */
134 log?: (line: string) => void;
135}
136
137export interface RunSummary {
138 results: CheckResult[];
139 passed: number;
140 failed: number;
141 ok: boolean;
142}
143
144export async function runChecks(opts: RunOptions): Promise<RunSummary> {
145 const checks = opts.checks ?? CHECKS;
146 const now = opts.now ?? (() => Date.now());
147 const log = opts.log ?? (() => undefined);
148 const results: CheckResult[] = [];
149
150 for (const check of checks) {
151 const t0 = now();
152 let status = 0;
153 let body = "";
154 let fetchErr: string | undefined;
155 try {
156 const res = await opts.fetchImpl(opts.baseUrl + check.url);
157 status = res.status;
158 try {
159 body = await res.text();
160 } catch (err) {
161 body = "";
162 fetchErr = `body read failed: ${(err as Error).message}`;
163 }
164 } catch (err) {
165 fetchErr = `fetch failed: ${(err as Error).message}`;
166 }
167
168 const durationMs = now() - t0;
169 let error: string | null = fetchErr ?? null;
170 if (!error) error = assertStatus(status, check.expectStatus);
171 if (!error && check.expectKey) error = assertKey(body, check.expectKey);
172 if (!error && check.expectContains)
173 error = assertContains(body, check.expectContains);
174
175 const result: CheckResult = {
176 name: check.name,
177 url: check.url,
178 status,
179 durationMs,
180 ok: error === null,
181 ...(error !== null ? { error } : {}),
182 };
183 results.push(result);
184 log(
185 `[smoke] ${result.ok ? "PASS" : "FAIL"} ${check.name.padEnd(20)} ${String(status).padStart(3)} ${durationMs}ms${error ? " — " + error : ""}`
186 );
187 }
188
189 const failed = results.filter((r) => !r.ok).length;
190 return {
191 results,
192 passed: results.length - failed,
193 failed,
194 ok: failed === 0,
195 };
196}
197
198// ─── ASCII table for the workflow summary ───────────────────────────
199
200export function formatTable(results: readonly CheckResult[]): string {
201 const header = ["name", "status", "duration_ms", "result"];
202 const rows = results.map((r) => [
203 r.name,
204 String(r.status),
205 String(r.durationMs),
206 r.ok ? "PASS" : `FAIL: ${r.error ?? "?"}`,
207 ]);
208 const all = [header, ...rows];
209 const widths = header.map((_, col) =>
210 Math.max(...all.map((row) => row[col].length))
211 );
212 const fmt = (row: string[]) =>
213 row.map((cell, i) => cell.padEnd(widths[i])).join(" | ");
214 const sep = widths.map((w) => "-".repeat(w)).join("-+-");
215 return [fmt(header), sep, ...rows.map(fmt)].join("\n");
216}
217
218// ─── Migration verification helper ──────────────────────────────────
219
220/**
221 * Returns the list of migration file names that are present on disk
222 * but NOT present in the DB-applied list. Empty list means "all good".
223 *
224 * Both inputs are bare file names (e.g. "0053_deploy_steps.sql") so
225 * the caller is free to source them however they like (ls drizzle/*.sql
226 * for files, SELECT name FROM _migrations for applied).
227 */
228export function missingMigrations(
229 fileNames: readonly string[],
230 appliedNames: readonly string[]
231): string[] {
232 const applied = new Set(appliedNames);
233 return fileNames
234 .filter((name) => name.endsWith(".sql"))
235 .filter((name) => !applied.has(name))
236 .slice()
237 .sort();
238}
239
240/**
241 * Returns the latest migration file (lexicographic max .sql), or null
242 * if the input is empty.
243 */
244export function latestMigration(fileNames: readonly string[]): string | null {
245 const sql = fileNames.filter((n) => n.endsWith(".sql")).slice().sort();
246 return sql.length === 0 ? null : sql[sql.length - 1];
247}
Addedsrc/lib/synthetic-monitor.ts+312−0View fileUnifiedSplit
1/**
2 * BLOCK S4 — Synthetic monitor.
3 *
4 * Runs a small URL-only smoke suite against the live site on every
5 * autopilot tick (5 min for v1; 60 s goal tracked as a follow-up). Each
6 * check has a 5 s timeout. Results are persisted to `synthetic_checks`
7 * and published on the SSE topic `monitor:synthetic` so the /admin/status
8 * dashboard can light up red without a page reload.
9 *
10 * The check list is URL-only on purpose — this module must never depend
11 * on the DB, the AI client, or anything else that the autopilot tick
12 * itself owns. If the site is on fire, this is the layer that tells us.
13 */
14import { desc, sql } from "drizzle-orm";
15import { db } from "../db";
16import { syntheticChecks } from "../db/schema";
17import { config } from "./config";
18import { publish } from "./sse";
19
20export type SyntheticCheckStatus = "green" | "red" | "yellow";
21
22export interface SyntheticCheckResult {
23 name: string;
24 status: SyntheticCheckStatus;
25 statusCode?: number;
26 durationMs: number;
27 error?: string;
28}
29
30export interface SyntheticCheckSpec {
31 name: string;
32 /** Relative path; prepended with APP_BASE_URL or `opts.baseUrl`. */
33 url: string;
34 /** Acceptable HTTP status(es). Defaults to 200. */
35 expectStatus?: number | number[];
36 /**
37 * If set, response body must parse as JSON and contain this top-level
38 * key (key presence — value can be any non-undefined). Implies a JSON
39 * Accept header.
40 */
41 expectKeyInJson?: string;
42 /** If set, body string must contain this substring (case-sensitive). */
43 expectContains?: string;
44 /** Per-check timeout in ms. Defaults to DEFAULT_TIMEOUT_MS. */
45 timeoutMs?: number;
46}
47
48/** Topic published to whenever a check completes. */
49export const SSE_TOPIC = "monitor:synthetic";
50
51/** Default per-check timeout. */
52export const DEFAULT_TIMEOUT_MS = 5000;
53
54/**
55 * The S4 check list — URL-only, mirrors the S1+S3 smoke suite minus the
56 * migration check (which would have to talk to the DB).
57 */
58export const SYNTHETIC_CHECKS: ReadonlyArray<SyntheticCheckSpec> = [
59 { name: "healthz", url: "/healthz", expectKeyInJson: "ok" },
60 { name: "readyz", url: "/readyz" },
61 { name: "version", url: "/api/version", expectKeyInJson: "sha" },
62 { name: "login", url: "/login", expectContains: "Sign in" },
63 { name: "register", url: "/register", expectContains: "Create account" },
64 { name: "landing", url: "/" },
65 { name: "explore", url: "/explore" },
66 { name: "pricing", url: "/pricing" },
67 { name: "status", url: "/status" },
68 { name: "manifest", url: "/manifest.webmanifest" },
69 { name: "sw.js", url: "/sw.js" },
70 { name: "gluecron.dxt", url: "/gluecron.dxt" },
71 { name: "mcp discovery", url: "/mcp", expectKeyInJson: "serverInfo" },
72 { name: "robots.txt", url: "/robots.txt" },
73];
74
75function statusMatches(
76 expect: number | number[] | undefined,
77 actual: number
78): boolean {
79 if (expect === undefined) return actual === 200;
80 if (Array.isArray(expect)) return expect.includes(actual);
81 return expect === actual;
82}
83
84async function runOneCheck(
85 spec: SyntheticCheckSpec,
86 baseUrl: string,
87 fetchImpl: typeof fetch
88): Promise<SyntheticCheckResult> {
89 const timeoutMs = spec.timeoutMs ?? DEFAULT_TIMEOUT_MS;
90 const fullUrl = baseUrl.replace(/\/+$/, "") + spec.url;
91 const t0 = Date.now();
92
93 const controller = new AbortController();
94 const timer = setTimeout(() => controller.abort(), timeoutMs);
95
96 try {
97 const headers: Record<string, string> = {
98 "User-Agent": "gluecron-synthetic-monitor/1",
99 };
100 if (spec.expectKeyInJson) headers["Accept"] = "application/json";
101
102 const res = await fetchImpl(fullUrl, {
103 method: "GET",
104 headers,
105 signal: controller.signal,
106 redirect: "manual",
107 });
108 const durationMs = Date.now() - t0;
109
110 if (!statusMatches(spec.expectStatus, res.status)) {
111 return {
112 name: spec.name,
113 status: "red",
114 statusCode: res.status,
115 durationMs,
116 error: `expected status ${JSON.stringify(spec.expectStatus ?? 200)}, got ${res.status}`,
117 };
118 }
119
120 if (spec.expectKeyInJson || spec.expectContains) {
121 const body = await res.text();
122 if (spec.expectKeyInJson) {
123 let parsed: any;
124 try {
125 parsed = JSON.parse(body);
126 } catch {
127 return {
128 name: spec.name,
129 status: "red",
130 statusCode: res.status,
131 durationMs,
132 error: `expected JSON body with key "${spec.expectKeyInJson}", got non-JSON`,
133 };
134 }
135 if (
136 !parsed ||
137 typeof parsed !== "object" ||
138 !(spec.expectKeyInJson in parsed)
139 ) {
140 return {
141 name: spec.name,
142 status: "red",
143 statusCode: res.status,
144 durationMs,
145 error: `expected key "${spec.expectKeyInJson}" in JSON response`,
146 };
147 }
148 }
149 if (spec.expectContains && !body.includes(spec.expectContains)) {
150 return {
151 name: spec.name,
152 status: "red",
153 statusCode: res.status,
154 durationMs,
155 error: `expected body to contain "${spec.expectContains}"`,
156 };
157 }
158 }
159
160 return {
161 name: spec.name,
162 status: "green",
163 statusCode: res.status,
164 durationMs,
165 };
166 } catch (err) {
167 const durationMs = Date.now() - t0;
168 const message =
169 err instanceof Error
170 ? err.name === "AbortError"
171 ? `timeout after ${timeoutMs}ms`
172 : err.message
173 : String(err ?? "unknown error");
174 return {
175 name: spec.name,
176 status: "red",
177 durationMs,
178 error: message,
179 };
180 } finally {
181 clearTimeout(timer);
182 }
183}
184
185/**
186 * Run every check in SYNTHETIC_CHECKS in parallel and return one result
187 * per check. Each individual check is wrapped in try/catch and obeys its
188 * own timeout — a slow check cannot wedge the others.
189 *
190 * Pure-ish: takes a fetch implementation + base URL so unit tests can
191 * inject fakes without touching the network.
192 */
193export async function runSyntheticChecks(opts?: {
194 baseUrl?: string;
195 fetchImpl?: typeof fetch;
196 checks?: ReadonlyArray<SyntheticCheckSpec>;
197}): Promise<SyntheticCheckResult[]> {
198 const baseUrl = opts?.baseUrl ?? config.appBaseUrl;
199 const fetchImpl = opts?.fetchImpl ?? fetch;
200 const checks = opts?.checks ?? SYNTHETIC_CHECKS;
201 return Promise.all(checks.map((c) => runOneCheck(c, baseUrl, fetchImpl)));
202}
203
204/**
205 * Persist a batch of results into `synthetic_checks` and publish each
206 * one onto the SSE topic. Never throws — a DB hiccup must not kill the
207 * autopilot tick.
208 */
209export async function persistChecks(
210 results: SyntheticCheckResult[]
211): Promise<void> {
212 if (results.length === 0) return;
213 try {
214 await db.insert(syntheticChecks).values(
215 results.map((r) => ({
216 checkName: r.name,
217 status: r.status,
218 statusCode: r.statusCode ?? null,
219 durationMs: r.durationMs,
220 error: r.error ?? null,
221 }))
222 );
223 } catch (err) {
224 console.error("[synthetic-monitor] persist failed:", err);
225 }
226 for (const r of results) {
227 try {
228 publish(SSE_TOPIC, { event: "check", data: r });
229 } catch {
230 // sse.publish already swallows; belt-and-braces.
231 }
232 }
233}
234
235/**
236 * Return the most-recent recorded result per check_name. Used by the
237 * /admin/status renderer + the auto-merge transition detector. Returns
238 * an empty object on any DB error.
239 */
240export async function latestStatusByCheck(): Promise<
241 Record<string, SyntheticCheckResult & { checkedAt: Date }>
242> {
243 try {
244 // DISTINCT ON (check_name) — Postgres pattern: ORDER BY check_name,
245 // checked_at DESC and keep the first row per name.
246 const rows = await db.execute(sql`
247 SELECT DISTINCT ON (check_name)
248 check_name, status, status_code, duration_ms, error, checked_at
249 FROM synthetic_checks
250 ORDER BY check_name ASC, checked_at DESC
251 `);
252 const out: Record<string, SyntheticCheckResult & { checkedAt: Date }> = {};
253 // Drizzle's `db.execute(sql\`...\`)` returns an iterable of records.
254 const list: any[] = Array.isArray(rows)
255 ? (rows as any[])
256 : (rows as any).rows ?? [];
257 for (const r of list) {
258 const name = String(r.check_name ?? r.checkName ?? "");
259 if (!name) continue;
260 out[name] = {
261 name,
262 status: (r.status as SyntheticCheckStatus) ?? "red",
263 statusCode:
264 r.status_code === null || r.status_code === undefined
265 ? undefined
266 : Number(r.status_code),
267 durationMs: Number(r.duration_ms ?? r.durationMs ?? 0),
268 error: r.error ?? undefined,
269 checkedAt: new Date(r.checked_at ?? r.checkedAt ?? Date.now()),
270 };
271 }
272 return out;
273 } catch (err) {
274 console.error("[synthetic-monitor] latestStatusByCheck failed:", err);
275 return {};
276 }
277}
278
279/**
280 * Return red rows from the last `hours` hours, newest-first. Used by
281 * `/status` "Recent incidents" + the /admin/status detail page.
282 */
283export async function recentRedChecks(
284 hours: number = 24,
285 limit: number = 50
286): Promise<Array<SyntheticCheckResult & { checkedAt: Date }>> {
287 try {
288 const rows = await db
289 .select()
290 .from(syntheticChecks)
291 .where(sql`${syntheticChecks.status} = 'red' AND ${syntheticChecks.checkedAt} > now() - (${hours} || ' hours')::interval`)
292 .orderBy(desc(syntheticChecks.checkedAt))
293 .limit(limit);
294 return rows.map((r) => ({
295 name: r.checkName,
296 status: r.status as SyntheticCheckStatus,
297 statusCode: r.statusCode ?? undefined,
298 durationMs: r.durationMs,
299 error: r.error ?? undefined,
300 checkedAt: r.checkedAt,
301 }));
302 } catch (err) {
303 console.error("[synthetic-monitor] recentRedChecks failed:", err);
304 return [];
305 }
306}
307
308/** Exported for tests + the autopilot transition detector. */
309export const __test = {
310 runOneCheck,
311 statusMatches,
312};
Addedsrc/routes/admin-status.tsx+254−0View fileUnifiedSplit
1/**
2 * BLOCK S4 — Site-admin synthetic-monitor dashboard.
3 *
4 * GET /admin/status — red/green dashboard (SSE-live)
5 * POST /admin/status/run — run the suite synchronously
6 *
7 * Both gated behind `isSiteAdmin`. The public `/status` page (see
8 * `src/routes/status.tsx`) stays open to everyone; this is the detailed
9 * 14-row health table for the owner.
10 */
11
12import { Hono } from "hono";
13import { Layout } from "../views/layout";
14import { softAuth } from "../middleware/auth";
15import type { AuthEnv } from "../middleware/auth";
16import { isSiteAdmin } from "../lib/admin";
17import {
18 latestStatusByCheck,
19 recentRedChecks,
20 runSyntheticChecks,
21 persistChecks,
22 SSE_TOPIC,
23 SYNTHETIC_CHECKS,
24 type SyntheticCheckResult,
25} from "../lib/synthetic-monitor";
26
27const adminStatus = new Hono<AuthEnv>();
28adminStatus.use("*", softAuth);
29
30async function gate(c: any): Promise<{ user: any } | Response> {
31 const user = c.get("user");
32 if (!user) return c.redirect("/login?next=/admin/status");
33 if (!(await isSiteAdmin(user.id))) {
34 return c.html(
35 <Layout title="Forbidden" user={user}>
36 <div class="empty-state">
37 <h2>403 — Not a site admin</h2>
38 <p>You don't have permission to view this page.</p>
39 </div>
40 </Layout>,
41 403
42 );
43 }
44 return { user };
45}
46
47function statusDot(status: SyntheticCheckResult["status"] | undefined): string {
48 if (status === "green") return "\u{1F7E2}"; // green circle
49 if (status === "red") return "\u{1F534}"; // red circle
50 if (status === "yellow") return "\u{1F7E1}"; // yellow circle
51 return "⚪"; // white circle — never run
52}
53
54function fmtAgo(checkedAt: Date | undefined): string {
55 if (!checkedAt) return "never";
56 const diffMs = Date.now() - checkedAt.getTime();
57 if (diffMs < 0) return "just now";
58 const s = Math.floor(diffMs / 1000);
59 if (s < 5) return "just now";
60 if (s < 60) return `${s}s ago`;
61 const m = Math.floor(s / 60);
62 if (m < 60) return `${m}m ago`;
63 const h = Math.floor(m / 60);
64 if (h < 24) return `${h}h ago`;
65 const d = Math.floor(h / 24);
66 return `${d}d ago`;
67}
68
69adminStatus.get("/admin/status", async (c) => {
70 const g = await gate(c);
71 if (g instanceof Response) return g;
72 const { user } = g;
73
74 const latest = await latestStatusByCheck();
75 const recent = await recentRedChecks(24, 25);
76
77 const allGreen = SYNTHETIC_CHECKS.every(
78 (spec) => latest[spec.name]?.status === "green"
79 );
80
81 // Find the most-recent checkedAt across all rows so we can render the
82 // "last run Xs ago" badge.
83 let lastRunAt: Date | null = null;
84 for (const spec of SYNTHETIC_CHECKS) {
85 const row = latest[spec.name];
86 if (!row) continue;
87 if (!lastRunAt || row.checkedAt > lastRunAt) lastRunAt = row.checkedAt;
88 }
89
90 return c.html(
91 <Layout title="Synthetic monitor — admin" user={user}>
92 <div style="max-width: 960px; margin: 0 auto; padding: 24px 16px">
93 <div style="display: flex; align-items: center; gap: 12px; margin-bottom: 8px">
94 <span
95 style={`display: inline-block; width: 14px; height: 14px; border-radius: 50%; background: ${allGreen ? "var(--green, #2da44e)" : "var(--red, #cf222e)"}`}
96 />
97 <h1 style="margin: 0; font-size: 26px">
98 {allGreen ? "All checks green" : "One or more checks failing"}
99 </h1>
100 </div>
101 <p style="color: var(--text-muted); margin-bottom: 24px">
102 Synthetic monitor — runs every autopilot tick. Last run{" "}
103 <span data-last-run-at>{fmtAgo(lastRunAt)}</span>.
104 </p>
105
106 <div class="panel" style="margin-bottom: 20px">
107 <table
108 style="width: 100%; border-collapse: collapse; font-size: 14px"
109 id="synthetic-table"
110 >
111 <thead>
112 <tr style="text-align: left; color: var(--text-muted); font-size: 12px; text-transform: uppercase">
113 <th style="padding: 8px 12px; width: 28px"></th>
114 <th style="padding: 8px 12px">Check</th>
115 <th style="padding: 8px 12px; width: 80px">Status</th>
116 <th style="padding: 8px 12px; width: 90px">Duration</th>
117 <th style="padding: 8px 12px; width: 110px">Last run</th>
118 </tr>
119 </thead>
120 <tbody>
121 {SYNTHETIC_CHECKS.map((spec) => {
122 const row = latest[spec.name];
123 return (
124 <tr
125 data-check-name={spec.name}
126 style="border-top: 1px solid var(--border)"
127 >
128 <td
129 style="padding: 8px 12px"
130 data-cell="dot"
131 >
132 {statusDot(row?.status)}
133 </td>
134 <td style="padding: 8px 12px">
135 <code>{spec.name}</code>
136 {row?.error ? (
137 <div
138 style="font-size: 11px; color: var(--red, #cf222e); margin-top: 2px"
139 data-cell="error"
140 >
141 {row.error}
142 </div>
143 ) : null}
144 </td>
145 <td style="padding: 8px 12px" data-cell="status-code">
146 {row?.statusCode ?? "—"}
147 </td>
148 <td style="padding: 8px 12px" data-cell="duration">
149 {row ? `${row.durationMs}ms` : "—"}
150 </td>
151 <td
152 style="padding: 8px 12px; color: var(--text-muted)"
153 data-cell="ago"
154 >
155 {fmtAgo(row?.checkedAt)}
156 </td>
157 </tr>
158 );
159 })}
160 </tbody>
161 </table>
162 </div>
163
164 <form
165 action="/admin/status/run"
166 method="post"
167 style="margin-bottom: 32px"
168 >
169 <button class="btn-primary" type="submit">
170 Run all checks now
171 </button>
172 </form>
173
174 <h2 style="font-size: 16px; margin-bottom: 12px">
175 Recent red checks (last 24h)
176 </h2>
177 {recent.length === 0 ? (
178 <p style="color: var(--text-muted); font-size: 13px">
179 No red checks in the last 24 hours.
180 </p>
181 ) : (
182 <div class="panel">
183 {recent.map((r) => (
184 <div
185 class="panel-item"
186 style="justify-content: space-between; font-size: 13px"
187 >
188 <div>
189 <code>{r.name}</code>{" "}
190 <span style="color: var(--text-muted)">
191 — {r.error || "(no error message)"}
192 </span>
193 </div>
194 <span style="color: var(--text-muted); font-size: 12px">
195 {fmtAgo(r.checkedAt)}
196 </span>
197 </div>
198 ))}
199 </div>
200 )}
201
202 <script
203 // Live update via SSE. Each event is a SyntheticCheckResult; we
204 // patch the matching <tr data-check-name=...> in place.
205 dangerouslySetInnerHTML={{
206 __html: `
207(function() {
208 try {
209 var src = new EventSource('/live-events/${SSE_TOPIC}');
210 src.addEventListener('check', function(ev) {
211 var data;
212 try { data = JSON.parse(ev.data); } catch (e) { return; }
213 var row = document.querySelector('tr[data-check-name="' + data.name + '"]');
214 if (!row) return;
215 var dot = row.querySelector('[data-cell="dot"]');
216 var statusCode = row.querySelector('[data-cell="status-code"]');
217 var duration = row.querySelector('[data-cell="duration"]');
218 var ago = row.querySelector('[data-cell="ago"]');
219 if (dot) dot.textContent = data.status === 'green' ? '\\uD83D\\uDFE2' : (data.status === 'red' ? '\\uD83D\\uDD34' : '\\uD83D\\uDFE1');
220 if (statusCode) statusCode.textContent = data.statusCode != null ? String(data.statusCode) : '\\u2014';
221 if (duration) duration.textContent = data.durationMs + 'ms';
222 if (ago) ago.textContent = 'just now';
223 var lastRun = document.querySelector('[data-last-run-at]');
224 if (lastRun) lastRun.textContent = 'just now';
225 });
226 } catch (e) { /* SSE not available — page still renders the SSR snapshot */ }
227})();
228`,
229 }}
230 />
231 </div>
232 </Layout>
233 );
234});
235
236adminStatus.post("/admin/status/run", async (c) => {
237 const g = await gate(c);
238 if (g instanceof Response) return g;
239
240 // Fire-and-forget the run so we don't block the redirect on a slow
241 // network. Persist + SSE happens inside the helper.
242 void (async () => {
243 try {
244 const results = await runSyntheticChecks();
245 await persistChecks(results);
246 } catch (err) {
247 console.error("[admin-status] manual run failed:", err);
248 }
249 })();
250
251 return c.redirect("/admin/status");
252});
253
254export default adminStatus;
Modifiedsrc/routes/pwa.ts+87−5View fileUnifiedSplit
9696// No fetch handler — every request goes straight to the network.
9797`;
9898
99/**
100 * Block S2 — deploy-SHA-pinned cache bust.
101 *
102 * The SW source we SERVE from `/sw.js` is built per-request and pins its
103 * cache name to the current deploy SHA. The locked Block-G1
104 * `SERVICE_WORKER_SRC` constant above is preserved (exported for tests +
105 * historical context); the served body is built afresh in the handler so
106 * the SW_VERSION is always current.
107 *
108 * Behaviour:
109 * - `install` calls `skipWaiting()` so the new SW activates immediately
110 * instead of waiting for every tab to close.
111 * - `activate` deletes every `gluecron-*` cache that isn't the current
112 * version's cache, then `clients.claim()`s so open tabs adopt the new
113 * SW without a manual reload.
114 *
115 * The `Cache-Control: no-store` header on `/sw.js` itself ensures browsers
116 * always re-fetch the SW source on update checks — critical for the new
117 * version to actually reach returning visitors.
118 *
119 * BUILD_SHA is read from `process.env.BUILD_SHA` at request time so the
120 * deploy pipeline can rotate it without a rebuild. Falls back to a stable
121 * per-process `dev-<pid>` string when unset; a one-shot warn() is logged
122 * so operators notice misconfigured deploys.
123 */
124
125// One-shot warning latch — exported only for tests to reset between cases.
126let _missingShaWarned = false;
127export function _resetSwShaWarningForTests(): void {
128 _missingShaWarned = false;
129}
130
131export function buildSwVersion(): string {
132 const sha = process.env.BUILD_SHA?.trim();
133 if (sha) return sha;
134 if (!_missingShaWarned) {
135 _missingShaWarned = true;
136 console.warn(
137 "[pwa] BUILD_SHA env not set — service worker will fall back to a dev-mode version string. Set BUILD_SHA in the deploy environment so cache-busting pins to the deploy SHA."
138 );
139 }
140 // Dev fallback: stable per-process so reloads don't churn the cache
141 // while developing locally, but distinct from any real SHA.
142 return `dev-${process.pid}`;
143}
144
145export function buildVersionedServiceWorker(version: string): string {
146 // Escape backslashes + double-quotes so the version is safe inside a
147 // double-quoted JS string literal. Real SHAs are hex, but the dev
148 // fallback could in principle contain anything — belt + braces.
149 const safe = version.replace(/\\/g, "\\\\").replace(/"/g, '\\"');
150 return `// gluecron service worker — Block S2 (deploy-SHA-pinned cache bust)
151const SW_VERSION = "${safe}";
152const CACHE_PREFIX = "gluecron-";
153const CURRENT_CACHE = CACHE_PREFIX + SW_VERSION;
154
155self.addEventListener("install", (e) => {
156 // Activate immediately — don't wait for every tab to close. Pairs with
157 // the layout's updatefound→reload hook so the user sees the new HTML
158 // on the very next page load instead of "forever until DevTools".
159 self.skipWaiting();
160});
161
162self.addEventListener("activate", (e) => {
163 e.waitUntil(
164 caches.keys().then((names) =>
165 Promise.all(
166 names
167 .filter((n) => n.startsWith(CACHE_PREFIX) && n !== CURRENT_CACHE)
168 .map((n) => caches.delete(n))
169 )
170 ).then(() => self.clients.claim())
171 );
172});
173
174// No fetch handler — every request goes straight to the network. The
175// version-pinned cache machinery is in place for future opt-in caching
176// without re-introducing the stale-HTML bug.
177`;
178}
179
99180pwa.get("/sw.js", (c) => {
100181 c.header("content-type", "application/javascript");
101 // No-cache: browser must check on every page load. Critical for the v4
102 // self-nuke SW to actually reach all returning visitors.
103 c.header("cache-control", "no-cache, no-store, must-revalidate");
182 // no-store on /sw.js itself: browsers must re-fetch the SW source on
183 // every update check so the new SW_VERSION can actually propagate.
184 c.header("cache-control", "no-store");
104185 c.header("pragma", "no-cache");
105 // Service-Worker-Allowed required for root-scope SW served from root
186 // Service-Worker-Allowed required for root-scope SW served from root.
106187 c.header("service-worker-allowed", "/");
107 return c.body(SERVICE_WORKER_SRC);
188 const version = buildSwVersion();
189 return c.body(buildVersionedServiceWorker(version));
108190});
109191
110192/**
Modifiedsrc/routes/status.tsx+37−1View fileUnifiedSplit
1818import { softAuth } from "../middleware/auth";
1919import type { AuthEnv } from "../middleware/auth";
2020import { getLastTick, getTickCount } from "../lib/autopilot";
21import { recentRedChecks } from "../lib/synthetic-monitor";
2122
2223const status = new Hono<AuthEnv>();
2324status.use("*", softAuth);
8283 const autopilotDisabled = process.env.AUTOPILOT_DISABLED === "1";
8384 const uptimeMs = Date.now() - started;
8485
85 const overallOk = dbOk;
86 // BLOCK S4 — Show any red synthetic-monitor results from the last 24h
87 // on the public status page. Never blocks the render.
88 let recentIncidents: Awaited<ReturnType<typeof recentRedChecks>> = [];
89 try {
90 recentIncidents = await recentRedChecks(24, 10);
91 } catch {
92 recentIncidents = [];
93 }
94
95 const overallOk = dbOk && recentIncidents.length === 0;
8696
8797 return c.html(
8898 <Layout title="Status — gluecron" user={user}>
215225 </div>
216226 </div>
217227
228 {recentIncidents.length > 0 ? (
229 <>
230 <h2 style="margin-bottom: 12px; font-size: 18px">
231 Recent incidents (last 24h)
232 </h2>
233 <div class="panel" style="margin-bottom: 24px">
234 {recentIncidents.map((r) => (
235 <div
236 class="panel-item"
237 style="justify-content: space-between; font-size: 13px"
238 >
239 <div>
240 <code>{r.name}</code>
241 <span style="color: var(--text-muted); margin-left: 8px">
242 — {r.error || "(no error message)"}
243 </span>
244 </div>
245 <span style="color: var(--text-muted); font-size: 12px">
246 {r.checkedAt.toISOString()}
247 </span>
248 </div>
249 ))}
250 </div>
251 </>
252 ) : null}
253
218254 <h2 style="margin-bottom: 12px; font-size: 18px">
219255 Latest autopilot tick
220256 </h2>
Modifiedsrc/routes/version.ts+74−2View fileUnifiedSplit
99 * - Monitoring (latency to seeing a new sha = end-to-end deploy time)
1010 *
1111 * Cache-control: no-store. Must be live, never cached.
12 *
13 * Block S3 (2026-05-14): additively reports the 5 most recently applied
14 * migrations from the live DB so the post-deploy smoke suite can verify
15 * the latest drizzle/*.sql file actually landed in the running schema.
16 * The migrations field is best-effort: if the DB query fails or the
17 * connection isn't configured the field is omitted, never throws.
1218 */
1319
1420import { Hono } from "hono";
21import { sql } from "drizzle-orm";
1522import { getBuildInfo } from "../lib/build-info";
23import { db } from "../db";
1624
1725const version = new Hono();
1826
19version.get("/api/version", (c) => {
27// In-process cache for the migrations list. /api/version is polled
28// every 15s by the auto-update banner — re-querying _migrations on
29// every hit would be wasteful. 10s TTL is a reasonable compromise: a
30// fresh deploy's new migration will show up in the smoke check within
31// at most 10s of when migrate.ts inserted the row.
32const MIGRATIONS_CACHE_TTL_MS = 10_000;
33let _migrationsCache: { at: number; names: string[] } | null = null;
34
35async function readRecentMigrations(): Promise<string[] | null> {
36 const now = Date.now();
37 if (_migrationsCache && now - _migrationsCache.at < MIGRATIONS_CACHE_TTL_MS) {
38 return _migrationsCache.names;
39 }
40 try {
41 // _migrations is created on first migration run; if the table doesn't
42 // exist (very first boot before migrate.ts has ever run) we degrade
43 // silently to an empty list.
44 const rows = (await db.execute(
45 sql`SELECT name FROM _migrations ORDER BY applied_at DESC LIMIT 5`
46 )) as unknown as Array<{ name: string }>;
47 const names = (rows ?? []).map((r) => r.name).filter(Boolean);
48 _migrationsCache = { at: now, names };
49 return names;
50 } catch {
51 // _migrations table missing, DB down, etc. — return null so the
52 // endpoint can omit the field without 500ing.
53 return null;
54 }
55}
56
57/**
58 * Test seam: lets `src/__tests__/post-deploy-smoke.test.ts` reset the
59 * cache between assertions. Not part of the public API.
60 */
61export const __test = {
62 clearMigrationsCache: () => {
63 _migrationsCache = null;
64 },
65};
66
67version.get("/api/version", async (c) => {
68 c.header("cache-control", "no-store, no-cache, must-revalidate");
69 c.header("pragma", "no-cache");
70 const build = getBuildInfo();
71 const migrations = await readRecentMigrations();
72 if (migrations !== null) {
73 return c.json({ ...build, migrations });
74 }
75 return c.json(build);
76});
77
78// Block S2 — minimal `/version` alias used by the service-worker cache
79// bust + external uptime/deploy monitors. Kept additive (single new
80// handler, no edits to existing shape) so the S1+S3 smoke-suite agent's
81// edits to this file can land alongside without merge conflicts.
82//
83// S3 (2026-05-14): mirror the migrations field here so the smoke
84// suite can check either endpoint.
85version.get("/version", async (c) => {
2086 c.header("cache-control", "no-store, no-cache, must-revalidate");
2187 c.header("pragma", "no-cache");
22 return c.json(getBuildInfo());
88 const migrations = await readRecentMigrations();
89 const body: Record<string, unknown> = {
90 sha: process.env.BUILD_SHA || "dev",
91 buildAt: process.env.BUILD_TIME || null,
92 };
93 if (migrations !== null) body.migrations = migrations;
94 return c.json(body);
2395});
2496
2597export default version;
Modifiedsrc/views/layout.tsx+22−1View fileUnifiedSplit
515515
516516// Block G1 — register service worker for offline / install support.
517517// Kept inline (and tiny) so we don't block first paint.
518//
519// Block S2 extension — when an updated SW activates we force a single
520// reload so the user picks up the fresh HTML immediately instead of
521// being stuck on the previous deploy's cached page (the "I saw /login's
522// old version even after merging" bug that triggered S2 in the first
523// place). The reload only fires when `controller` is already set, which
524// means this is NOT the first-ever install (no double-load on first
525// visit). A page-scoped guard prevents reload loops if the SW updates
526// twice in quick succession.
518527const pwaRegisterScript = `
519528 if ('serviceWorker' in navigator) {
520529 window.addEventListener('load', function(){
521 navigator.serviceWorker.register('/sw.js').catch(function(){});
530 navigator.serviceWorker.register('/sw.js').then(function(reg){
531 var reloaded = false;
532 reg.addEventListener('updatefound', function(){
533 var newSW = reg.installing;
534 if (!newSW) return;
535 newSW.addEventListener('statechange', function(){
536 if (newSW.state === 'activated' && navigator.serviceWorker.controller && !reloaded) {
537 reloaded = true;
538 window.location.reload();
539 }
540 });
541 });
542 }).catch(function(){});
522543 });
523544 }
524545`;
525546