CodeIssuesDiscussionsWikiPull RequestsProjectsCommitsActionsReleasesContributorsPulse● GatesSecuritySettingsDeploymentsPipelineInsightsAgents✨ Explain✨ Ask AI✨ Workspace✨ Spec✨ Tests▓ Debt Map✨ NL Search🏛 Archaeology
Blame · Line-by-line history
AUDIT-v2.md
Each line is annotated with the commit that last touched it. Click any SHA to jump to that commit and see the surrounding change.
| 74a8784 | 1 | > **SUPERSEDED (2026-06-10):** historical record only. The current audit is `AUDIT-2026-06-10.md` at the repo root. |
| 2 | ||
| 976d7f7 | 3 | # Gluecron Audit v2 — 2026-05-16 |
| 4 | ||
| 5 | Five parallel agents (schema-vs-code drift, route smoke crawl, form | |
| 6 | submission, silent failures, deploy pipeline) audited the codebase | |
| 7 | at commit `a358c63` after the AA-loop / Hetzner-deploy firefight. | |
| 8 | ||
| 9 | ## Headline | |
| 10 | ||
| 11 | The codebase is **structurally sound** — schema matches code, forms are | |
| 12 | correctly wired, route mounting is consistent. The user-visible | |
| 13 | "website is broken" symptoms trace to a small set of **silent failure | |
| 14 | modes** in observability, integrations, and the deploy pipeline. | |
| 15 | ||
| 16 | ## P0 — User-blocking, fix now | |
| 17 | ||
| 18 | | # | Bug | File | Effect | | |
| 19 | |---|---|---|---| | |
| 20 | | 1 | `/orgs/:slug/people` null-deref on anonymous users | `src/routes/orgs.tsx:338` | 500 instead of 302 to login | | |
| 21 | | 2 | `/settings` 500 from missing migration 0045 | n/a (deploy bug) | All `auto_close_stale_*` reads crash. **Fix shipped in `a358c63`** — next deploy applies it. | | |
| 22 | | 3 | Email silently disabled when `RESEND_API_KEY` missing | `src/routes/auth.tsx:222`, `src/lib/email.ts:49-85` | Registration succeeds, verification email never sends, user locked out. No error surface. | | |
| 23 | | 4 | AI review/triage comments swallowed on DB error | `src/lib/ai-review.ts:281,297,317`, `src/lib/pr-triage.ts:233` | DB blip → comments never appear → user thinks AI never ran | | |
| 24 | | 5 | Deploy failure diagnostics curl wrong port | `.github/workflows/hetzner-deploy.yml:403` | Service runs on 3010, diagnostics check 3000 → every failure dump useless | | |
| 25 | | 6 | Rollback `bun install` uses `\|\| true` | `.github/workflows/hetzner-deploy.yml:364` | Corrupted deps don't abort rollback → service restarted with broken node_modules | | |
| 26 | | 7 | GateTest fires unauthenticated when `GATETEST_API_KEY` missing | `src/lib/gate.ts:128`, `src/lib/gate.ts:122` | 401 silent → scans never run | | |
| 27 | | 8 | Crontech deploy webhook fires without signature when secret missing | `src/hooks/post-receive.ts:309` | Rejected silently → deploy looks fired, never landed | | |
| 28 | ||
| 29 | ## P1 — Real bugs, lower frequency | |
| 30 | ||
| 31 | | # | Bug | File | Effect | | |
| 32 | |---|---|---|---| | |
| 33 | | 9 | Repo-scoped routes 500 instead of 404/empty on missing record | `issues.tsx`, `pulls.tsx`, `packages.tsx`, `releases.tsx`, etc. | DB blip → 500 instead of graceful degradation | | |
| 34 | | 10 | No DB-blip protection — every page blocks 5-15s on a sick DB | All routes that read DB | Sick DB → site appears down instead of slow | | |
| 35 | | 11 | Two deploy paths drifted | `scripts/self-deploy.sh` vs `.github/workflows/hetzner-deploy.yml` | One has compile/cache/verifier, the other doesn't. Inconsistent state on prod. | | |
| 36 | | 12 | No Fly rollback path | `.github/workflows/fly-deploy.yml` | Bad Fly deploy = manual recovery | | |
| 37 | | 13 | Deployment INSERT `.catch(() => {})` | `src/hooks/post-receive.ts:282-296` | If DB blip, deploy row never persists, Crontech callbacks for unknown deployId | | |
| 38 | | 14 | PR close-keyword closing comment swallowed | `src/lib/pr-merge.ts:179-181` | Issue closes but no back-link comment posted on issue | | |
| 39 | | 15 | Workflow log truncation has no UX warning | `src/lib/workflow-runner.ts:94-97` | User sees "[... truncated ...]" with no indication of how much was lost | | |
| 40 | ||
| 41 | ## P2 — Cleanup / risk | |
| 42 | ||
| 43 | | # | Item | File | Notes | | |
| 44 | |---|---|---|---| | |
| 45 | | 16 | `pr_risk_scores` table is dead schema | `drizzle/0044_pr_risk_scores.sql` | Created, never written to or read | | |
| 46 | | 17 | `scripts/deploy-crontech.sh` is dead code | `scripts/deploy-crontech.sh` | Referenced in comments, never invoked | | |
| 47 | | 18 | Stripe webhook `continue-on-error: true` on volume creation | `.github/workflows/fly-deploy.yml:48` | Mountpoint can be wrong without alarm | | |
| 48 | | 19 | `/admin/sso`, `/admin/github-oauth`, `/admin/mirrors/sync-all` use middleware `requireAuth` (not `requireAdmin`), gated only by inline `isSiteAdmin()` check | Multiple admin routes | Inconsistent pattern; current gate is correct but easy to forget on a new route | | |
| 49 | ||
| 50 | ## False alarms — investigated, no issue | |
| 51 | ||
| 52 | - **Schema drift**: agents found zero columns the code reads that aren't in `schema.ts` and zero columns in `schema.ts` without a matching migration. All 54 migrations are consistent with the code. | |
| 53 | - **Broken nav**: 172 routes, every internal `href` and every form `action` resolves to a registered handler. (Single exception was `/legal/terms` link from register form, fixed in `2e8a4d5`.) | |
| 54 | - **Admin auth bypass**: every admin route is properly gated via either middleware or inline `isSiteAdmin()` check. Inconsistent pattern, not a security hole. | |
| 55 | - **Form submission**: every form's POST handler exists, reads the right fields, redirects to a real page, persists data correctly. | |
| 56 | ||
| 57 | ## What the user experienced | |
| 58 | ||
| 59 | 1. **PWA reload loop** — fixed (commits `d7ba05d`, `904927d`, `44fe49b` ripped out PWA, added kill-switch). | |
| 60 | 2. **"Buttons don't work"** — root cause was the reload loop cancelling clicks before they fired. Fixed by the above. | |
| 61 | 3. **`/settings` 500** — root cause was deploy pipeline silently failing for 17 hours, so migration 0045 never ran on prod. Fixed by `a358c63` (re-added migration step to deploy). | |
| 62 | 4. **Hours of "still broken"** — root cause was the deploy pipeline being non-functional (Hetzner git remote 404, silent script abort, no migration step). Fixed by `ec16b67`, `d8b9606`, `a358c63`. | |
| 63 | ||
| 64 | ## Roadmap | |
| 65 | ||
| 66 | This commit adds AUDIT-v2.md. The next batch of commits will burn | |
| 67 | through P0s 1, 3, 4, 5, 6, 7, 8 — each as a small commit pushed | |
| 68 | straight to main. P1s and P2s will get their own session. | |
| 119321f | 69 | |
| 70 | --- | |
| 71 | ||
| 72 | ## RELIABILITY SWEEP — 2026-05-16 (executed) | |
| 73 | ||
| 74 | After AUDIT-v2.md landed, a four-phase reliability sweep targeting | |
| 75 | the systemic issues was executed in a single session. All P0s above | |
| 76 | are fixed; key P1s and Level 1–4 of the AI-intelligent-server stack | |
| 77 | shipped: | |
| 78 | ||
| 79 | ### Phase A — Silent failure sweep (Level 1) — `a28cede` | |
| 80 | Replaced every `.catch(() => {})` in production code (~30 sites across | |
| 81 | 18 files) with structured `console.warn` calls that include the | |
| 82 | operation name, relevant identifier, and error message. Operators can | |
| 83 | now `journalctl -u gluecron | grep '\[<component>\]'` to find why | |
| 84 | something stopped working, instead of guessing. Sites: boot path, | |
| 85 | auth surface, git surface, AI flows, auto-repair, background workers, | |
| 86 | SSO, MCP tools, action cleanup. | |
| 87 | ||
| 88 | ### Phase B — `/admin/health` + JSON endpoint (Level 2) — `115c66b` | |
| 89 | Built on the existing `/admin/diagnose` traffic-light dashboard: | |
| 90 | - Added `/admin/health` as a friendly alias (302 to /admin/diagnose) | |
| 91 | - Added `/admin/diagnose.json` for programmatic monitoring (same | |
| 92 | site-admin gate — never leaks deploy state) | |
| 93 | - Added 4 new health checks: Autopilot (background loop ticking on | |
| 94 | schedule), Latest deploy (red on failed / yellow if > 48h stale), | |
| 95 | Workflow queue (red if > 25 queued), Crontech webhook (red on | |
| 96 | misconfigured HMAC) | |
| 97 | - Updated /admin dashboard button: "Diagnose" → "Health / Diagnose" | |
| 98 | - Regression test in `src/__tests__/admin-health.test.ts` | |
| 99 | ||
| 100 | External monitors can now poll `GET /admin/diagnose.json` every | |
| 101 | minute and alert on any red status. The "17 hours of silent deploy | |
| 102 | failure" failure mode is mitigated by the autopilot + deploy-staleness | |
| 103 | checks visible in one URL. | |
| 104 | ||
| 105 | ### Phase C — AI incident responder for platform deploys (Level 3) — `89a0761` | |
| 106 | `src/lib/ai-incident.ts` previously had `onDeployFailure` for | |
| 107 | downstream-app deploys only. Added sibling `analyzePlatformDeployFailure` | |
| 108 | that targets the `platform_deploys` table (populated by | |
| 109 | hetzner-deploy.yml). When a deploy event with status="failed" arrives | |
| 110 | at `/deploy/finished`: | |
| 111 | 1. Loads last 10 commits to main from the box-side repo | |
| 112 | 2. Calls Claude Sonnet for a structured RCA (title, likely cause, | |
| 113 | suspected commit, remediation) | |
| 114 | 3. Logs the analysis as `[platform-incident]` to journalctl | |
| 115 | 4. Inserts an `audit_log` row with action=`platform.deploy.failed` | |
| 116 | containing the run_id, sha, error excerpt, and AI RCA | |
| 117 | ||
| 118 | Degrades gracefully when ANTHROPIC_API_KEY is unset (falls back to | |
| 119 | deterministic markdown with raw error + recent commits, marked | |
| 120 | "AI unavailable"). Operators get the diagnosis at the same moment | |
| 121 | the failure happens — no manual investigation required. | |
| 122 | ||
| 123 | ### Phase D — Deploy path unification (Level 4 partial) — `ff4423b` | |
| 124 | `scripts/self-deploy.sh` and `.github/workflows/hetzner-deploy.yml` | |
| 125 | had drifted into two divergent deploy scripts. Collapsed them: the | |
| 126 | GH workflow now does `cd /opt/gluecron && git fetch + reset && bash | |
| 127 | scripts/self-deploy.sh --inline`. The script is the single source of | |
| 128 | truth, invoked by both the GH workflow and the post-receive hook on | |
| 129 | the gluecron-hosted git server. No more drift. | |
| 130 | ||
| 131 | Bonus: `self-deploy.sh` gains `set -x` tracing (was opaque on | |
| 132 | failure). And the GH workflow gains the script's full robustness: | |
| 133 | real post-deploy smoke suite, healthz polling, bun build --compile, | |
| 134 | automatic rollback to previous SHA on failure. | |
| 135 | ||
| 136 | ### What's still open (P1 follow-ups not in this sweep) | |
| 137 | ||
| 138 | - Repo-scoped routes that 500 instead of 404/empty on missing record | |
| 139 | (issues.tsx, pulls.tsx, packages.tsx, releases.tsx, etc.) — each | |
| 140 | needs a try/catch around the drizzle query | |
| 141 | - Request-level resilience middleware (catch DB connection errors, | |
| 142 | return graceful 503) — design + implementation | |
| 143 | - No Fly rollback path (`fly-deploy.yml` doesn't have one yet) | |
| 144 | - Workflow log truncation has no UX warning | |
| 145 | - Dead schema (`pr_risk_scores` table) | |
| 146 | - Dead code (`scripts/deploy-crontech.sh`) | |
| 147 | ||
| 148 | These are tractable individually but didn't fit in this session. |