Commit119321funknown_key

docs: AUDIT-v2.md — record the 2026-05-16 reliability sweep outcome

docs: AUDIT-v2.md — record the 2026-05-16 reliability sweep outcome

Four phases (A-D) shipped in one session: silent-failure sweep,
/admin/health + JSON endpoint + 4 new checks, AI incident responder
for platform deploys, deploy path unification.

This is the 'top-notch AI experience' baseline the user asked for:
Level 1 (visible) + Level 2 (self-monitoring) + Level 3 (self-
diagnosing) + Level 4 partial (deploy unification). The remaining
P1s are tractable but didn't fit in this session — listed at the
bottom of the doc for the next focused pass.

Claude committed on May 16, 2026Parent: ff4423b

1 file changed+80−0119321fa2028c8100795b14bc854e8fe2c439f10

1 changed file+80−0

ModifiedAUDIT-v2.md+80−0View fileUnified Split

@@ -64,3 +64,83 @@ modes** in observability, integrations, and the deploy pipeline.

6464This commit adds AUDIT-v2.md. The next batch of commits will burn

6565through P0s 1, 3, 4, 5, 6, 7, 8 — each as a small commit pushed

6666straight to main. P1s and P2s will get their own session.

68---

70## RELIABILITY SWEEP — 2026-05-16 (executed)

72After AUDIT-v2.md landed, a four-phase reliability sweep targeting

73the systemic issues was executed in a single session. All P0s above

74are fixed; key P1s and Level 1–4 of the AI-intelligent-server stack

75shipped:

77### Phase A — Silent failure sweep (Level 1) — `a28cede`

78Replaced every `.catch(() => {})` in production code (~30 sites across

7918 files) with structured `console.warn` calls that include the

80operation name, relevant identifier, and error message. Operators can

81now `journalctl -u gluecron | grep '\[<component>\]'` to find why

82something stopped working, instead of guessing. Sites: boot path,

83auth surface, git surface, AI flows, auto-repair, background workers,

84SSO, MCP tools, action cleanup.

86### Phase B — `/admin/health` + JSON endpoint (Level 2) — `115c66b`

87Built on the existing `/admin/diagnose` traffic-light dashboard:

88 - Added `/admin/health` as a friendly alias (302 to /admin/diagnose)

89 - Added `/admin/diagnose.json` for programmatic monitoring (same

90 site-admin gate — never leaks deploy state)

91 - Added 4 new health checks: Autopilot (background loop ticking on

92 schedule), Latest deploy (red on failed / yellow if > 48h stale),

93 Workflow queue (red if > 25 queued), Crontech webhook (red on

94 misconfigured HMAC)

95 - Updated /admin dashboard button: "Diagnose" → "Health / Diagnose"

96 - Regression test in `src/__tests__/admin-health.test.ts`

98External monitors can now poll `GET /admin/diagnose.json` every

99minute and alert on any red status. The "17 hours of silent deploy

100failure" failure mode is mitigated by the autopilot + deploy-staleness

101checks visible in one URL.

102

103### Phase C — AI incident responder for platform deploys (Level 3) — `89a0761`

104`src/lib/ai-incident.ts` previously had `onDeployFailure` for

105downstream-app deploys only. Added sibling `analyzePlatformDeployFailure`

106that targets the `platform_deploys` table (populated by

107hetzner-deploy.yml). When a deploy event with status="failed" arrives

108at `/deploy/finished`:

109 1. Loads last 10 commits to main from the box-side repo

110 2. Calls Claude Sonnet for a structured RCA (title, likely cause,

111 suspected commit, remediation)

112 3. Logs the analysis as `[platform-incident]` to journalctl

113 4. Inserts an `audit_log` row with action=`platform.deploy.failed`

114 containing the run_id, sha, error excerpt, and AI RCA

115

116Degrades gracefully when ANTHROPIC_API_KEY is unset (falls back to

117deterministic markdown with raw error + recent commits, marked

118"AI unavailable"). Operators get the diagnosis at the same moment

119the failure happens — no manual investigation required.

120

121### Phase D — Deploy path unification (Level 4 partial) — `ff4423b`

122`scripts/self-deploy.sh` and `.github/workflows/hetzner-deploy.yml`

123had drifted into two divergent deploy scripts. Collapsed them: the

124GH workflow now does `cd /opt/gluecron && git fetch + reset && bash

125scripts/self-deploy.sh --inline`. The script is the single source of

126truth, invoked by both the GH workflow and the post-receive hook on

127the gluecron-hosted git server. No more drift.

128

129Bonus: `self-deploy.sh` gains `set -x` tracing (was opaque on

130failure). And the GH workflow gains the script's full robustness:

131real post-deploy smoke suite, healthz polling, bun build --compile,

132automatic rollback to previous SHA on failure.

133

134### What's still open (P1 follow-ups not in this sweep)

135

136- Repo-scoped routes that 500 instead of 404/empty on missing record

137 (issues.tsx, pulls.tsx, packages.tsx, releases.tsx, etc.) — each

138 needs a try/catch around the drizzle query

139- Request-level resilience middleware (catch DB connection errors,

140 return graceful 503) — design + implementation

141- No Fly rollback path (`fly-deploy.yml` doesn't have one yet)

142- Workflow log truncation has no UX warning

143- Dead schema (`pr_risk_scores` table)

144- Dead code (`scripts/deploy-crontech.sh`)

145

146These are tractable individually but didn't fit in this session.

67147