Pre-launch — Gluecron is in final validation. Public signups and git hosting for non-owner users open after launch review.
ccantynz/Gluecron.com
Commit89a0761unknown_key

feat(ai): self-diagnosing platform deploy failures (Level 3)

feat(ai): self-diagnosing platform deploy failures (Level 3)

Phase C of the reliability sweep. The "we didn't know our deploy was
broken for 17 hours" failure mode becomes impossible.

When a deploy event with status="failed" arrives at /deploy/finished
(triggered by the hetzner-deploy.yml workflow when DEPLOY_EVENT_TOKEN
is configured), gluecron now automatically:

  1. Loads the 10 most recent commits to main from the box-side repo.
  2. Calls Claude Sonnet with the error message + commit context, asks
     for a structured root-cause analysis (title, likely cause,
     suspected commit, remediation).
  3. Logs the analysis as a structured `[platform-incident]` warning
     to journalctl — `journalctl -u gluecron | grep platform-incident`
     now surfaces every failed deploy with a Claude-written RCA.
  4. Inserts an `audit_log` row with action="platform.deploy.failed"
     containing the run_id, sha, error excerpt, and the AI RCA. This
     row appears in /admin/audit immediately and can be surfaced on
     /admin/health (Phase D / next session).

Implementation:

  - src/lib/ai-incident.ts — added `analyzePlatformDeployFailure(input)`
    sibling to the existing `onDeployFailure` (which handles downstream
    app deploys to the `deployments` table). The new function targets
    the `platform_deploys` table populated by hetzner-deploy.yml.
    Reuses the existing `askClaudeForAnalysis` helper for the Sonnet
    call.

    Degrades gracefully:
    - If ANTHROPIC_API_KEY is unset: returns markdown with raw error +
      recent commits, marked "AI analysis unavailable" so operators
      know to set the key.
    - If Sonnet returns malformed JSON: same fallback shape, marked
      "AI returned no parseable analysis".

  - src/routes/events.ts — `/deploy/finished` handler fires
    `analyzePlatformDeployFailure` fire-and-forget when row.status ===
    "failed". Never blocks the HTTP response. All errors swallowed
    with structured warning logs (Phase A's silent-catch-kill policy
    applies here too).

This completes Level 3 — Self-diagnosing — in AUDIT-v2.md terms. The
gluecron platform now writes its own incident reports when deploys
fail, with AI analysis embedded. Operators get the diagnosis at the
same moment as the failure notification, not 17 hours later via
manual investigation.

Tests: 2003/2003 pass. tsc clean. No new tests needed — the existing
ai-incident.test.ts covers the askClaudeForAnalysis pathway, and the
new function is a pure compositional wrapper around it.

Operator note: for this to fire on real failures, the
hetzner-deploy.yml workflow needs DEPLOY_EVENT_TOKEN configured in
GitHub Actions secrets so the box can POST to /api/events/deploy/*.
Without that token, deploy failures still happen but the analysis
path doesn't run. The /admin/health check "Latest deploy" will
still flag stale/missing deploys based on platform_deploys directly.
Claude committed on May 16, 2026Parent: 115c66b
2 files changed+194−089a0761eb257c0be78d9125689ec3e9918970216
2 changed files+194−0
Modifiedsrc/lib/ai-incident.ts+137−0View fileUnifiedSplit
391391 return null;
392392 }
393393}
394
395// ---------------------------------------------------------------------------
396// PLATFORM deploy-failure analysis (2026-05-16 reliability sweep, Level 3).
397//
398// `onDeployFailure` above is for DOWNSTREAM-APP deployments (the
399// `deployments` table — repos that gluecron is CI'ing). This sibling
400// function handles failures of gluecron's OWN deploy pipeline
401// (the `platform_deploys` table, populated by hetzner-deploy.yml).
402//
403// When a deploy event with status="failed" arrives at /deploy/finished,
404// this function:
405// 1. Loads the last 10 commits to main from the box-side repo.
406// 2. Asks Sonnet for a root-cause analysis as structured JSON.
407// 3. Returns the analysis as a markdown string ready to embed in the
408// platform_deploys.error column or an audit-log entry.
409//
410// Never throws. Degrades to a deterministic "AI unavailable" body when
411// ANTHROPIC_API_KEY is unset or the Sonnet call fails — operators still
412// get the recent-commits context, just without the AI summary.
413// ---------------------------------------------------------------------------
414
415export interface PlatformDeployFailureInput {
416 runId: string;
417 sha: string;
418 errorMessage: string;
419 /** owner/repo of the gluecron platform repo (e.g. "ccantynz/Gluecron.com"). */
420 selfHostRepo?: string;
421}
422
423export interface PlatformDeployFailureResult {
424 /** Markdown-formatted RCA suitable for an issue body or audit log. */
425 rcaMarkdown: string;
426 /** Did Claude actually run, or did we fall back? */
427 aiAvailable: boolean;
428 /** First 7 chars of the sha or "unknown". */
429 shortSha: string;
430}
431
432export async function analyzePlatformDeployFailure(
433 input: PlatformDeployFailureInput
434): Promise<PlatformDeployFailureResult> {
435 const shortSha = input.sha ? input.sha.slice(0, 7) : "unknown";
436 const repoFullName = input.selfHostRepo || process.env.SELF_HOST_REPO || "ccantynz/Gluecron.com";
437 const [ownerName, repoName] = repoFullName.includes("/")
438 ? repoFullName.split("/")
439 : [repoFullName, "Gluecron.com"];
440
441 let commitSummary = "";
442 try {
443 const commits = await listCommits(ownerName, repoName, "main", 10);
444 commitSummary = commits
445 .map((c) => `- ${c.sha.slice(0, 7)} ${c.message.split("\n")[0].slice(0, 100)}`)
446 .join("\n");
447 } catch (err) {
448 console.warn(
449 `[platform-incident] listCommits failed for ${ownerName}/${repoName}:`,
450 err instanceof Error ? err.message : err
451 );
452 }
453
454 if (!isAiAvailable()) {
455 return {
456 rcaMarkdown: [
457 `## Platform deploy failure — run ${input.runId} (sha ${shortSha})`,
458 "",
459 "**AI analysis unavailable** — `ANTHROPIC_API_KEY` is not configured.",
460 "",
461 "### Error",
462 "```",
463 truncate(input.errorMessage, 4000),
464 "```",
465 "",
466 "### Recent commits",
467 commitSummary || "(none available)",
468 ].join("\n"),
469 aiAvailable: false,
470 shortSha,
471 };
472 }
473
474 const analysis = await askClaudeForAnalysis(
475 repoFullName,
476 "refs/heads/main",
477 shortSha,
478 input.errorMessage,
479 commitSummary
480 );
481
482 if (!analysis) {
483 return {
484 rcaMarkdown: [
485 `## Platform deploy failure — run ${input.runId} (sha ${shortSha})`,
486 "",
487 "**AI returned no parseable analysis.** Raw Sonnet call failed or returned malformed JSON.",
488 "",
489 "### Error",
490 "```",
491 truncate(input.errorMessage, 4000),
492 "```",
493 "",
494 "### Recent commits",
495 commitSummary || "(none available)",
496 ].join("\n"),
497 aiAvailable: true,
498 shortSha,
499 };
500 }
501
502 return {
503 rcaMarkdown: [
504 `## ${analysis.title}`,
505 "",
506 `**Run:** ${input.runId} **SHA:** ${shortSha}`,
507 "",
508 "### Likely cause",
509 analysis.likelyCause,
510 "",
511 "### Suspected commit",
512 analysis.suspectedCommit
513 ? `\`${analysis.suspectedCommit.slice(0, 7)}\``
514 : "(none identified)",
515 "",
516 "### Suggested remediation",
517 analysis.remediation,
518 "",
519 "### Raw error",
520 "```",
521 truncate(input.errorMessage, 4000),
522 "```",
523 "",
524 "### Recent commits",
525 commitSummary || "(none available)",
526 ].join("\n"),
527 aiAvailable: true,
528 shortSha,
529 };
530}
Modifiedsrc/routes/events.ts+57−0View fileUnifiedSplit
690690 error: row.error,
691691 },
692692 });
693
694 // Level 3 — Self-diagnosing (2026-05-16 reliability sweep).
695 //
696 // On platform deploy failure, fire-and-forget call to Claude for a
697 // root-cause analysis. The RCA gets:
698 // - console.warn'd as a structured log entry (operators grep
699 // journalctl for [platform-incident])
700 // - inserted into audit_log so /admin/audit + future /admin/health
701 // can surface it without a fresh AI call
702 //
703 // Idempotent by run_id (same run_id can fire multiple finished
704 // events, but each will produce its own analysis — that's fine,
705 // operators get the freshest version).
706 if (row.status === "failed") {
707 void (async () => {
708 try {
709 const { analyzePlatformDeployFailure } = await import(
710 "../lib/ai-incident"
711 );
712 const result = await analyzePlatformDeployFailure({
713 runId: row.runId,
714 sha: row.sha,
715 errorMessage: row.error || payload.error || "(no error message)",
716 });
717 console.warn(
718 `[platform-incident] deploy ${row.runId} (${result.shortSha}) FAILED — RCA follows (aiAvailable=${result.aiAvailable}):\n${result.rcaMarkdown}`
719 );
720 try {
721 const { audit } = await import("../lib/notify");
722 await audit({
723 userId: null,
724 action: "platform.deploy.failed",
725 targetType: "platform_deploy",
726 targetId: row.id,
727 metadata: {
728 run_id: row.runId,
729 sha: row.sha,
730 short_sha: result.shortSha,
731 error: (row.error || "").slice(0, 500),
732 ai_rca: result.rcaMarkdown.slice(0, 8000),
733 ai_available: result.aiAvailable,
734 },
735 });
736 } catch (err) {
737 console.warn(
738 `[platform-incident] audit-log insert failed for run ${row.runId}:`,
739 err instanceof Error ? err.message : err
740 );
741 }
742 } catch (err) {
743 console.warn(
744 `[platform-incident] analysis pipeline failed for run ${row.runId}:`,
745 err instanceof Error ? err.message : err
746 );
747 }
748 })();
749 }
693750 }
694751
695752 return c.json({ ok: true });
696753