Pre-launch — Gluecron is in final validation. Public signups and git hosting for non-owner users open after launch review.
ccantynz/Gluecron.com
Commita28cedeunknown_key

fix: kill silent failures — phase A of reliability sweep (Level 1)

fix: kill silent failures — phase A of reliability sweep (Level 1)

Audit found ~30 production sites where async operations failed silently
via `.catch(() => {})`. When DB blips, when integrations fail, when
worktrees can't be cleaned up — none of it surfaced. Operators couldn't
grep journalctl for "why didn't X happen" because nothing was logged.

This commit replaces every `.catch(() => {})` in production code with
structured `console.warn` calls including:
  - The operation that failed
  - The relevant identifier (PR id, repo, user, etc.)
  - The error message

Sites updated (18 files, ~30 catches):

  Boot path:
    src/index.ts          — ensureEnvSiteAdmin, notifySystemdReady

  Auth surface:
    src/middleware/auth.ts — OAuth + PAT lastUsedAt updates
    src/routes/hooks.ts    — PAT lastUsedAt update
    src/routes/auth.tsx    — admin bootstrap on register

  Git surface:
    src/routes/git.ts          — clone tracking + push-reject audit
    src/hooks/post-receive.ts  — getDefaultBranch fallback
    src/routes/web.tsx         — view tracking

  AI / observability:
    src/routes/pulls.tsx     — auto-merge trigger, PR risk, AI re-review
    src/routes/issues.tsx    — issue triage triggers
    src/routes/advisories.tsx — seedAdvisories

  Auto-repair (highest impact — observability of self-healing path):
    src/lib/auto-repair.ts   — 6 sites: worktree cleanup, mkdir, audit
                               inserts for mechanical + AI repair tiers

  Background workers:
    src/lib/workflow-runner.ts — 4 sites: tmp dir cleanup
    src/lib/sso.ts             — 2 sites: stale link cleanup
    src/lib/mcp-tools.ts       — 2 sites: gluecron notify publishes
    src/lib/merge-resolver.ts  — worktree cleanup
    src/lib/actions/{download,upload,cache}-action.ts — tmp file cleanup

Left untouched (legitimate null-return fallbacks):
  - pr-risk.ts, ai-tests.tsx, dashboard.tsx, admin-ops.tsx, pulls.tsx
    .catch(() => null) sites — these treat absence as a legitimate
    return value (cache miss, feature unavailable, file not found).
    Not silent failures in the user-impact sense.
  - pwa.ts SW source — inside the service-worker body string, not a
    Node-side catch.

Net effect: when something stops working in prod, `journalctl -u
gluecron | grep '\[<component>\]'` now tells you what failed and why,
instead of the previous "site is broken, no idea where."

Tests: 2000/2000 pass. tsc clean. No behaviour change — every catch
still swallows the error for control flow; only adds a log line on the
way through.
Claude committed on May 16, 2026Parent: 976d7f7
18 files changed+227−40a28cede08af852511dcde10f619654fa7020c0bd
18 changed files+227−40
Modifiedsrc/hooks/post-receive.ts+8−1View fileUnifiedSplit
9494 // default branch. The branch case (`Main` vs `main`) is determined by
9595 // the bare repo's HEAD, not hardcoded.
9696 if (`${owner}/${repo}` === config.crontechRepo) {
97 let defaultBranch = (await getDefaultBranch(owner, repo).catch(() => null)) || "main";
97 let defaultBranch =
98 (await getDefaultBranch(owner, repo).catch((err) => {
99 console.warn(
100 `[post-receive] getDefaultBranch failed for ${owner}/${repo}, defaulting to "main":`,
101 err instanceof Error ? err.message : err
102 );
103 return null;
104 })) || "main";
98105 const targetRef = `refs/heads/${defaultBranch}`;
99106 const deployPush = refs.find(
100107 (r) => r.refName === targetRef && !r.newSha.startsWith("0000")
Modifiedsrc/index.ts+12−2View fileUnifiedSplit
3232// Site-admin bootstrap from env (SITE_ADMIN_USERNAME). Idempotent — if the
3333// user exists, they get a row in site_admins; if not, logged and retried
3434// on next boot. Background-fired so a slow DB doesn't block startup.
35void ensureEnvSiteAdmin().catch(() => {});
35void ensureEnvSiteAdmin().catch((err) => {
36 console.warn(
37 "[admin-bootstrap] ensureEnvSiteAdmin failed:",
38 err instanceof Error ? err.message : err
39 );
40});
3641
3742// Opt-in demo content seed on boot (DEMO_SEED_ON_BOOT=1). Idempotent, never
3843// throws — safe to run on every start. Block L3 layers extra activity (more
5863
5964// BLOCK N2 — tell systemd we're ready. No-op when NOTIFY_SOCKET is unset
6065// (dev / non-systemd hosts). Fire-and-forget; never throws.
61void notifySystemdReady().catch(() => {});
66void notifySystemdReady().catch((err) => {
67 console.warn(
68 "[systemd] notifySystemdReady failed:",
69 err instanceof Error ? err.message : err
70 );
71});
6272
6373export default {
6474 port: config.port,
Modifiedsrc/lib/actions/cache-action.ts+8−1View fileUnifiedSplit
153153 try {
154154 await Bun.file(tmpPath).exists();
155155 // Best-effort cleanup; ignore errors.
156 await import("fs/promises").then((fs) => fs.unlink(tmpPath)).catch(() => {});
156 await import("fs/promises")
157 .then((fs) => fs.unlink(tmpPath))
158 .catch((err) => {
159 console.warn(
160 `[cache-action] tmpPath cleanup failed for ${tmpPath}:`,
161 err instanceof Error ? err.message : err
162 );
163 });
157164 } catch {
158165 /* noop */
159166 }
Modifiedsrc/lib/actions/download-artifact-action.ts+6−1View fileUnifiedSplit
6464 } finally {
6565 try {
6666 const fs = await import("fs/promises");
67 await fs.unlink(tmpPath).catch(() => {});
67 await fs.unlink(tmpPath).catch((err) => {
68 console.warn(
69 `[download-artifact] tmpPath cleanup failed for ${tmpPath}:`,
70 err instanceof Error ? err.message : err
71 );
72 });
6873 } catch {
6974 /* noop */
7075 }
Modifiedsrc/lib/actions/upload-artifact-action.ts+6−1View fileUnifiedSplit
7070 } finally {
7171 try {
7272 const fs = await import("fs/promises");
73 await fs.unlink(tmpPath).catch(() => {});
73 await fs.unlink(tmpPath).catch((err) => {
74 console.warn(
75 `[upload-artifact] tmpPath cleanup failed for ${tmpPath}:`,
76 err instanceof Error ? err.message : err
77 );
78 });
7479 } catch {
7580 /* noop */
7681 }
Modifiedsrc/lib/auto-repair.ts+45−7View fileUnifiedSplit
9393}
9494
9595async function cleanupWorktree(repoDir: string, worktree: string): Promise<void> {
96 // Cleanup failures are non-fatal but should be visible — stale worktrees
97 // eat disk space and `git worktree prune` won't recover them if the
98 // directory itself is left behind.
9699 await exec(["git", "worktree", "remove", "--force", worktree], {
97100 cwd: repoDir,
98 }).catch(() => {});
99 await rm(worktree, { recursive: true, force: true }).catch(() => {});
101 }).catch((err) => {
102 console.warn(
103 `[auto-repair] git worktree remove failed for ${worktree}:`,
104 err instanceof Error ? err.message : err
105 );
106 });
107 await rm(worktree, { recursive: true, force: true }).catch((err) => {
108 console.warn(
109 `[auto-repair] rm worktree failed for ${worktree}:`,
110 err instanceof Error ? err.message : err
111 );
112 });
100113}
101114
102115/**
113126 for (const patch of patches) {
114127 const fullPath = join(worktree, patch.path);
115128 try {
116 await mkdir(join(fullPath, "..").replace(/[^/]+\/\.\.$/, ""), { recursive: true }).catch(() => {});
129 await mkdir(join(fullPath, "..").replace(/[^/]+\/\.\.$/, ""), { recursive: true }).catch((err) => {
130 console.warn(
131 `[auto-repair] mkdir parent failed for ${patch.path}:`,
132 err instanceof Error ? err.message : err
133 );
134 });
117135 await writeFile(fullPath, patch.content, "utf8");
118136 filesChanged.push(patch.path);
119137 } catch (err) {
417435 filesChanged: mech.filesChanged,
418436 commitSha: mech.commitSha,
419437 outcome: "success",
420 }).catch(() => {});
438 }).catch((err) => {
439 console.warn(
440 "[auto-repair] mechanical-tier audit-log insert failed:",
441 err instanceof Error ? err.message : err
442 );
443 });
421444 return {
422445 attempted: true,
423446 success: true,
438461 filesChanged: mech.filesChanged,
439462 commitSha: null,
440463 outcome: "failed",
441 }).catch(() => {});
464 }).catch((err) => {
465 console.warn(
466 "[auto-repair] mechanical-tier failure-audit insert failed:",
467 err instanceof Error ? err.message : err
468 );
469 });
442470 }
443471
444472 // ── Tier 2: AI-powered repair via Claude Sonnet
517545 filesChanged: result.filesChanged,
518546 commitSha: null,
519547 outcome: "failed",
520 }).catch(() => {});
548 }).catch((err) => {
549 console.warn(
550 "[auto-repair] ai-tier failure-audit insert failed:",
551 err instanceof Error ? err.message : err
552 );
553 });
521554 return {
522555 attempted: true,
523556 success: false,
535568 filesChanged: result.filesChanged,
536569 commitSha: result.sha ?? null,
537570 outcome: "success",
538 }).catch(() => {});
571 }).catch((err) => {
572 console.warn(
573 "[auto-repair] ai-tier success-audit insert failed:",
574 err instanceof Error ? err.message : err
575 );
576 });
539577 return {
540578 attempted: true,
541579 success: true,
Modifiedsrc/lib/mcp-tools.ts+12−2View fileUnifiedSplit
586586 title: `New issue: ${title}`,
587587 url: issueUrl(owner, repo, inserted.number),
588588 repositoryId: gate.repoId,
589 }).catch(() => {});
589 }).catch((err) => {
590 console.warn(
591 `[mcp] issue_opened notify failed for ${owner}/${repo}#${inserted.number}:`,
592 err instanceof Error ? err.message : err
593 );
594 });
590595 }
591596
592597 return {
844849 title: `New PR: ${title}`,
845850 url: prUrl(owner, repo, pr.number),
846851 repositoryId: gate.repoId,
847 }).catch(() => {});
852 }).catch((err) => {
853 console.warn(
854 `[mcp] pr_opened notify failed for ${owner}/${repo}#${pr.number}:`,
855 err instanceof Error ? err.message : err
856 );
857 });
848858 }
849859
850860 return { number: pr.number, url: prUrl(owner, repo, pr.number) };
Modifiedsrc/lib/merge-resolver.ts+6−1View fileUnifiedSplit
159159 return { success: true, resolvedFiles, commitSha: sha.trim() };
160160 } finally {
161161 // Clean up the worktree
162 await exec(["git", "worktree", "remove", "--force", worktree], { cwd: repoDir }).catch(() => {});
162 await exec(["git", "worktree", "remove", "--force", worktree], { cwd: repoDir }).catch((err) => {
163 console.warn(
164 `[merge-resolver] worktree cleanup failed for ${worktree}:`,
165 err instanceof Error ? err.message : err
166 );
167 });
163168 }
164169}
165170
Modifiedsrc/lib/sso.ts+12−2View fileUnifiedSplit
320320 await db
321321 .delete(ssoUserLinks)
322322 .where(eq(ssoUserLinks.subject, claims.sub))
323 .catch(() => {});
323 .catch((err) => {
324 console.warn(
325 "[sso] stale link cleanup failed:",
326 err instanceof Error ? err.message : err
327 );
328 });
324329 }
325330
326331 // 2. Domain gate
569574 await db
570575 .delete(ssoUserLinks)
571576 .where(eq(ssoUserLinks.subject, subject))
572 .catch(() => {});
577 .catch((err) => {
578 console.warn(
579 "[sso] orphan link cleanup failed:",
580 err instanceof Error ? err.message : err
581 );
582 });
573583 }
574584
575585 // 2. Domain gate (only meaningful if admin set one)
Modifiedsrc/lib/workflow-runner.ts+24−4View fileUnifiedSplit
415415 const cloneExit = await cloneProc.exited;
416416 clearTimeout(cloneTimer);
417417 if (cloneExit !== 0) {
418 await rm(dir, { recursive: true, force: true }).catch(() => {});
418 await rm(dir, { recursive: true, force: true }).catch((err) => {
419 console.warn(
420 `[workflow-runner] cleanup rm failed for ${dir}:`,
421 err instanceof Error ? err.message : err
422 );
423 });
419424 return { error: `git clone failed: ${truncate(cloneErr, 2048)}` };
420425 }
421426 } catch (err) {
422 await rm(dir, { recursive: true, force: true }).catch(() => {});
427 await rm(dir, { recursive: true, force: true }).catch((err) => {
428 console.warn(
429 `[workflow-runner] cleanup rm failed for ${dir}:`,
430 err instanceof Error ? err.message : err
431 );
432 });
423433 return { error: `git clone spawn failed: ${(err as Error).message}` };
424434 }
425435
437447 .catch(() => "");
438448 const coExit = await coProc.exited;
439449 if (coExit !== 0) {
440 await rm(dir, { recursive: true, force: true }).catch(() => {});
450 await rm(dir, { recursive: true, force: true }).catch((err) => {
451 console.warn(
452 `[workflow-runner] cleanup rm failed for ${dir}:`,
453 err instanceof Error ? err.message : err
454 );
455 });
441456 return { error: `git checkout ${target} failed: ${truncate(coErr, 2048)}` };
442457 }
443458 } catch (err) {
444 await rm(dir, { recursive: true, force: true }).catch(() => {});
459 await rm(dir, { recursive: true, force: true }).catch((err) => {
460 console.warn(
461 `[workflow-runner] cleanup rm failed for ${dir}:`,
462 err instanceof Error ? err.message : err
463 );
464 });
445465 return { error: `git checkout spawn failed: ${(err as Error).message}` };
446466 }
447467 }
Modifiedsrc/middleware/auth.ts+14−3View fileUnifiedSplit
5959 .where(eq(users.id, row.userId))
6060 .limit(1);
6161 if (!user) return null;
62 // Best-effort: update lastUsedAt. Never fail auth on this.
62 // Best-effort: update lastUsedAt. Never fail auth on this — but log so
63 // a persistent DB write failure surfaces (was a silent .catch(() => {}).)
6364 db
6465 .update(oauthAccessTokens)
6566 .set({ lastUsedAt: new Date() })
6667 .where(eq(oauthAccessTokens.id, row.id))
67 .catch(() => {});
68 .catch((err) => {
69 console.warn(
70 "[auth] oauth lastUsedAt update failed:",
71 err instanceof Error ? err.message : err
72 );
73 });
6874 return {
6975 user,
7076 scopes: row.scopes ? row.scopes.split(/\s+/).filter(Boolean) : [],
103109 .update(apiTokens)
104110 .set({ lastUsedAt: new Date() })
105111 .where(eq(apiTokens.id, row.id))
106 .catch(() => {});
112 .catch((err) => {
113 console.warn(
114 "[auth] PAT lastUsedAt update failed:",
115 err instanceof Error ? err.message : err
116 );
117 });
107118 return {
108119 user,
109120 scopes: row.scopes ? row.scopes.split(/[,\s]+/).filter(Boolean) : [],
Modifiedsrc/routes/advisories.tsx+6−1View fileUnifiedSplit
281281 )}`
282282 );
283283 }
284 await seedAdvisories().catch(() => {});
284 await seedAdvisories().catch((err) => {
285 console.warn(
286 "[advisories] seedAdvisories failed:",
287 err instanceof Error ? err.message : err
288 );
289 });
285290 const result = await scanRepositoryForAlerts(repo.id);
286291 await audit({
287292 userId: user.id,
Modifiedsrc/routes/auth.tsx+8−3View fileUnifiedSplit
210210
211211 // If username matches SITE_ADMIN_USERNAME env, grant site admin instantly
212212 // so the operator doesn't have to wait for the next boot's bootstrap pass.
213 await import("../lib/admin-bootstrap").then((m) =>
214 m.ensureEnvAdminOnRegister({ userId: user.id, username })
215 ).catch(() => {});
213 await import("../lib/admin-bootstrap")
214 .then((m) => m.ensureEnvAdminOnRegister({ userId: user.id, username }))
215 .catch((err) => {
216 console.warn(
217 `[admin-bootstrap] ensureEnvAdminOnRegister failed for ${username}:`,
218 err instanceof Error ? err.message : err
219 );
220 });
216221
217222 // Create session
218223 const token = generateSessionToken();
Modifiedsrc/routes/git.ts+14−3View fileUnifiedSplit
6565 if (!(await repoExists(owner, repo))) {
6666 return c.text("Repository not found", 404);
6767 }
68 // F1 — fire-and-forget clone tracking.
68 // F1 — fire-and-forget clone tracking. Log on failure so traffic-stats
69 // gaps are diagnosable (was a silent .catch(() => {}).)
6970 trackByName(owner, repo, "clone", {
7071 ip: c.req.header("x-forwarded-for") || c.req.header("x-real-ip") || null,
7172 userAgent: c.req.header("user-agent") || null,
72 }).catch(() => {});
73 }).catch((err) => {
74 console.warn(
75 `[git] clone tracking failed for ${owner}/${repo}:`,
76 err instanceof Error ? err.message : err
77 );
78 });
7379 return serviceRpc(owner, repo, "git-upload-pack", c.req.raw.body);
7480});
7581
118124 refs: refs.map((r) => r.refName),
119125 pusherSource: pusher?.source || "anonymous",
120126 },
121 }).catch(() => {});
127 }).catch((err) => {
128 console.warn(
129 `[git] push.rejected audit write failed for ${owner}/${repo}:`,
130 err instanceof Error ? err.message : err
131 );
132 });
122133 // Returning 403 with a plain-text body — git smart-HTTP clients
123134 // surface the body to the user (`remote: ` prefix). Existing
124135 // behaviour for repos with no policy is unchanged.
Modifiedsrc/routes/hooks.ts+8−2View fileUnifiedSplit
345345 if (row.expiresAt && row.expiresAt < new Date()) {
346346 return { ok: false, error: "Token expired" };
347347 }
348 // Update last-used timestamp (fire and forget)
348 // Update last-used timestamp (fire and forget). Log persistent
349 // failures so DB issues surface (was silent .catch(() => {}).)
349350 db.update(apiTokens)
350351 .set({ lastUsedAt: new Date() })
351352 .where(eq(apiTokens.id, row.id))
352 .catch(() => {});
353 .catch((err) => {
354 console.warn(
355 "[hooks] PAT lastUsedAt update failed:",
356 err instanceof Error ? err.message : err
357 );
358 });
353359 return { ok: true, userId: row.userId };
354360 } catch {
355361 return { ok: false, error: "Auth lookup failed" };
Modifiedsrc/routes/issues.tsx+12−2View fileUnifiedSplit
278278 authorId: user.id,
279279 title,
280280 body: issueBody,
281 }).catch(() => {});
281 }).catch((err) => {
282 console.warn(
283 `[issue-triage] triage trigger failed for issue ${issue.id}:`,
284 err instanceof Error ? err.message : err
285 );
286 });
282287
283288 return c.redirect(
284289 `/${ownerName}/${repoName}/issues/${issue.number}`
663668 body: issue.body || "",
664669 },
665670 { force: true }
666 ).catch(() => {});
671 ).catch((err) => {
672 console.warn(
673 `[issue-triage] re-triage failed for issue ${issue.id}:`,
674 err instanceof Error ? err.message : err
675 );
676 });
667677
668678 return c.redirect(
669679 `/${ownerName}/${repoName}/issues/${issueNum}?info=${encodeURIComponent(
Modifiedsrc/routes/pulls.tsx+20−3View fileUnifiedSplit
599599 }).catch((err) => console.error("[pr-triage] Failed:", err));
600600
601601 // R3 — fast-lane auto-merge evaluation. Fires after AI review lands.
602 import("../lib/auto-merge").then((m) => m.tryAutoMergeNow(pr.id)).catch(() => {});
602 import("../lib/auto-merge")
603 .then((m) => m.tryAutoMergeNow(pr.id))
604 .catch((err) => {
605 console.warn(
606 `[auto-merge] tryAutoMergeNow failed for PR ${pr.id}:`,
607 err instanceof Error ? err.message : err
608 );
609 });
603610
604611 return c.redirect(`/${ownerName}/${repoName}/pulls/${pr.number}`);
605612 }
684691 prRisk = await getCachedPrRisk(pr.id).catch(() => null);
685692 if (!prRisk) {
686693 prRiskCalculating = true;
687 void computePrRiskForPullRequest(pr.id).catch(() => {});
694 void computePrRiskForPullRequest(pr.id).catch((err) => {
695 console.warn(
696 `[pr-risk] computePrRiskForPullRequest failed for PR ${pr.id}:`,
697 err instanceof Error ? err.message : err
698 );
699 });
688700 }
689701 }
690702
13831395 pr.baseBranch,
13841396 pr.headBranch,
13851397 { force: true }
1386 ).catch(() => {});
1398 ).catch((err) => {
1399 console.warn(
1400 `[ai-rereview] triggerAiReview failed for PR ${pr.id}:`,
1401 err instanceof Error ? err.message : err
1402 );
1403 });
13871404
13881405 return c.redirect(
13891406 `/${ownerName}/${repoName}/pulls/${prNum}?info=${encodeURIComponent(
Modifiedsrc/routes/web.tsx+6−1View fileUnifiedSplit
489489 ip: c.req.header("x-forwarded-for") || c.req.header("x-real-ip") || null,
490490 userAgent: c.req.header("user-agent") || null,
491491 referer: c.req.header("referer") || null,
492 }).catch(() => {});
492 }).catch((err) => {
493 console.warn(
494 `[web] view tracking failed for ${owner}/${repo}:`,
495 err instanceof Error ? err.message : err
496 );
497 });
493498
494499 if (!(await repoExists(owner, repo))) {
495500 return c.html(
496501