CodeIssuesDiscussionsWikiPull RequestsProjectsCommitsActionsReleasesContributorsPulse● GatesSecuritySettingsDeploymentsPipelineInsightsAgents✨ Explain✨ Ask AI✨ Workspace✨ Spec✨ Tests▓ Debt Map✨ NL Search🏛 Archaeology
Blame · Line-by-line history
README.md
Each line is annotated with the commit that last touched it. Click any SHA to jump to that commit and see the surrounding change.
| 6d2b36e | 1 | # Alerting rules |
| 2 | ||
| 3 | Prometheus-format alert rules for gluecron. Provider-neutral — works | |
| 4 | with Prometheus + Alertmanager, Grafana Mimir + Grafana OnCall, | |
| 5 | VictoriaMetrics + vmalert, or Thanos Ruler. | |
| 6 | ||
| 7 | ## File | |
| 8 | ||
| 9 | - `gluecron.rules.yml` — five groups: availability, errors, latency, | |
| 10 | autopilot, resources. Three severity levels: `critical`, `warning`, `info`. | |
| 11 | ||
| 12 | ## Wire-up | |
| 13 | ||
| 14 | ### Prometheus + Alertmanager | |
| 15 | ||
| 16 | 1. Scrape gluecron: | |
| 17 | ```yaml | |
| 18 | # prometheus.yml | |
| 19 | scrape_configs: | |
| 20 | - job_name: gluecron | |
| 21 | metrics_path: /metrics | |
| 22 | static_configs: | |
| 23 | - targets: ['gluecron.com:443'] | |
| 24 | scheme: https | |
| 25 | ``` | |
| 26 | 2. Include the rules file: | |
| 27 | ```yaml | |
| 28 | # prometheus.yml | |
| 29 | rule_files: | |
| 30 | - /etc/prometheus/gluecron.rules.yml | |
| 31 | ``` | |
| 32 | 3. Point Alertmanager at Slack / PagerDuty / email. Route `severity=critical` | |
| 33 | to the pager channel, `severity=warning` to Slack-warning. | |
| 34 | ||
| 35 | ### Blackbox probes for /readyz | |
| 36 | ||
| 37 | One rule (`GluecronNotReady`) assumes you also run the Prometheus | |
| 38 | [blackbox_exporter](https://github.com/prometheus/blackbox_exporter) probing | |
| 39 | `https://gluecron.com/readyz`. If you skip blackbox, that rule will never | |
| 40 | fire — safe, just redundant with `GluecronDown`. | |
| 41 | ||
| 42 | Minimal blackbox scrape: | |
| 43 | ```yaml | |
| 44 | - job_name: gluecron-blackbox | |
| 45 | metrics_path: /probe | |
| 46 | params: | |
| 47 | module: [http_2xx] | |
| 48 | static_configs: | |
| 49 | - targets: | |
| 50 | - https://gluecron.com/readyz | |
| 51 | relabel_configs: | |
| 52 | - source_labels: [__address__] | |
| 53 | target_label: __param_target | |
| 54 | - source_labels: [__param_target] | |
| 55 | target_label: instance | |
| 56 | - target_label: __address__ | |
| 57 | replacement: blackbox-exporter:9115 | |
| 58 | ``` | |
| 59 | ||
| 60 | ## Severity routing | |
| 61 | ||
| 62 | | Severity | Where it goes | Examples | | |
| 63 | |------------|------------------------|-----------------------------------------| | |
| 64 | | `critical` | page (SMS / phone) | `GluecronDown`, `GluecronSpike5xx` | | |
| 65 | | `warning` | Slack / email | `GluecronHigh5xxRate`, `GluecronHighLatencyP95` | | |
| 66 | | `info` | dashboard only | (reserved; none yet) | | |
| 67 | ||
| 68 | ## Tuning | |
| 69 | ||
| 70 | - `GluecronHigh5xxRate` fires at >5% for 10m. Tighten once baseline is known. | |
| 71 | - `GluecronHighLatencyP95` fires at >2s for 15m. Tune after the first | |
| 72 | 24 hours of real traffic. | |
| 73 | - `GluecronAutopilotStalled` expects `autopilot_last_run_timestamp_seconds` | |
| 74 | metric. If the autopilot ticker doesn't emit this gauge yet, either | |
| 75 | add it to `src/lib/autopilot.ts` (one `gauge.set(Date.now()/1000)` | |
| 76 | per tick) or drop this rule until the metric exists. | |
| 77 | ||
| 78 | ## Validation | |
| 79 | ||
| 80 | Before rolling out: | |
| 81 | ```sh | |
| 82 | promtool check rules infra/alerts/gluecron.rules.yml | |
| 83 | ``` | |
| 84 | ||
| 85 | No output = valid. |