Day 100: Prometheus recording rules and Alertmanager
Recording rules: precomputing expensive queries
A dashboard re-running a complex aggregation query every few seconds, for every viewer, is wasteful. A recording rule precomputes a query on a schedule and saves the result as a new time series — dashboards and alerts then query the cheap, precomputed series instead.
groups:
- name: api_rules
rules:
- record: api:error_rate:5m
expr: sum(rate(http_requests_total{status=~'5..'}[5m])) / sum(rate(http_requests_total[5m]))Alertmanager
Prometheus evaluates alert rules and fires alerts; Alertmanager is a separate component handling what happens next — deduplicating identical alerts, grouping related ones into a single notification, routing to the right channel (Slack, PagerDuty) based on labels, and silencing/inhibiting alerts you already know about (e.g. suppress "API down" if "cluster down" is already firing).
Why grouping and inhibition matter so much
Without them, one real incident (a node dying) can trigger dozens of individually-firing alerts (every service on that node, every dependent check) — alert fatigue that trains on-call engineers to ignore pages. This is the tooling side of Phase 18's 'alert design: symptoms, not causes' principle.
Key terms
- Recording rule
- A precomputed, scheduled query saved as its own time series.
- Alertmanager
- Deduplicates, groups, routes, and silences alerts fired by Prometheus.
A single node failure triggers 40 individual alerts from 40 affected services. What Alertmanager feature addresses this specific problem?
