SRE Practice: Incidents & Reliability
20 min
Day 109: Alert design: symptoms, not causes
Alert on what users experience, not on internal causes
A cause-based alert ("CPU > 80%") fires constantly for conditions that never actually hurt users, and misses problems that don't show up as a resource spike at all. A symptom-based alert ("error rate > 1% for 5 minutes", "p99 latency > 2s") fires exactly when users are actually affected — regardless of which internal cause is responsible this time.
This is Phase 17's burn-rate alerts, generalized
The SLO/burn-rate alerting from Day 108 is really a special case of this broader principle: alert on the user-facing symptom (SLI violation), and let your dashboards (not your pages) help you find the cause afterward.
- Cause-based (avoid as a primary paging trigger): "disk usage > 90%", "CPU > 80%"
- Symptom-based (page on these): "error rate above threshold", "latency SLO burn rate too high", "queue depth growing unboundedly"
Key terms
- Symptom-based alerting
- Alerting on user-facing impact rather than internal resource conditions.
Why is "page the on-call engineer when CPU exceeds 80%" usually poor alert design?
