Published - July 16, 2026
Page on user impact, not noise—burn-rate alerts, clear ownership, and catalogs that keep on-call human attention for real decisions.
On-call teams drown when every CPU spike, disk warning, and dependency blip pages someone. Alert fatigue is not a soft culture problem—it is a reliability failure: engineers silence noisy channels, miss the rare signal that matters, and burn out. The more metrics you collect without policy, the worse the noise becomes.
SLO-based alerting flips the model. You page on symptoms that threaten user experience or error budgets, not on every cause. Infrastructure signals still matter for diagnosis, but they become tickets, dashboards, or daytime follow-ups unless they are proven precursors to customer impact. The goal is a high-signal paging system that preserves human attention for decisions only humans can make.
This guide is for SRE and service owners redesigning alerts after incidents or after a quarter of ignored pages. You will learn symptom vs cause alerts, multi-window burn rates, ownership hygiene, and review rituals that keep alert catalogs honest.
Start from service-level objectives: availability, latency, and correctness for the journeys customers pay for. Derive paging alerts from burn rates—fast burn for acute outages, slow burn for creeping regressions—using multi-window policies so brief spikes do not wake people at 3 a.m. Pair each page with a runbook link, severity, and escalation path.
Classify remaining alerts into three buckets: page now, ticket within business hours, and observe-only. Cause-based alerts (disk full, pod restart loops) move to ticket or observe unless they have a documented correlation to SLO burn. Require an owner team and expiry date on every alert rule; orphaned alerts are deleted or muted after review.
Instrument the alerting system itself: pages per week per service, acknowledge time, false-positive rate, and "no action taken" outcomes. Use those metrics in monthly reliability reviews. When false positives exceed a threshold, fix or remove the rule before adding new ones.
Multi-window burn-rate alerts (for example, 2% budget in 1 hour and 5% in 6 hours) reduce flapping while catching both explosions and slow leaks. Tune windows to your error budget period and traffic patterns; weekend troughs need different sensitivity than peak sales events. Document the math so product partners understand why a page fired.
For dependencies, prefer customer-impact alerts on your service over paging for every upstream 5xx. Upstream owners keep their own symptom alerts; you alert when your SLO burns because of them and include dependency context in the page. Shared platforms publish golden signals and suppress duplicate pages via routing or grouping.
Human factors matter: rotation length, secondary coverage, and quiet hours for non-critical services. Game the system with alert reviews after every SEV: which page helped, which wasted time, which was missing. Treat alert debt like tech debt with a backlog and capacity allocation.
SLO-only paging can hide impending capacity cliffs if you never look at predictive tickets. Keep a small set of early-warning tickets for saturation. Another pitfall is copying vendor default alert packs—they optimize for coverage, not for your team's sleep.
Do not confuse silence with health. After aggressive pruning, validate with chaos drills and synthetic checks that true outages still page within your MTTA targets.
A payments team cut weekly pages from 47 to 9 after rewriting alerts around checkout availability and latency burn rates. Cause alerts for Redis restarts became daytime tickets. MTTA for true SEVs improved because engineers stopped muting the primary channel.
Alert quality compounds. Inventory one noisy service this week, convert its pages to SLO burn rules, and measure false positives for two rotations before expanding the program.