Alert fatigue is the single biggest threat to on-call effectiveness. When engineers receive 50+ alerts per shift and most are noise, they stop investigating. The result is missed critical alerts that cause extended outages. This guide covers alert severity design, escalation policies, runbook integration, on-call rotation strategies, and practical techniques to reduce alert volume by 80% while catching more real incidents.
The 2025 PagerDuty State of Digital Operations report found that the average on-call engineer receives 37 alerts per shift, but only 4 require action. Teams that implemented the practices in this guide reduced alert volume to 8 per shift with 6 actionable, achieving a 75% action rate versus the industry average of 11%.
---
Alert Severity Levels
| Severity | Response Time | Notification | Example | Auto-Resolve |
| P1 - Critical | 5 minutes | Phone call + SMS | Service down for all users | No |
| P2 - High | 15 minutes | Push notification | Error rate > 5%, degraded performance | No |
| P3 - Medium | 1 hour | Slack/Email | Disk usage > 80%, certificate expiry < 14d | After 4 hours |
| P4 - Low | Next business day | Ticket/Email | Dependency deprecation, minor drift | After 24 hours |
Alert Severity Decision Tree
Use these questions to assign severity:
---
Reducing Alert Fatigue
The Three Rules of Actionable Alerts
Every alert must pass three tests before it should exist:
If an alert fails any of these tests, it should be deleted or downgraded to a dashboard metric.
<h1>Prometheus alerting rules: good vs bad examples</h1>
groups:
<h1>BAD: Not actionable, no context, too sensitive</h1>
<ul><li>name: bad_alerts</li>
</ul> rules:
- alert: HighCPU
expr: node_cpu_seconds_total > 80
for: 1m
labels:
severity: critical
<h1>GOOD: Actionable, context-rich, appropriate threshold</h1>
<ul><li>name: good_alerts</li>
</ul> rules:
- alert: APIErrorRateHigh
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: high
team: backend
runbook: "https://runbooks.internal/api-error-rate"
annotations:
summary: "API error rate is {{ $value | humanizePercentage }}"
description: "Error rate has exceeded 5% for 5 minutes. Check recent deployments and downstream dependencies."
dashboard: "https://grafana.internal/d/api-overview"
Alert Deduplication and Grouping
<h1>Alertmanager configuration</h1>
global:
resolve_timeout: 5m
route:
receiver: default
group_by: [alertname, cluster, namespace]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: pagerduty-critical
group_wait: 10s
repeat_interval: 1h
- match:
severity: high
receiver: pagerduty-high
repeat_interval: 2h
- match:
severity: medium
receiver: slack-alerts
repeat_interval: 8h
receivers:
- name: pagerduty-critical
pagerduty_configs:
- routing_key: YOUR_PD_KEY
severity: critical
- name: pagerduty-high
pagerduty_configs:
- routing_key: YOUR_PD_KEY
severity: error
- name: slack-alerts
slack_configs:
- api_url: YOUR_SLACK_WEBHOOK
channel: "#alerts"
| Technique | Alert Reduction | Implementation |
| Increase "for" duration | 30-50% | Change "for: 1m" to "for: 5m" |
| Group related alerts | 40-60% | group_by in Alertmanager |
| Remove non-actionable | 20-30% | Quarterly alert review |
| Add auto-resolve | 15-25% | resolve_timeout in Alertmanager |
| Suppress during deploys | 10-20% | Inhibition rules |
---
Runbook Integration
Every P1 and P2 alert must link to a runbook that tells the on-call engineer exactly what to do.
<h2>Runbook: API Error Rate High</h2>
<h3>Symptoms</h3>
API error rate exceeds 5% for 5+ minutes.
<h3>Likely Causes</h3>
<li>Recent deployment (most common)</li>
<li>Database connection exhaustion</li>
<li>Downstream service failure</li>
<h3>Diagnostic Steps</h3>
<li>Check recent deployments: was anything deployed in the last 30 minutes?</li>
<li>Check database connections: `SELECT count(*) FROM pg_stat_activity;`</li>
<li>Check downstream services: visit /api/health on each dependency</li>
<h3>Remediation</h3>
<li>If caused by deployment: rollback with `kubectl rollout undo deployment/api`</li>
<li>If database connections exhausted: restart PgBouncer</li>
<li>If downstream failure: enable circuit breaker fallback</li>
<h3>Escalation</h3>
If not resolved in 15 minutes, escalate to backend-oncall-secondary.
---
On-Call Rotations
| Aspect | Recommendation | Reasoning |
| Rotation length | 1 week | Long enough for context, short enough to prevent burnout |
| Team size | 4-6 engineers | Enough coverage, manageable frequency |
| Handoff time | Weekday morning | Avoid weekend handoffs |
| Shadow period | 1 week | New members shadow before primary |
| Compensation | Time off or pay | On-call is extra work, compensate it |
| Maximum pages/shift | 10 | If consistently higher, fix alerts first |
<h1>PagerDuty schedule via API</h1>
curl -X POST "https://api.pagerduty.com/schedules" \
-H "Authorization: Token token=YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"schedule": {
"type": "schedule",
"name": "Backend On-Call",
"time_zone": "America/New_York",
"schedule_layers": [{
"name": "Primary",
"start": "2026-03-10T09:00:00-04:00",
"rotation_virtual_start": "2026-03-10T09:00:00-04:00",
"rotation_turn_length_seconds": 604800,
"users": [
{"user": {"id": "P1234AB", "type": "user_reference"}},
{"user": {"id": "P5678CD", "type": "user_reference"}}
]
}]
}
}'
For the monitoring metrics that feed your alerting system, see our Kubernetes observability stack guide. Teams implementing distributed tracing alongside alerting should read our distributed tracing guide. For log-based alerting patterns, check our log management guide.
---
Frequently Asked Questions
How many alerts per shift is acceptable?
Target fewer than 10 alerts per 12-hour shift, with at least 70% being actionable. If your on-call engineers receive more than 20 alerts per shift, your alerting system needs a review. The most effective on-call teams receive 2-5 alerts per shift, all of which require and receive action.
How do I handle alert fatigue in existing systems?
Start with a two-week alert audit. Track every alert that fires: was it actionable? Was it a real problem? Did someone take action? After two weeks, delete or downgrade every alert with less than 50% action rate. Then increase the "for" duration on remaining alerts to filter transient spikes. This process typically eliminates 60-70% of alert volume.
Should I alert on symptoms or causes?
Alert on symptoms (error rate high, latency increased, queue depth growing) rather than causes (CPU high, memory usage high, disk usage high). Symptom-based alerts tell you that users are affected. Cause-based alerts fire constantly for non-impacting resource usage changes. The exception is disk usage above 90%, which should always alert because running out of disk causes cascading failures.
How do I write good runbooks?
Good runbooks answer four questions: (1) What is happening? (2) What should I check first? (3) How do I fix it? (4) When should I escalate? Write runbooks as step-by-step procedures that someone woken at 3 AM can follow without deep system knowledge. Test runbooks by having a new team member follow them during a simulated incident.
How do I handle on-call during holidays?
Offer premium compensation (2x-3x pay or equivalent time off) for holiday on-call shifts. Allow shift swaps with no management approval needed. Implement "follow the sun" rotations for global teams so no one is on-call during their local holidays. Freeze deployments during major holidays to reduce incident likelihood.