An alert is a contract with a sleeping person

When an alert pages someone at night, it makes three promises:

1. Something is actually wrong, for users, right now or very soon.
2. It can’t wait until working hours.
3. A human can do something about it, and knows roughly what.

Most noisy on-call rotations come from alerts that break at least one of these. CPU at 85% on one host breaks all three. A disk filling over the next nine days breaks the second. A flapping health check on a service with five healthy replicas breaks the first.

Each broken promise costs more than one bad night. People learn that pages are often wrong, so they respond more slowly to the ones that aren’t.

AI triage doesn’t fix a bad contract

In 2026 many teams have an assistant that reads an alert, pulls recent deploys and logs, and posts a summary before the engineer finishes unlocking their phone. It’s genuinely useful. But it summarizes whatever the alert says. A page that shouldn’t have fired now comes with a confident three-paragraph explanation of why it fired. Triage tools make good alerts faster to handle. They don’t turn bad alerts into good ones.

Page on symptoms, investigate causes

Users experience symptoms: errors, slowness, things not arriving. Causes, like high CPU, a failed node or a full queue, are only worth a page if they’re about to produce a symptom.

• Page on error rate, latency and freshness, measured where users meet the system.
• Ticket on causes that need attention this week: certificates expiring in 20 days, disks growing, replicas below target but still serving.
• Dashboard everything else, for whoever is investigating.

This works the same in Prometheus, CloudWatch, New Relic or Zabbix. The tool changes the syntax, not the rule.

Burn rate instead of thresholds

“Error rate above 1% for 5 minutes” fires on short blips and misses slow leaks. If you have an SLO, alert on how fast you’re spending the error budget instead. The Google SRE Workbook’s multi-window approach is a good default for a 30-day, 99.9% SLO:

Each alert also checks a short window (for example, the last 5 minutes for the 1-hour alert) so it stops firing soon after the problem is fixed. Fast burns wake someone. Slow burns become a ticket before they become an outage.

Every page links to a next step

An alert should arrive with a link to a runbook: what this means, how to confirm it, the first two things to try, and who to escalate to. If nobody can write that runbook, the alert isn’t ready to page anyone.

Review the contract monthly

Once a month, list every page from the last 30 days and mark each one: actionable or not, urgent or not. Delete or downgrade the alerts that failed. Healthy rotations average a handful of pages per shift. Anything more and people stop listening, however good the summaries are.