Alerting

An alert is a claim that a human must act now. Every alert that does not meet that bar reduces the credibility of the ones that do.

What deserves a page

• User-visible symptoms. Error rate, latency beyond the objective, unavailability. These are always worth waking someone for.
• Saturation with a deadline. Disk projected to fill in under four hours; connections at 90% of the limit; transaction ID age approaching wraparound. The value is the prediction, not the current number.
• Silent durability failures. Backup did not complete, replication stopped, a replication slot is inactive and retaining log. Nobody notices these until they need them, which is the worst moment.

What should not page

• CPU utilisation alone. High CPU with good latency is a well-used machine.
• A single slow query.
• Any metric crossing a round number without a consequence attached.
• Anything the on-call engineer would look at, shrug, and go back to sleep — that belongs in a dashboard or a ticket.

Write thresholds from consequences

# Bad: a number with no rationale, guaranteed to be wrong for some replica.
- alert: ReplicationLagHigh
  expr: pg_replication_lag_seconds > 30
 
# Better: derived from what the application tolerates, and sustained.
- alert: ReplicaTooStaleForReads
  expr: pg_replication_lag_seconds{role="read-serving"} > 10
  for: 2m
  annotations:
    summary: "Replica {{ $labels.instance }} exceeds the 10s read-staleness budget"
    runbook: "https://dbpilot.dev/playbooks/replication-lag"

Two details matter as much as the threshold: for: prevents a single scrape from paging, and the runbook link turns a page into a procedure.

Predict rather than observe

For anything that fills, alert on time-to-exhaustion, not on percentage. A disk at 85% that has been at 85% for a year is fine; a disk at 60% that will be full in three hours is an emergency.

- alert: DiskWillFillWithinFourHours
  expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/var/lib/postgresql"}[6h], 4*3600) < 0
  for: 15m

Keeping alerts trustworthy

• Every alert names an owner and links to a playbook. An alert nobody owns will be ignored.
• Review fired alerts monthly. Anything that fired and required no action should be retuned or deleted.
• Alert on absence. No metrics for ten minutes is itself an alert; otherwise a broken exporter looks exactly like a healthy database.
• Group related alerts. A node failure that fires eleven alerts should notify once.