Alert Design

Every alert makes a claim: a human must act now. An alert that does not meet that bar reduces the credibility of the ones that do.

The three questions

Before adding an alert, answer all three:

1. What action does the recipient take? If the answer is "look at it", it is a dashboard, not an alert.
2. What happens if nobody responds for an hour? If the answer is "nothing", it is a ticket.
3. How would this fire spuriously? Design the condition so that it does not.

Anatomy of a good alert

- alert: ReplicaTooStaleForReads
  expr: pg_replication_lag_seconds{role="read-serving"} > 10
  for: 2m
  labels:
    severity: page
    team: platform-data
  annotations:
    summary: "Replica {{ $labels.instance }} lag {{ $value }}s exceeds the 10s read budget"
    description: >
      Read traffic routed to this replica may serve data older than the application's
      staleness budget. Reads may need to be moved to the primary.
    runbook: "https://dbpilot.dev/playbooks/replication-lag"

Each element earns its place: for: prevents a single scrape from paging, the threshold comes from a stated budget rather than a round number, team routes it to an owner, and the runbook link turns a page into a procedure.

Symptom over cause

Alert on what users experience, and let causes be diagnosed from dashboards.

• Symptom: "Write latency p99 above 500 ms for 5 minutes." Always worth acting on.
• Cause: "CPU above 80%." Might be fine.

The exception is silent failures — a stopped backup, an inactive replication slot, a halted replica — which have no symptom until it is far too late. Those must be alerted on directly.

Avoiding Noisy Alerts

Use for: on everything. A metric crossing a threshold for one scrape is noise.

Alert on projections for anything that fills. A disk at 85% and stable needs no page; one at 55% filling in three hours does.

- alert: DiskWillFillWithinFourHours
  expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/var/lib/postgresql"}[6h], 4*3600) < 0
  for: 15m

Group related alerts. A node failure should notify once, not eleven times. Use Alertmanager grouping by cluster and instance.

Inhibit the downstream. When a node is down, suppress its lag, connection and latency alerts — they are consequences, not new information.

inhibit_rules:
  - source_matchers: [ alertname="DatabaseDown" ]
    target_matchers: [ severity="page" ]
    equal: [ cluster, instance ]

Distinguish severities. page wakes someone; ticket creates work for tomorrow. Most alerts should be tickets.

Review monthly. List every alert that fired, and what was done. Anything that fired and required no action gets retuned or deleted. This review is the single most effective way to keep an alerting system trustworthy.

Alert on absence

- alert: DatabaseMetricsMissing
  expr: up{job="postgres"} == 0
  for: 5m

Silence is ambiguous: no metrics looks exactly like no problems. Alert on missing data, on backups that have not completed within their expected interval, and on restore verifications that have not succeeded recently.