Grafana Dashboards

A dashboard's job is to answer a question quickly. Most database dashboards answer no question at all, because they display everything the exporter provides.

Three dashboards, not thirty panels

Overview — one screen, answering "is this database healthy". Latency percentiles, traffic, error rate, saturation of every bounded resource, replication lag, backup freshness. If this screen is green, stop looking.

Diagnostic — reached when the overview is not green. Wait events, lock waits, slow query statistics, cache hit ratios, background maintenance backlog, per-table detail.

Capacity — reviewed monthly, not during incidents. Long-range growth of data volume, connection high-water marks, cardinality, and projected exhaustion dates.

Panel practices

• Percentiles, not averages. Show p50, p95 and p99 on the same axis; the spread is the information.
• Show the limit alongside the value. "412 connections" means nothing; "412 of 500" means everything. Use a threshold line or a percentage.
• Annotate deploys and migrations. Half of incident diagnosis is "what changed", and an annotation answers it instantly.
• Consistent units and time ranges across panels, so two graphs can be compared at a glance.
• A stat panel for each silent failure — last successful backup, last verified restore, slot retention. Green or red, no interpretation required.

Template variables

$cluster    label_values(pg_up, cluster)
$instance   label_values(pg_up{cluster="$cluster"}, instance)

One parameterised dashboard per engine, not one dashboard per cluster. Copies drift, and a fix applied to one is not applied to the others.

Queries worth having

# Connection saturation as a fraction — immediately interpretable.
pg_stat_activity_count{cluster="$cluster"} / pg_settings_max_connections{cluster="$cluster"}
 
# Replication lag in seconds, per replica.
pg_replication_lag_seconds{cluster="$cluster"}
 
# Cache hit ratio, as a trend rather than an absolute.
rate(pg_stat_database_blks_hit[5m])
  / (rate(pg_stat_database_blks_hit[5m]) + rate(pg_stat_database_blks_read[5m]))
 
# Time since the last successful backup, in hours.
(time() - pgbackrest_last_full_backup_timestamp) / 3600

Keep them in version control

Export dashboards as JSON and store them in the repository, provisioned by configuration. A dashboard edited in the UI by whoever was on call last is not reproducible, and its history is not reviewable.