Prometheus Monitoring
Scraping database metrics with exporters, the permissions they need, and keeping cardinality under control.
Prometheus scrapes metrics endpoints on an interval and stores them as time series. For databases, the endpoint is usually an exporter that translates the engine's own statistics.
Exporters
# prometheus.yml
scrape_configs:
- job_name: postgres
scrape_interval: 15s
static_configs:
- targets: ['db-primary:9187', 'db-replica-a:9187']
labels: { cluster: shop, env: prod }Least privilege for the exporter
-- PostgreSQL: pg_monitor grants read access to statistics without data access.
CREATE ROLE metrics LOGIN PASSWORD :'password';
GRANT pg_monitor TO metrics;-- MySQL
CREATE USER 'metrics'@'localhost' IDENTIFIED BY '…';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON performance_schema.* TO 'metrics'@'localhost';Custom queries
Exporters cover server-level counters. Table-level facts — bloat, per-table dead tuples, replication slot retention — usually need custom queries:
# postgres_exporter queries.yaml
pg_replication_slots:
query: |
SELECT slot_name, active::int AS active,
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes
FROM pg_replication_slots
metrics:
- slot_name: { usage: "LABEL" }
- active: { usage: "GAUGE", description: "Whether the slot has a connected consumer" }
- retained_bytes: { usage: "GAUGE", description: "WAL retained for this slot" }Cardinality
Prometheus is a time-series database, and everything in Cardinality applies. The common database-monitoring mistakes:
- Per-query-hash metrics. A label per normalised query multiplies series and grows unbounded.
- Per-table metrics on a schema with thousands of tables.
- Per-key metrics from
redis_exporter. - Container ids or pod names as labels on a frequently redeployed service.
Drop what you do not query:
metric_relabel_configs:
- source_labels: [__name__]
regex: 'pg_stat_statements_.*'
action: dropScrape interval
Fifteen seconds is a reasonable default. Shorter intervals multiply storage and load on the
exporter; longer ones miss short spikes. Match the interval to the alert you intend to write — an
alert with for: 2m gains nothing from five-second scrapes.