Monitoring
What to collect from a database, why saturation metrics matter more than averages, and how to tell a monitoring gap from a healthy system.
Monitoring exists to answer two questions: is the database healthy right now, and what changed before it stopped being healthy. Everything you collect should serve one of those.
The four groups worth collecting
Traffic and latency. Queries per second and query duration percentiles, ideally split by statement or normalised query. Averages are close to useless — a p50 of 2 ms with a p99 of 6 seconds looks fine on an average and is a broken system.
Saturation. Every resource with a hard ceiling: connections used versus maximum, disk used versus capacity, memory, IOPS, replication slots. Saturation metrics are the ones that predict outages, because crossing 100% is not gradual.
Errors. Failed connections, deadlocks, serialization failures, replication errors, backup failures. A rising error rate usually precedes a visible outage.
Replication and durability. Lag per replica, last successful backup, WAL or binlog retention, and the status of any replication slot. See Important Database Metrics.
Collect from the database, not only about it
Host metrics tell you a machine is busy. Engine metrics tell you why. Every engine exposes internal state that no host metric approximates:
Active sessions by wait event
Source: pg_stat_activity, performance_schema
Distinguishes "waiting on a lock" from "waiting on I/O" from "actually computing" — three problems with three different fixes.
Cache hit ratio
Source: pg_statio_user_tables, Innodb_buffer_pool_reads
Falling hit ratio is the earliest signal that the working set no longer fits in memory.
Replication lag
Source: pg_stat_replication, SHOW REPLICA STATUS, rs.status()
Both in bytes and in seconds; bytes catch a stalled replica that reports zero seconds.
Dead rows and vacuum age
Source: pg_stat_user_tables
Predicts bloat and transaction ID wraparound long before either becomes an incident.
Retention and resolution
Keep enough history to answer "was this normal last month". Fifteen-second resolution for a few days and one-minute resolution for a year covers both incident investigation and capacity forecasting.
Instrument the client side too
The database's own view is incomplete. A query that takes 2 ms in the database and 400 ms as seen by the application is a connection pool or network problem, and only client-side measurement reveals it. Record query duration, connection acquisition time and pool saturation in the application. See Distributed Tracing.