Database Health Checks

A health check decides whether traffic is routed to a node. A bad one either hides a broken database or removes a healthy one from service.

Levels of check

TCP connect — the port accepts connections. Nearly useless: a database can accept connections while refusing every query.

Authenticate and ping — a connection is established and a trivial statement succeeds. This is the right default.

Query a real table — proves the storage layer responds, not just the connection handler.

Role-aware check — confirms the node is in the state the caller expects: primary for writes, replica within a staleness bound for reads.

-- Is this node a primary?  (PostgreSQL)
SELECT NOT pg_is_in_recovery() AS is_primary;
 
-- Is this replica fresh enough to serve reads?
SELECT extract(epoch FROM (now() - pg_last_xact_replay_timestamp())) < 10 AS fresh;

-- MySQL: replica health as a routing decision.
SELECT
  service_state = 'ON' AS io_ok
FROM performance_schema.replication_connection_status;

// MongoDB
db.hello().isWritablePrimary;

Liveness versus readiness

Separate them, because they trigger different actions:

• Liveness — is the process functioning? Failure means restart it. Keep it very simple; a liveness check that fails under load causes restart loops that turn a slowdown into an outage.
• Readiness — should it receive traffic now? Failure means remove it from rotation. This is where the role and freshness checks belong.

# Kubernetes: a slow database must not cause a restart loop.
livenessProbe:
  exec: { command: ["pg_isready", "-U", "postgres"] }
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 6
 
readinessProbe:
  exec: { command: ["/usr/local/bin/check-replica-fresh.sh"] }
  periodSeconds: 5
  failureThreshold: 2

Keep them cheap

Give health checks their own connection or a reserved slot, so an exhausted application pool does not make a healthy database appear dead.

Application-side checks

The application's own health endpoint should report on its database dependency without failing outright:

{
  "status": "degraded",
  "checks": {
    "database_primary": { "status": "ok", "latency_ms": 2 },
    "database_replica": { "status": "degraded", "lag_seconds": 45 },
    "cache":            { "status": "ok" }
  }
}

Returning degraded rather than unhealthy when a replica is stale lets a load balancer keep serving from the primary instead of removing the whole instance — a distinction that matters during partial failures.