Connection Exhaustion

Connection exhaustion is one of the most common database outages, and one of the most preventable.

What it looks like

New connections are refused while existing ones keep working, so the application fails partially: some requests succeed, new instances cannot start, and monitoring — which also needs a connection — may go blind at the same moment.

FATAL: sorry, too many clients already                 -- PostgreSQL
ERROR 1040 (HY000): Too many connections               -- MySQL
connection() error occurred during connection handshake -- MongoDB driver

The usual causes

• Pool sizes multiplied by instance count. Twenty pods with a pool of 25 is 500 connections, and a deploy that briefly doubles the pods is 1000.
• A slowdown upstream. When queries take longer, connections are held longer, and a pool that was comfortable becomes exhausted at the same request rate.
• Connection leaks. Code paths that fail without returning a connection.
• Retry storms. Failed requests retried immediately, each attempting a new connection.
• Long transactions holding connections while waiting on something external.

Diagnosing

-- PostgreSQL: who is connected, and in what state.
SELECT state, count(*) FROM pg_stat_activity GROUP BY state;
SELECT usename, application_name, client_addr, count(*)
FROM pg_stat_activity GROUP BY 1,2,3 ORDER BY 4 DESC;

-- MySQL
SHOW STATUS LIKE 'Threads_connected';
SHOW STATUS LIKE 'Max_used_connections';
SELECT user, host, count(*) FROM information_schema.processlist GROUP BY 1,2 ORDER BY 3 DESC;

A large idle in transaction count points at application code holding transactions open. A large plain idle count points at oversized pools.

Immediate mitigation

-- Terminate idle-in-transaction sessions older than a threshold.
SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE state = 'idle in transaction' AND now() - state_change > interval '5 min';

Then reduce the source: scale down the noisiest consumer, pause background jobs, or restart the application tier holding the leaked connections.

Preventing it

Cap the total with a pooler. This is the only mechanism that genuinely bounds it. PgBouncer in transaction mode or ProxySQL sits between the application and the database and enforces a ceiling regardless of how many application instances exist. See Connection Pooling.

Fail fast on acquisition. A short pool acquisition timeout means a request fails in a second rather than piling up threads waiting for a connection that is not coming.

Set idle and transaction timeouts so abandoned connections are reclaimed:

ALTER ROLE app_service SET idle_in_transaction_session_timeout = '30s';

Give each workload its own pool. Background jobs, migrations and reporting should not be able to consume the connections the user-facing path needs.

Alert on saturation, not on exhaustion. An alert at 80% of the limit is actionable; one at 100% is an incident report.