Too many connections
Restoring service when the connection limit is reached, and finding which consumer caused it.
Symptom
FATAL: sorry, too many clients already -- PostgreSQL
ERROR 1040 (HY000): Too many connections -- MySQLNew connections are refused while existing ones keep working, so the failure is partial: some requests succeed, new instances cannot start, and monitoring may go blind at the same moment.
Impact
Partial outage that worsens as instances restart and fail to reconnect.
Triage
psql -h db.internal -U postgres -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"Uses the reserved superuser connections. The state distribution tells you which problem you have.
-- Who is holding them.
SELECT usename, application_name, client_addr, state, count(*)
FROM pg_stat_activity GROUP BY 1,2,3,4 ORDER BY 5 DESC;-- MySQL
SHOW STATUS LIKE 'Threads_connected';
SELECT user, host, command, count(*) FROM information_schema.processlist
GROUP BY 1,2,3 ORDER BY 4 DESC;Mitigation
Reclaim connections from the safest category first.
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle in transaction' AND now() - state_change > interval '5 min';Terminates sessions holding open transactions while doing nothing. Their transactions are rolled back — which is what should have happened anyway — but confirm none is a legitimate long-running job before running it.
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND now() - state_change > interval '30 min';Idle connections are safe to close; clients reconnect. Do this only after the idle-in-transaction sweep, since those are the harmful ones.
Reduce the source. Scale down the noisiest consumer, pause background jobs, or restart the
application tier that is leaking. client_addr in the query above identifies which one.
If queries are genuinely active and slow, the connection limit is a symptom — see Slow queries.
Verification
- Connection count is comfortably below the limit.
- New connections succeed from an application host.
- Application error rates return to baseline.
Follow-up
- Put a pooler in front and cap the total. This is the only durable fix — see PgBouncer.
- Set timeouts so the situation self-corrects:
ALTER ROLE app_service SET idle_in_transaction_session_timeout = '30s';
ALTER ROLE app_service SET statement_timeout = '2s';- Give background jobs their own smaller pool, so they cannot consume the user-facing budget.
- Set a short pool acquisition timeout, so requests fail fast instead of piling up.
- Alert at 80% of the limit, not at exhaustion.