Incident response
Troubleshooting playbooks
Each playbook starts from a symptom you can observe from monitoring, not from a cause you have already diagnosed. Diagnosis commands are marked read-only or destructive so nothing surprising happens at 3am.
Cassandra
ClickHouse
Elasticsearch
General
- Backup failedA backup job failed or stopped running — establishing current exposure and restoring protection quickly.
- Cache hit ratio droppedThe cache hit ratio fell — determining whether it matters, what changed, and what to do about it.
- Database is unavailableThe database refuses connections or does not respond — establishing whether it is down, unreachable or saturated, and restoring service.
- Deadlocks increasedA rising deadlock rate — finding the conflicting statements and the lock ordering that causes them.
- Database is running out of disk spaceFreeing space safely when a database volume is filling, and identifying which of the usual causes it is.
- High CPU usageDetermining whether high CPU is a problem, finding the queries responsible, and reducing load safely.
- High memory usageDistinguishing cache pressure from an allocation problem, and preventing an out-of-memory kill.
- Replica is unavailableA replica is down or disconnected — assessing the risk to the primary and deciding between repair and rebuild.
- Replication lagIdentifying which component of replication is lagging, and reducing it without making the primary the next problem.
- Restore failedA restore is failing during an incident — working through the causes without destroying the remaining options.
- Slow queriesFinding which queries degraded, why they changed, and what to do about it during an incident.
- Too many connectionsRestoring service when the connection limit is reached, and finding which consumer caused it.