Skip to content
Navigation

Type at least two characters. Search covers page titles, headings, tags and database names.

↑ ↓ to navigateEnter to openEsc to close0 pages
Production Best Practicesintermediate

Incident Response

How to work a database incident — stabilise before diagnosing, change one thing at a time, and record what you did while you do it.

3 min readIntermediateUpdated Edit this page

Database incidents have a property that makes them dangerous: the fastest-looking action is often the one that turns a degradation into data loss.

Order of operations

  1. Establish impact. Which users, which operations, since when. This determines urgency and who needs to be told.
  2. Stabilise. Restore service before understanding the cause. Shedding load, killing a runaway query or failing over buys you the time to diagnose properly.
  3. Diagnose with read-only tools. Sessions, waits, locks, disk, replication. Never run diagnostics that themselves consume significant resources on a saturated system.
  4. Change one thing. Then observe. Two simultaneous changes make the outcome uninterpretable and the rollback ambiguous.
  5. Record as you go. Timestamped notes of every command run. Memory is unreliable afterwards, and the timeline is what makes the review useful.

First commands, by symptom

SymptomLook at first
Everything slowActive session count and wait events; connection saturation
Writes failingDisk space, replication slots, read-only mode
One query slowIts plan, then whether the plan changed
Replica behindWhich lag component is growing — send, receive or apply
Rising errorsThe specific error class; deadlocks and connection refusals mean different things

Symptom-specific procedures are in the playbooks.

Load shedding is a legitimate fix

If the database is saturated, reducing what reaches it is faster than making it faster:

  • Pause background jobs, batch imports and reindexing.
  • Disable non-essential features behind flags.
  • Lower connection pool sizes so requests fail fast rather than queueing.
  • Serve stale cache entries instead of recomputing them.

A database that serves 70% of traffic well is a better outcome than one that serves 100% badly.

Communication

State impact and what you are doing, not root-cause speculation. "Order creation is failing for about 30% of requests; we have failed over to the standby and are monitoring" is useful. "Investigating a possible index issue" is not, and is often wrong.

Update on a fixed cadence even when there is nothing new — silence gets interpreted as absence.

After the incident

Write the review within a few days, while details are recoverable. The useful content is a timeline, the contributing factors, and specific actions with owners.

Ask what made the incident long rather than only what made it start. Missing monitoring, absent runbooks, ambiguity about who could authorise a failover and a restore path nobody had tested typically account for far more downtime than the triggering fault.