Change Data Capture

Change data capture (CDC) turns a database's replication log into a stream of row-level change events that other systems consume. It is how you feed a search index, an analytical store or a message bus without dual writes.

Why the log, not the table

The alternatives are worse:

• Polling updated_at misses deletes, misses changes within the polling window, and requires an index on a constantly changing column. Clock skew and long transactions make it lose rows.
• Application-level publishing cannot be made atomic with the database write — see Distributed Transactions.
• Triggers write to an outbox table transactionally, which is correct, but they add write latency to every statement and the trigger logic becomes schema you must migrate.

The replication log already contains every committed change in commit order, and reading it costs the primary almost nothing beyond retaining the log.

Per engine

Debezium is the common connector implementation across the first three.

The replication slot hazard

This is the operational risk that surprises teams, and it is worth understanding before you enable CDC.

Mitigations, all of which you should have in place before the first consumer connects:

• Alert on slot lag. In PostgreSQL:

SELECT slot_name,
       active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

• Set max_slot_wal_keep_size (PostgreSQL 13+) so a stalled slot is invalidated rather than filling the disk. The consumer then has to resnapshot — a deliberate, bounded failure instead of an outage.
• Drop slots for consumers you decommission. This is the most common cause of the incident.

Semantics consumers must handle

• At-least-once delivery. Events can be redelivered after a connector restart. Consumers need idempotent application, keyed by primary key plus log position.
• Ordering is per key, not global. Preserve per-key order by partitioning the downstream topic on the primary key.
• Schema changes appear in the stream. A column added upstream reaches the consumer as a new field; a dropped column breaks consumers that require it. Roll out consumer changes before the producing migration when possible.
• Initial snapshot. A new consumer needs the current state before the tail of the log. The snapshot phase is read-heavy on the source — schedule it away from peak, or take it from a replica.

Reasonable uses

• Keeping a search index or cache in step with the system of record.
• Feeding an analytical store such as ClickHouse from an operational database.
• Zero-downtime migrations between engines, where CDC keeps the target current while traffic is cut over gradually.
• Publishing domain events from an outbox table, so only rows you intend to expose are streamed.