Distributed Transactions

A distributed transaction makes changes on several nodes or systems atomic. It is well understood, widely implemented, and worth avoiding when the data model can be arranged to make it unnecessary.

Two-phase commit

1. Prepare. The coordinator asks every participant to prepare. Each writes the change durably, takes locks, and replies "ready" — promising it can commit if asked.
2. Commit. If all replied ready, the coordinator records the commit decision and tells everyone to commit. If any refused, it tells everyone to abort.

The protocol is correct, and its cost is structural:

• It blocks. If the coordinator fails after prepare but before broadcasting the decision, participants hold locks and cannot decide alone. They wait for the coordinator to return.
• Latency is additive. Two rounds of messages to every participant, plus durable writes at each step, on every transaction.
• Locks are held longer, so contention rises sharply under load.

Distributed SQL engines mitigate this by making the coordinator itself fault-tolerant: the transaction record is stored in a consensus-replicated range, so a coordinator crash does not leave participants stranded. That is why 2PC is practical inside CockroachDB or YugabyteDB and painful between independent systems.

Across separate systems

The outbox pattern

Write the state change and the event to the same database in one local transaction:

BEGIN;
UPDATE orders SET status = 'paid' WHERE id = 4711;
INSERT INTO outbox (aggregate_id, event_type, payload, created_at)
VALUES (4711, 'order.paid', '{"order_id":4711}'::jsonb, now());
COMMIT;

A separate process reads the outbox — by polling, or from the replication stream via change data capture — publishes each event, and marks it sent. The publish is retried until it succeeds, so delivery is at-least-once and consumers must be idempotent.

This turns an impossible atomic operation into a solvable delivery problem, which is why it has become the default answer.

Sagas

A saga is a sequence of local transactions, each with a defined compensating action. If step three fails, run the compensations for steps two and one.

Sagas are not atomic: there is a window where the system is partially updated, and compensation is business logic, not a rollback. "Refund the payment" is not the inverse of "take the payment" — it is a new, visible event. Use sagas when steps are genuinely independent and compensation is meaningful; avoid them when a partial state would be unacceptable to observe.

Idempotency

Every distributed protocol here ends up retrying, so every operation must be safe to apply twice. The standard mechanism is a client-supplied idempotency key stored with a unique constraint:

INSERT INTO payments (idempotency_key, order_id, amount_cents)
VALUES ('c1f2…', 4711, 19900)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;

If the insert returns no row, the operation already happened and the stored result should be returned instead of performing it again.