Redis and Valkeyintermediate

Redis Streams

An append-only log with consumer groups and acknowledgements — the right Redis primitive for durable work queues.

2 min readIntermediateUpdated 31 Jul 2026Edit this page

A Stream is an append-only log of entries, each with an auto-generated, time-ordered id. Unlike Pub/Sub, entries persist until you remove them, and consumer groups track what each consumer has acknowledged.

XADD shop:orders '*' order_id 4711 status paid
> "1753968000000-0"
 
XLEN shop:orders
XRANGE shop:orders - + COUNT 10

Consumer groups

# Create the group, starting from the beginning of the stream.
XGROUP CREATE shop:orders fulfilment 0 MKSTREAM
 
# Each consumer reads entries not yet delivered to the group.
XREADGROUP GROUP fulfilment worker-1 COUNT 10 BLOCK 5000 STREAMS shop:orders '>'
 
# Acknowledge after the work succeeds.
XACK shop:orders fulfilment 1753968000000-0

An entry read but not acknowledged stays in the consumer's pending entries list. That is what makes the queue durable: a worker that crashes leaves its entries visible and reclaimable rather than silently lost.

XPENDING shop:orders fulfilment                 # summary
XPENDING shop:orders fulfilment - + 10 worker-1 # detail per entry
 
# Reassign entries idle for more than 60 seconds to another consumer.
XAUTOCLAIM shop:orders fulfilment worker-2 60000 0 COUNT 10

A recovery loop that periodically runs XAUTOCLAIM is required — without it, a dead worker's entries stay pending forever.

Trimming

A stream without trimming grows until memory runs out

Acknowledging an entry does not remove it from the stream; it only clears it from the pending list. The stream itself grows indefinitely unless trimmed.

# Approximate trimming is much cheaper and is what you want in production.
XADD shop:orders MAXLEN '~' 1000000 '*' order_id 4711
 
# Or by time, keeping entries newer than a given id.
XTRIM shop:orders MINID '~' 1753900000000

The ~ makes trimming approximate — Redis removes whole macro-nodes rather than counting exactly, which is far faster. Use exact trimming only when the bound must be precise.

Delivery semantics

At-least-once. An entry can be delivered again if a worker crashes after processing but before acknowledging, so consumers must be idempotent. Use the entry id or a business key to deduplicate.

Streams versus a real broker

Streams are a good fit when Redis is already in the architecture and the volume is moderate. Be clear about the limits before choosing them over Kafka or a dedicated queue:

The stream lives in memory; retention is bounded by RAM, not by disk.
Replication is asynchronous, so a failover can lose recently added entries.
There is no partitioning across nodes for one stream — a single stream lives in one slot, so throughput is bounded by one shard.
There are no built-in dead-letter semantics; XPENDING delivery counts let you build them.