Jan Mikoláš
Practical guide
How a dead-letter queue works and when to use it
Separate a permanently failed message from normal traffic, preserve its failure reason, and return it only after fixing the cause.
In short
A DLQ is not infinite retry
RabbitMQ does not move a message directly into a special queue. The source queue dead-letters it to a normal dead-letter exchange and a binding routes it to an inspection queue.
A message can be dead-lettered after reject or nack without requeue, TTL expiry, queue length overflow, or a quorum queue delivery limit. A DLQ preserves a problem; it does not fix it.
Prepare
What you need
Distinguish transient and permanent failures first. Otherwise a DLQ merely hides an unbounded retry loop.
- A source queue, stable routing key, and documented message format with message ID and schema version.
- Failure classes: transient dependency, invalid data, unsupported version, and unknown error.
- An attempt limit and increasing delay outside the main queue.
- An alert owner, retention policy, secure inspection tool, and controlled replay procedure.
Steps 1 to 3
Design the message failure path
The main queue must keep moving. One poison message must not occupy the same consumer forever.
1. Create a dead-letter exchange and target queue
- Declare a dedicated exchange such as orders.dlx and a durable orders.failed queue bound for the intended routing keys.
- Set dead-letter-exchange and optionally dead-letter-routing-key on the source queue. Prefer a RabbitMQ policy that can change without redeclaring it.
- Verify the target exchange and binding exist. An unroutable dead-lettered message must not disappear silently.
- Set a length or retention limit for the DLQ and monitor disk. It is not an endless archive or a backup replacement.
rabbitmqctl set_policy orders-dlx "^orders$" '{"dead-letter-exchange":"orders.dlx"}' --apply-to queues2. Separate retry from permanent failure
- Send a transient error through a bounded retry path with a delay and return. nack with requeue=true and no pause creates a hot loop.
- Reject invalid data or an unsupported version without requeue directly to the DLQ. Another attempt with identical input cannot help.
- Set delivery-limit explicitly by policy on quorum queues and configure a DLX. Do not depend on endless poison-message redelivery.
- Keep original message ID, correlation ID, and a safe failure reason. Do not copy secrets or complete personal data into headers and logs.
transient → delayed retry → source; permanent → nack(requeue=false) → DLX3. Prepare inspection and controlled replay
- Alert on the first message and the DLQ growth rate. Waiting for a full disk is too late.
- Show the dead-letter reason, age, type, and a safe payload excerpt. Audit and restrict access according to data sensitivity.
- Fix code or data and select specific messages before replay. Never pour the whole DLQ automatically into the main queue.
- Publish replay as a new delivery operation but preserve the original business identity. The consumer must be idempotent.
app:messages:replay --queue=orders.failed --message-id=...Step 4
Test every failure path
Manually placing one message in a failed queue is not enough. Verify routing, metadata, alerts, and recovery after a fix.
-
Reject a permanent failure
The consumer must nack without requeue, the message appears once in the target DLQ, and the main queue continues.
-
Exhaust the retry limit
Let a transient failure pass through planned delays. It must enter the DLQ after the final attempt, not loop forever.
-
Fix and replay one message
Verify one business result, acknowledgement, and removal or marking of the original DLQ item.
When it goes wrong
Common mistakes
A message disappeared after nack
Check requeue=false, DLX configuration, permissions, binding, and routing key. A failed queue name alone is not enough.
rabbitmqctl list_queues name argumentsThe consumer repeats the same failure without delay
Do not immediately requeue a failing dependency. Use bounded retry queues with delay or another scheduler.
The DLQ grows unnoticed
Alert on count, oldest message age, and growth rate. Every message type needs an operational owner.
Bulk replay overloaded the service again
Replay in small rate-limited batches with a stop condition. Verify one message and the fixed cause first.
Done
Failed messages have a controlled path.
The message queue is no longer blocked by a poison message. Retry is bounded, the DLQ is monitored, and replay follows a verified fix and deliberate selection.