Jan Mikoláš
Reliable background work
How to implement retry and exponential backoff
Retries help only with temporary errors. Set them up poorly and a small issue becomes an outage.
First, the short version
What are retry and backoff?
A retry is another attempt at the same job after a temporary failure. Exponential backoff increases the wait between attempts, giving the other side room to recover.
A common case is sending data through an external API. When the network briefly fails, the job goes back to the message queue and a worker tries later.
Get ready
What you need
Before retrying a job, decide when it is safe.
- An asynchronous job or another clearly scoped unit of work.
- A message queue and worker. In Symfony, Symfony Messenger handles them.
- A list of temporary failures: network, timeout, 429, or 5xx.
- An idempotent handler. The same message must not create a second payment, email, or order. Idempotence helps.
Step 1
Set up retries in small steps
Start conservatively. Three attempts are usually a better beginning than endless delivery.
1. Separate errors you may retry
- A network error, timeout, 429, and most 5xx responses often mean a temporary issue.
- 400, 401, 403, and missing data are usually not temporary. Send the message to failed transport and fix the cause.
- For a write to an external API, use an idempotency key. The response may have failed after the write succeeded.
if (in_array($status, [429, 500, 502, 503, 504], true)) { /* retry */ }2. Configure exponential backoff
- Start at one second, for example. Double the next waits: 1 s, 2 s, 4 s.
- Set a maximum delay. After a long outage, you do not want a job returning days later without review.
- Add a small random jitter so several workers do not fire in the same second.
delay = min(1000 * 2 ** attempt, 60000) + random(0, 250)3. Keep the failed job
- After the final attempt, send it to failed transport. It needs a readable cause and source record ID.
- Alert on failed-job count. Do not wait for a customer to notice.
- After fixing the cause, return the job to processing manually. Do not blindly retry the whole queue.
php bin/console messenger:failed:retry --forceStep 2
Check the retry plan
Before production, simulate a failure that disappears after a while.
-
Fail the first two attempts
Your log must show increasing intervals and then success, not an immediate loop.
-
Send a non-retryable failure
For a 400 or invalid data, the job must not return forever. Expect failed transport and a readable record.
php bin/console messenger:failed:show -
Send the same message twice
After success, there must be no second side effect. This verifies handler idempotence.
If something goes wrong
Common problems
A worker keeps attacking the API
Backoff or an attempt limit is missing. Add growing delays and a finite number of attempts.
An important job disappears after an error
Configure failed transport and an alert. The message must be discoverable, not only logged.
A retry creates duplicate data
The handler is not safe to repeat. Store an external ID or idempotency key and check idempotence.
Every error is retried
Separate temporary network failures from data and permission errors. Fix the latter instead of returning them to the queue.
Done
Retries have clear limits.
Retries are limited, the wait grows, and failed work remains discoverable.