Jan Mikoláš
Glossary
Retry
Repeating an operation after an error helps only when it does not amplify the outage or create a second order, payment, or shipment.
Short definition
Do not repeat the same request blindly.
A retry gives another chance for an operation interrupted by a timeout, brief network outage, or temporary service unavailability, for example. In an integration worker, it can reduce manual intervention, but it needs a maximum number of attempts, a time limit, and a traceable final state.
A timeout is not proof that the remote service did nothing. For a write operation, the result is unknown. Without an idempotency key, external identity, or database constraint, another attempt may create a duplicate order or shipment. A retry controls the timing of an attempt; idempotence protects its business outcome.
Use cases
When a second attempt can help
The specific contract and meaning of the operation matter, not only the HTTP response code.
- a network timeout, connection reset, or brief DNS unavailability
- HTTP 502, 503, or 504 when the service documentation and operation allow it
- HTTP 429 with a delay according to Retry-After
- a temporary database deadlock when the transaction can safely restart
- a queued message that a worker could not finish because a dependent service was unavailable
Practical example
A marketplace API temporarily returns 503
An importer sends an inventory change with the identity of the product and import operation. The marketplace returns 503. The worker makes a second attempt only after a short backoff, with subsequent attempts delayed further and spread with jitter. If the service sends Retry-After, that is the earliest possible time for the next request.
For HTTP 422, the job is not retried—the data needs correction. After a timeout, it is repeated with the same identity because the remote side may have accepted the write. Once the defined budget is exhausted, the work is marked as failed for an operator, not left as a silent endless retry.
How it works
From error classification to a controlled outcome
A sound policy does not make another attempt until it understands the boundaries and cost of the error.
- Classification Distinguishes a transient technical error, capacity restriction, invalid input, and an unknown result after a timeout.
- Retry safety Verifies an idempotency key, external ID, or another safeguard for a write operation.
- Service instructions A valid Retry-After determines the earliest next attempt; the client must not bypass it with an immediate request.
- Backoff and jitter Increasing, capped intervals with random variation prevent a retry storm and synchronised traffic spike.
- Final state After all attempts are exhausted, the job is marked as failed and the reason is recorded. Depending on its meaning, it then goes to a DLQ or targeted remediation.
Important concepts
Errors, delays, and shared capacity
A technical error classification is guidance, not a universal automatic rule.
Transient and permanent errors
HTTP 400 and 422 usually require correcting the input; 401 may require controlled token renewal; 403 is usually a permissions issue. A 5xx error may be temporary, but that depends on the service.
Exponential backoff and jitter
The interval gradually grows to an upper limit. Jitter spreads attempts from many workers so they do not overload a newly recovered service all at once.
Retry-After
HTTP 429 and 503 may specify a number of seconds or a date before which the request should not be repeated. Local backoff may extend that deadline, not shorten it.
Retry budget
A shared limit on repeated attempts protects a dependent service during a widespread failure. It is not the same as a limit on incoming API requests.
Benefits and limitations
Fewer false failures, but resilience is not free.
Benefits
- more resilient handling of brief outages without manual intervention
- clearly defined behaviour for HTTP clients and workers
- lower risk of immediately overloading a recovered service
- traceability of the number of attempts and the final cause
Common mistakes
- repeating every POST without deduplication
- multiplying retries in the HTTP client, worker, and broker at the same time
- immediately repeating a 429 or 503 request
- leaving a job in an endless requeue loop
Scope of use
Manage attempts at one deliberately chosen layer.
A transient error should be retried at the layer that understands the context, deadline, and idempotence of the operation. When the client, frontend, job, and broker all use the same policy without coordination, a few expected attempts can easily become dozens of actual requests.
A retry will not fix an invalid data format, incorrect configuration, missing permission, or an outage lasting longer than makes business sense. After the final failure, the system must know what happened rather than silently discarding the work.
What to consider
Test the policy even when the outcome is uncertain.
The most important scenario occurs when the operation may have succeeded but the response did not arrive.
- error categories based on the specific service contract
- an idempotency key or stable business identity for write operations
- capped backoff, jitter, number of attempts, and an overall deadline
- respect for Retry-After and a shared retry budget
- logging a correlation ID and error reason without tokens or sensitive payloads
Common questions
When repetition does not make sense
Should every 5xx response be retried?
No. A 5xx response often indicates a temporary problem, but retry safety depends on the operation, error code, and provider documentation.
Is a POST automatically unsafe to retry?
It may be without a stable identity or idempotency key. A particular POST can be designed so a second attempt creates the same result.
Is Retry-After the same as backoff?
No. Retry-After is an instruction from the provider; backoff is a local client policy. The next attempt must not occur before a valid Retry-After allows it.
Is a retry the same as redelivery from a queue?
Not necessarily. Redelivery can follow a consumer crash; in both cases, however, processing needs idempotence.
How I approach integration resilience in practice
I treat an uncertain outcome as part of the design, not an exception.
For API integrations, I address limits, timeouts, retries, duplicates, and the ability to safely trace or recover a failed flow.