Jan Mikoláš
Resilient integration
How to handle an external API outage
An external service will fail one day. The question is whether it also stops your application.
First, the short version
What happens in an outage?
An external API call can fail, wait for too long, or return incomplete data. An outage is not an edge case. Design for it as a normal state.
Set a short timeout, retry only temporary errors sensibly, and stop burdening the service during a longer outage. Users should get a clear result.
Get ready
What you need
Before adding retries, learn which calls you may safely repeat.
- A list of every external API call and a clear owner for each one.
- An agreement with the provider: limits, error codes, timeouts, and status page.
- A message queue for work that does not need to finish immediately. The user then does not wait for a third-party API.
- Logs and metrics for error count, call duration, and failed jobs.
Step 1
Add a safety net
Protection has three layers: spot the problem quickly, attempt a sensible recovery, and keep working in a limited mode.
1. Set short timeouts
- Separate connection timeout from full response timeout. Choose values for the feature, not by instinct.
- Do not let one slow request block a PHP worker or a user response.
- Log a URL without secrets, duration, status, and error type.
$client->request('GET', $url, ['timeout' => 5])2. Retry only temporary failures
- For a network error, 429, or 5xx use limited retries. Usually fix 400 to 404 errors in data, not by retrying.
- Increase the wait between attempts. Add a small random jitter so workers do not hit at the same moment.
- Retry a write only if the provider offers an idempotency key or its own safe mechanism.
retry: 3; delay: 1000; multiplier: 23. Switch to limited mode
- After several failures, open a circuit breaker: do not make further calls for a short time and return a known safe response.
- A fallback can be the last known value, an option to finish later, or a clear message to the user.
- Put work that must finish onto a queue. Process it in a controlled way once the service recovers.
external_api_state = "open"Step 2
Rehearse the outage
A test, not a configuration file, earns trust.
-
Return a timeout
In test, point the client at a slow or nonexistent address. Your own application must respond within the planned time.
-
Return 503 and 429
Check that only a limited number of retries happens and the interval grows.
-
Check return to normal
After the service recovers, let one probe request pass. The queue should process gradually, not empty all at once.
php bin/console messenger:failed:show
If something goes wrong
Common problems
The application freezes during an outage
A timeout is missing or too long. Set a limit for connection and full response.
Retries make the outage worse
Reduce attempts and add growing delays. Also respect the provider’s rate limit.
An order is created twice
A write was retried without protection. Use a unique key and idempotence.
The open circuit breaker never closes
Add a short cooldown and one probe request. Measure breaker state and show it in monitoring.
Done
An outage is no longer a catastrophe.
Your application has timeouts, limited retries, and a safe limited mode. Now watch metrics and adjust the rules from real traffic.