Jan Mikoláš
Glossary
Message queue
A queue separates an immediate change from work that happens later. However, it does not automatically guarantee exactly-once delivery, global ordering, or business consistency.
Short definition
Work can wait without blocking its sender.
A producer sends a message with a clear meaning and identity to a queue or broker. A consumer picks it up later, for example to create a shipment, send an email, or import a product. Asynchronous processing allows the user to receive an order confirmation without waiting for every downstream integration.
A message queue is a general concept. It can be provided by a broker such as RabbitMQ, a cloud service, or another system. The queue itself does not remove the need for database transactions, duplicate protection, or operational monitoring.
Use cases
When to move work into the background
The greatest benefit comes from work that can finish later or runs at a different pace from the main application.
- sending an email, document, or notification after an order is placed
- importing a product catalogue in batches
- synchronising inventory and prices with a marketplace or ERP system
- processing a webhook event without blocking the HTTP endpoint
- absorbing a temporary spike when the producer works faster than the consumer
Practical example
Catalogue import without blocking administration
An administrator starts a product catalogue import. The application splits the file into batches and places ImportProductBatch messages with an import ID in the queue. Consumers can run in parallel while the administration interface shows the import status instead of waiting for one long request.
Each batch stores its result by import ID and batch order. A temporary problem with an external media service is retried after a delay; a faulty product does not end up in an endless loop but remains traceable for correction. Parallel processing does not require global ordering across different products.
How it works
Handing off a job without waiting for completion
A message needs a clear meaning, a stable identity, and an agreed way to handle failures.
- Producer Creates a command or event with an ID and the necessary data, then passes it to the queue.
- Storage and delivery The queue or broker holds the message until a consumer picks it up.
- Consumer Performs an independent task, such as calling a carrier API or creating a document.
- Acknowledgement Confirms processing after safe completion; if the process crashes before confirmation, the message may be delivered again.
- Failure flow A transient error gets a limited retry, while a permanent problem ends up in a DLQ or a state designed for targeted resolution.
Important concepts
An asynchronous flow has rules of its own.
A queue is more than a list of jobs: it affects ordering, throughput, outages, and recovery.
Command and event
A command requests a specific action, such as CreateInvoice. An event announces what happened, such as order.paid, and may have several independent consumers.
Acknowledgement and redelivery
The acknowledgement follows a recorded result. If the process crashes first, repeated delivery is an expected scenario, not an exception.
Backpressure
Queue length shows that consumers are falling behind. Limits, autoscaling, or slowing the producer prevent the backlog from merely shifting the problem into memory usage and latency.
DLQ and retry
A dead-letter queue separates messages that should no longer circulate automatically. Retries have a limited number of attempts, a delay, and a traceable reason.
Benefits and limitations
A more resilient flow, but more states to operate.
Benefits
- separating a fast user action from a slow integration
- gradually clearing a spike according to consumer capacity
- isolating a temporary outage of a dependent service
- independently measuring and scaling downstream processing
Common mistakes
- relying on exactly-once delivery instead of an idempotent consumer
- allowing retries without a limit or an observable final state
- assuming global ordering with parallel consumers
- recording a business change but losing the downstream event when publishing fails after the commit
Scope of use
Asynchronous processing should solve a specific problem.
A queue helps when work can finish later, needs retries, or should have independent capacity. If the user cannot continue without an immediate result, the system needs a synchronous operation or a clear “processing” state.
Passing a message does not ensure an atomic change across a database and a broker. For an important event after saving an order, it can make sense to use an outbox pattern, for example; the details always depend on the cost of lost or duplicated work.
What to consider
Measure both the queue and consumer outcomes.
Background processing should not allow errors to disappear from view.
- an unambiguous message type, event ID, and business identity
- an idempotent consumer and acknowledgement only after a safe result
- metrics for queue length, message age, and failure count
- retries with a limit, jitter, and a controlled DLQ
- a procedure for manual correction and targeted reprocessing
Common questions
What a message queue does not guarantee
Is a message queue the same as an API?
No. An API is an interface between systems. A queue is a way to hand off work asynchronously; an API can be synchronous or asynchronous.
Does a queue guarantee exactly one processing attempt?
Usually not. If a process crashes between a side effect and acknowledgement, the message may be delivered again, so the consumer should be idempotent.
Will message order be preserved?
Only in limited scenarios. Multiple producers, parallel consumers, and redelivery can change the actual processing order.
Should every error be retried?
No. A transient technical error may receive a controlled retry. Invalid data or a rejected request often belongs in a DLQ for correction.
How I approach integration flows in practice
I design resilient synchronisation from intake through error diagnosis.
For imports, orders, and API integrations, I address outages, retries, duplicates, and the gradual processing of larger data volumes.