What Is a Message Queue and How to Use One

A checkout endpoint that resizes a product image, sends a confirmation email, and updates a recommendation engine before returning 200 OK is only as fast as its slowest step. If any one of them times out, the whole request fails. A message queue lets the endpoint hand that work off as messages and return immediately, while separate workers pick them up on their own schedule.

What a message queue actually does

A message queue sits between two parts of a system that don’t need to run at the same moment. A producer publishes a message describing work to be done. The queue holds it. A consumer picks it up, processes it, and acknowledges it. The two never talk to each other directly.

Think of a mailbox. The sender drops a letter in and walks away; the reader checks it whenever it’s free, and if it’s away for an hour, the letters wait.

Three things follow that a direct function call can’t give you:

• Decoupling. The checkout endpoint doesn’t need to know how image resizing works, or that it’s slow. It publishes {"event": "order_placed", "order_id": 4821} and moves on.
• Buffering. If 500 orders land in the same second, the queue absorbs the spike. Workers process them at whatever rate they can sustain, instead of 500 requests blocking at once.
• Independent scaling and failure. Run three image-resizing workers and one email worker, restart either without touching the other, deploy the checkout service without redeploying its workers.

Running RabbitMQ in Docker

RabbitMQ is the most common general-purpose message broker and a reasonable default if you don’t already have a queue. Pull the management image so you get a web UI alongside the broker:

1
2
3
4
5

docker run -d --hostname mq --name rabbitmq \
  -e RABBITMQ_DEFAULT_USER=app \
  -e RABBITMQ_DEFAULT_PASS=changeme \
  -p 5672:5672 -p 15672:15672 \
  rabbitmq:4-management

Port 5672 is the AMQP protocol your application connects to. Port 15672 is the management UI: open http://localhost:15672, log in with app / changeme, and you can watch queues, message rates, and connections live. The default guest/guest login only works from inside the container, so set your own user for anything you’ll connect to from the host.

If you haven’t used Docker before, What Is Docker and What Is It Used For covers images, containers, and ports. For a real stack, run the broker alongside your application with Compose — see What Is Docker Compose and How to Use It:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

services:
  rabbitmq:
    image: rabbitmq:4-management
    environment:
      - RABBITMQ_DEFAULT_USER=app
      - RABBITMQ_DEFAULT_PASS=changeme
    ports:
      - "5672:5672"
      - "15672:15672"

  worker:
    build: ./worker
    depends_on:
      - rabbitmq
    environment:
      - RABBITMQ_URL=amqp://app:changeme@rabbitmq:5672/

worker reaches the broker at the hostname rabbitmq. Compose puts both services on the same user-defined network, so the service name doubles as a resolvable hostname, the mechanism covered in What Is a Docker Network and How to Use It.

Publishing and consuming a message

pika is the standard Python client for RabbitMQ’s protocol (AMQP 0-9-1). Install it with pip install pika. A producer that publishes one task looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(
        host="localhost",
        credentials=pika.PlainCredentials("app", "changeme"),
    )
)
channel = connection.channel()

channel.queue_declare(queue="resize_image", durable=True)

channel.basic_publish(
    exchange="",
    routing_key="resize_image",
    body='{"order_id": 4821, "image_url": "https://example.com/p/4821.jpg"}',
    properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent),
)
connection.close()

durable=True on queue_declare tells RabbitMQ to keep the queue itself across a broker restart. delivery_mode=pika.DeliveryMode.Persistent tells it to write each message to disk instead of holding it in memory. Without both, a docker restart on the broker silently drops whatever hadn’t been consumed yet.

The consumer pulls from the same queue and acknowledges each message once the work is actually done:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(
        host="localhost",
        credentials=pika.PlainCredentials("app", "changeme"),
    )
)
channel = connection.channel()
channel.queue_declare(queue="resize_image", durable=True)
channel.basic_qos(prefetch_count=1)

def handle_message(ch, method, properties, body):
    resize_image(body)  # your actual work
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(queue="resize_image", on_message_callback=handle_message)
channel.start_consuming()

Two settings do the reliability work here:

• basic_qos(prefetch_count=1) stops RabbitMQ from handing a worker a second message before it has acknowledged the first. Without it, one busy worker can be sitting on ten messages while an idle one gets nothing.
• basic_ack after the work, not before. If the process crashes mid-resize, the message was never acknowledged, so RabbitMQ redelivers it to another worker instead of losing it. Leave auto_ack off, which is pika’s default.

Run three copies of the consumer script and RabbitMQ splits the queue across them round-robin. That’s the whole scaling story for this pattern: no coordination code, no shared state between workers.

Some messages fail every single time they’re delivered: a malformed payload, a call to a service that’s gone for good. Redelivery turns that into an infinite loop through your consumers. Set x-dead-letter-exchange on the queue, and messages you reject without requeueing (basic_nack with requeue=False) go to a separate exchange where you inspect them by hand.

How RabbitMQ exchanges route messages

The examples above publish with exchange="", the default exchange, which routes a message straight to the queue named in routing_key. That covers most task-queue use cases. RabbitMQ’s actual model has an exchange in front of every queue, and swapping the exchange type changes the routing behavior:

A fanout exchange is how you get publish/subscribe out of RabbitMQ: publish order_placed once, and the email service, the analytics service, and the fraud-check service each get their own copy through their own queue, instead of competing for the same messages.

When a message queue is the wrong tool

• The caller needs the answer to return the response. A queue is for “do this eventually,” not “compute this now.” If your checkout endpoint needs the calculated shipping cost before it can respond, that’s a synchronous call, not a queued job.
• You need strict ordering across the whole queue. RabbitMQ guarantees order per queue with a single consumer, but add a second consumer for throughput and two messages can finish out of order. If sequence matters, as in an event log or a state machine, you need a tool built for ordered streams, like Kafka partitions keyed by entity ID.
• The job is small, synchronous, and you don’t already run a broker. A background thread or an in-process job runner is less operational surface than standing up and monitoring RabbitMQ for one cron-sized task.
• You already have Redis and can tolerate the occasional lost job. Redis Streams or a library like BullMQ give you a queue with no new infrastructure, at the cost of weaker delivery guarantees under a broker crash. Check what Redis already does in your stack first: How Redis Caching Works and How to Use It.

Moving your first task to a queue

Pick one thing that blocks a request without needing to: the confirmation email, the thumbnail, the analytics ping. Run RabbitMQ in Docker, declare one durable queue for it, move it across. Set prefetch_count=1 and acknowledge manually from day one, because adding reliability after a crash has already eaten a message costs far more than writing those two lines now. Reach for a fanout or topic exchange only once you have a second consumer that wants the same event.

Related articles

• How Redis Caching Works and How to Use It
• What Is Docker Compose and How to Use It
• What Is a Docker Network and How to Use It