Message Queuing

Modern systems don't always operate in perfect sync. Services start and stop, networks lag, and processing speeds vary. To keep communication reliable across these moving parts, teams often rely on tools built for asynchronous workflows, such as message queues. 

What Is Message Queuing?

Message queuing is an asynchronous communication mechanism that allows messages to be sent and received without needing a direct connection or instant processing.

It's a technique in distributed systems, especially serverless environments and microservices architectures. It enables the different parts of a system to send and receive messages even if they aren't always available or running simultaneously.

The message queue is an intermediate buffer that temporarily stores the messages until the intended recipient is available. This process decouples the sender and receiver and enables efficient, scalable, and fault-tolerant communication.

For example, when a user sends a message in a chat app, that message might be sent to a queue before it's processed and delivered to other participants. This ensures the app stays responsive, even during spikes in activity or temporary service interruptions.

How Does Message Queuing Work?

To understand message queuing, we must look at the software components that comprise it: producers, consumers, message queues, and, in some implementations, message brokers.

Producers create and send messages to the queue while consumers retrieve and process those messages. Producers and consumers can be servers, microservices, user devices, Application Programming Interfaces (APIs),  Internet of Things (IoT) devices, and more.

Since message queuing decouples them, consumers frequently operate at their own pace and are completely independent of the producers. A producer can be offline or handling another task when a consumer receives its message, and vice versa.

The central component of message queuing is the message queue itself. The queue is what holds the producer's message until the consumer is ready for it.

There are many ways to implement message queuing. In some cases, it's as simple as using a built-in queue data structure in memory. In others, you might use a brokerless messaging library like ZeroMQ, which handles transport directly between producers and consumers.

Commonly, message brokers act as queue managers. They may also take care of other tasks like routing, filtering, and transforming messages, as well as guaranteeing delivery. Brokers are typically tied to a specific communication protocol.

Some popular brokers include RabbitMQ, Apache Kafka, and Microsoft Azure Service Bus.

Queues most often deploy the First In, First Out (FIFO) method, where the first message the queue receives is the first one it sends out. However, some implementations may involve factors such as priority settings, message grouping, deduplication, and error handling, which can lead to deviations from FIFO.

Message Structure

The messages sent in a message queuing architecture will vary greatly depending on factors like what protocol or message broker (if any) you use. Typically, messages will have a structure that includes at least some of the following:

• Headers: Metadata such as routing information, timestamps, message ID, priority level, and expiration time are found in the headers.

• Body: This is where the payload data is stored and, depending on the implementation, may be in JSON, XML, binary, or another format.

• Properties: Other metadata such as content type, encoding, delivery mode, and reply-to address may be found in properties.

Queue Structure

A message queue typically uses several data structures to store and manage the messages it receives. These include:

• Message buffer: In setups that use a broker, this is a memory-based data structure that temporarily stores messages in a FIFO order while the broker processes them.

• Message store: A persistent data structure for on-disk storage of messages not yet processed or acknowledged.

• Index: A data structure holding metadata about the messages in the queue that helps to speed up the retrieval process.

Messaging Patterns

There are several variations of messaging patterns, some of which include:

• Point-to-point (P2P): A one-to-one model where messages are sent from a single producer to a single consumer. This makes sure that messages are only processed once.

• Publish-subscribe (Pub/Sub): A one-to-many structure where messages are broadcast to multiple consumers, which is useful when multiple services must process a message.

• Priority queue: Messages are assigned a priority level and processed on that basis, sending time-sensitive tasks to consumers first.

• Work queue: This is a model where message consumers compete with each other, enabling load balancing.

• Dead letter queue (DLQ): DLQ is effectively a trash can for messages that couldn't be processed due to errors, which helps in debugging and prevents data loss.

Popular Protocols With Message Queuing

Much of the specifics of message queuing (like messaging patterns and maximum message size) will depend on what protocols and brokers the development team chooses for their specific use case.

There are several messaging protocols to choose from, including:

• Advanced Message Queuing Protocol (AMQP): Commonly seen in enterprise and financial environments, AMQP is an open standard, wire-level protocol that standardizes message exchange between clients and brokers like RabbitMQ, Apache Qpid, and Red Hat AMQ Broker. It provides a flexible, reliable way to connect message-oriented middleware systems and has a high degree of interoperability across platforms, languages, and even with other protocols like WebSockets and HTTP.

• Message Queuing Telemetry Transport (MQTT): A lightweight protocol that uses a pub/sub architecture and is especially suited for IoT use cases. It has three quality of service levels or delivery models that guarantee messages reach their destination exactly as they need to, even if the devices are weak or the network is unreliable. Popular brokers include HiveMQ, Mosquitto, and RabbitMQ. 

• Simple Text-Oriented Messaging Protocol (STOMP): Also called the Streaming Text-Oriented Messaging Protocol, STOMP is an easy-to-implement text-based protocol for message queuing similar to HTTP. It offers a simple interface for exchange between producers and consumers. RabbitMQ, Apache Active MQ, and HornetQ are brokers developers commonly use for STOMP.

Message Queue Operations

There are typically four basic operations in a message queue:

1. Send adds a message to the queue.

2. Receive retrieves a message from the message queue.

3. Acknowledge (ACK) confirms that a consumer received and processed a message.

4. Negative Acknowledge (NACK) reports that a consumer can't process a message, injecting a measure of fault tolerance into the message queue. It may result in an immediate or delayed retry or send the message to the DLQ.

In a broker architecture, the message broker acts as an intermediary between the producer and consumer for these operations. In a brokerless architecture, there is a direct exchange between the two.

Message Flow

Message queuing is generally a four-step process:

1. Message production: When a producer creates a message, it includes, at the very least, a header and body. The producer then either drops the message in the queue or passes it to the message broker, if there is one.

2. Message queuing and persistence: The message queue or message broker receives and stores it. If durable queues have been enabled in the implementation, the messages will persist on disk to survive broker restarts.

3. Message consumption: One or more consumers retrieve their messages from the queue using one of two models: pull or push. In the pull model, consumers fetch messages from the queue. In the push model, the broker pushes messages to consumers.

4. Message processing and acknowledgment: The consumer processes the message and returns an ACK to the broker. If the consumer fails before acknowledgment, the message can be requeued or sent to a DLQ for troubleshooting.

Delivery Models

The specific terminology and functionality will vary, but many protocols with message queuing have delivery models to accommodate messages not successfully being sent or received on the first try.

Message queuing typically employs one of three delivery models: 

• At-most-once: Messages are delivered only once. The weakness of this model is that messages can be lost if failures occur. It's usually applied when occasional message loss is acceptable and is the least resource-intensive.

• At-least-once: Messages are delivered once and re-delivered if an acknowledgment isn't received. The weaknesses are that it may lead to duplicate processing, and it uses more resources than at-most-once.

• Exactly-once: Messages are processed once and only once, which is achieved through deduplication and transactional messaging. The weakness of this model is that it tends to be more complex and is the most resource-intensive.

In an MQTT implementation, a temperature sensor publisher (producer in a pub/sub architecture) that frequently reports its readings may be set to at-most-once since the subscriber (consumer) dashboard doesn't need every single reading.

In AMQP, business-critical messages and financial transactions may be sent between producer and consumer with exactly-once to guarantee delivery and prevent processing the transaction multiple times.

Frequently Asked Questions