What is a message queue? How an observability platform eases message queue monitoring

What is a message queue?

A message queue is a form of middleware used in software development to enable communications between services, programs, and dissimilar components, such as operating systems and communication protocols. A message queue enables the smooth flow of information to make complex systems work.

The concept is like text messaging—a feature most mobile phone users understand. If you need to send a message, you can call someone. However, that assumes they are available and have time to talk. For non-urgent messages, texting is a more efficient approach. The message goes into a queue on the recipient’s phone, and they respond when possible.

In a distributed processing environment, message queuing is similar, although the speed and volume of messages are much greater. Queued messages are typically small and specific. They may be requests, replies, error messages, or information needed for logging or tracing. A producer creates the message, and a consumer processes it.

Consumers store messages in a queue—usually in a buffer or on a storage medium—until they can process and delete them. A given consumer only processes each message once. Teams can use message queues to store tasks during periods of heavy processing for later attention. Additionally, a message queue can smooth out spiky workloads by enabling the producers and consumers to work at a consistent pace without losing data.

Advanced Message Queuing Protocol

The Advanced Message Queuing Protocol (AMQP) is an open standard for passing messages between applications and organizations. It provides a consistent platform that integrates with a variety of message queuing types to enable consistent performance regardless of the platform.

Two styles of message queuing

Message queuing has been used for many years, most prominently in email. Without it, sending an email over a long distance would require the immediate availability of every node on the routing network to forward each message. Using queuing, IT teams store messages in intermediate nodes until they can forward them. This enables email message processing in a quick and reliable way, even during periods of heavy network congestion.

A queue manager is the control-plane component of a message queuing system. It specifies such details as which messages receive priority, the necessary types of responses, and where to store and process messages. A complex system may have many message queues, with the queue manager ensuring they all work properly.

There are two basic styles of message queuing:

• Point-to-point messaging occurs between two discrete entities that share responsibility for a given task, such as a controller on a factory floor and a discrete piece of machinery. The producer application needs to know something about the consumer, such as where to send the message and the expected actions. This approach is common in transaction-intensive environments in which teams perform the same tasks repetitively.
• Publish-and-subscribe messaging decouples the provider of the information from the consumer. Neither requires information about the other. And messages are sent to all subscribers at the same time—usually without needing a response. RSS feeds are an example of a widely used publish-and-subscribe mechanism. A user visiting a website that supports RSS can cut and paste a URL into a reader or compatible email client. Thereafter, the user is notified of all updates to the site. No action is required by the consumer or expected by the publisher.

Use cases of message queuing

In today’s digital landscape, message queuing plays a pivotal role in driving the performance and reliability of distributed systems. Let’s delve into some key use cases.

1. Asynchronous Processing: Message queues decouple processes, allowing them to run independently and perform tasks asynchronously. This is particularly useful in scenarios where tasks are resource-intensive or time-consuming, such as video encoding or sending bulk emails.
2. Load Balancing: In high-traffic applications, sudden surges can overwhelm servers, leading to performance degradation. Message queues can act as a buffer, absorbing these spikes and distributing the load evenly across the system.
3. Resilience and Fault Tolerance: Message queues ensure that no message is lost in case of a system failure. They store messages until they are successfully processed, thereby enhancing the system’s resilience and fault tolerance.
4. Ordering and Sequencing: Certain applications require tasks to be executed in a specific order. Message queues can enforce this sequencing, ensuring that messages are consumed in the order they were added.
5. Real-time Updates: In applications like live dashboards or real-time analytics, message queues can push updates to the client as soon as they occur, enabling real-time data flow.
6. Inter-Service Communication: In a microservices architecture, message queues facilitate communication between services, ensuring loose coupling and high cohesion.

The benefits of message queuing

Message queuing is a critical component in modern distributed systems, offering a host of benefits that enhance performance, scalability, and reliability. Let’s explore these benefits in detail.

1. Scalability: Message queues can easily scale to accommodate growing workloads. They allow systems to handle high volumes of data by distributing messages across multiple consumer processes or threads.
2. Decoupling: Message queues decouple the sender (producer) and receiver (consumer), allowing them to operate independently. This means that the producer doesn’t need to wait for the consumer to process a message before moving on to the next task.
3. Resilience: Message queues provide robustness against system failures. If a consumer fails while processing a message, the message can be returned to the queue or redirected to a dead-letter queue for later analysis.
4. Load Balancing: Message queues distribute the processing load evenly among consumers. If a consumer is overwhelmed with messages, new consumers can be added to share the load.
5. Asynchronous Communication: Message queues enable asynchronous communication between microservices in a distributed system. This means that a sender can continue with other tasks after dispatching a message, improving system throughput.
6. Order Preservation: Some message queues guarantee that messages are delivered in the order they were sent. This is crucial in scenarios where the order of operations matters.
7. Delivery Guarantees: Many message queues provide at-least-once or exactly-once delivery guarantees, ensuring that no message is lost in transit.

How message queuing benefits microservices

Microservices are an increasingly popular way to build software because of their speed and flexibility compared with traditional monolithic approaches. Applications built with microservices are broken into discrete modules so development teams can work on them in parallel.

Services communicate using application programming interfaces, which means there is no need to write them in specific programming languages or frameworks. Developers can easily modify applications by adding or swapping out microservices, and testing requirements are reduced because microservices are isolated and often pretested. A failure in one service doesn’t trigger unexpected problems elsewhere in the application.

Message queues coordinate microservices in a distributed application. Because queues are persistent, they reduce the risk of errors if parts of the system go offline. A queue manager’s rules always determine how to process messages. As a result of persistent queues, a system benefits from improved performance, reliability, and scalability. It isn’t surprising that 99% of large organizations either use microservices now or plan to in the future.

Serverless platforms provision microservices as needed and shut them down immediately thereafter, allowing applications to be highly flexible, inexpensive to operate, and customizable. In this scenario, message queues coordinate large numbers of microservices, which operate autonomously without the need to provision virtual machines or allocate hardware resources.

Message queue software options

There are several message queuing software options available for use today, including the following:

• Apache Kafka is an open source, distributed platform optimized for streaming information.
• RabbitMQ is a flexible, lightweight, and easy-to-deploy platform, and it supports many messaging protocols.
• IBM MQ was originally built for high-volume applications and is valued for its messaging reliability.
• TIBCO Enterprise Message Service features compatibility with software written in Java.

However, this profusion of options can create problems, particularly when two organizations want to exchange messages with each other, but each uses a different queuing solution.

Challenges of message queue observability

For all its virtues, message queuing presents some challenges from an observability perspective. Queuing doesn’t process messages in a well-defined sequence. Therefore, detecting and fixing anomalies can be difficult, and the queue itself can be a black box.

Additionally, it can be challenging to determine why messages are delayed or fail to reach a user. The problem could be in the database, the HTTP connection, the configuration of the message, or an outage on the sending or receiving end. Capturing copies of all messages for forensic purposes consumes system resources and requires time-consuming manual effort.

An observability platform approach

Observability platforms address the challenge of message queue monitoring by capturing and analyzing queue data. The huge volume of topics entering and exiting message queues, particularly in microservices environments, can be extremely challenging for many traditional observability solutions.

The Dynatrace Software Intelligence Platform supports a broad range of queuing platforms, including Apache Kafka, RabbitMQ, and IBM MQ. It automatically discovers and monitors message queues and uses AI to instantly identify issues and their impact on applications and user experience. As a result, the platform scales to meet the requirements of the largest environments and dramatically improves application performance and availability.