MQTT Protocol

Internet of Things (IoT) networks rely on messaging protocols to transfer data to and from devices. There are many protocols you can use to accomplish this. The MQTT protocol is a popular choice for several use cases thanks to its publish/subscribe framework, delivery guarantees, and ability to function with low-power devices in poor network conditions. Read on to learn how this protocol works, how to get started with an implementation, and best practices.

What Is the MQTT Protocol?

The Message Queuing Telemetry Transport (MQTT) protocol is a standardized, lightweight, bi-directional messaging protocol commonly used to connect IoT devices. Devices are able to exchange data by using a centralized server that receives one device's message and then relays it to all others listening for messages at a specific location. MQTT is built to work with low-power devices and enable communication between devices regardless of network stability, bandwidth, and latency. It is highly scalable and able to accommodate millions of devices securely. This makes it ideal for industrial IoT (IIoT) use cases, like factories where machines contain sensors that report their statuses, send alerts, provide production updates, and receive commands. However, you can also use it for connecting smart home and other consumer devices.

How Does MQTT Work?

The MQTT protocol works over TCP and uses a client-server Publish/Subscribe (or Pub/Sub) messaging framework. Devices, referred to as clients, publish their messages to a centralized MQTT server running broker software. Clients also subscribe to topics to receive relevant messages from the broker. MQTT's flexibility allows it to run on a myriad of platforms, usually via an MQTT library. The libraries can be run on Windows, macOS, iOS, Android, various Linux distros, and more. They support many common programming languages, like JavaScript, TypeScript, Flutter, C#, C++, C, Python, and others. Clients can range from telematic units in vehicles, temperature sensors, moisture sensors for agriculture, medical devices like insulin pumps, smartwatches, and many other IoT devices. They can also be workstations, desktops, laptops, or similar. Brokers are programs that run on a server and serve as the middleman between clients. You manage them and their connected devices via a CLI or GUI. Broker software can be open source or commercial, and it can be on-premise or in the cloud. A broker might be managed by a third party with their own proprietary solution in a cloud, by a first-party company on a public cloud or on-premise servers, or by individual IoT enthusiasts who use open-source software on a desktop or Raspberry Pi. Brokers enable client decoupling. Clients can send or receive commands and data in the form of messages via the broker without connecting directly with each other or needing to: 

• Know the other client's ID or address
• Be active at the same time
• Be in a ready state at the same time

Clients connect to the message broker by sending a CONNECT command packet and receiving a CONACK acknowledgment back. The CONNECT message must contain the MQTT protocol name and version, flags related to the nature of the connection, the keep alive interval (the time the broker should wait for a client's message before the client is considered disconnected), and sometimes the client ID. CONNECT packets might also include optional content, such as a username and password or information relating to Last Will and Testament (LWT) messages and Quality of Service (QoS). The messages a subscriber client receives from the broker are determined by the topics they subscribe to, which function similarly to URLs in HTTP. The topic for a temperature sensor might look like mysmarthome/livingroom/thermostat/temperature, with the forward slash functioning as a topic-level separator. For example, an industrial robot at a car manufacturing plant has sensors that monitor productivity and maintenance needs. These sensor clients publish their status information to a broker that relays this information to clients subscribed to status-based topics, such as a monitoring dashboard on a workstation and a relational database for keeping status logs of the machines at the plant. The industrial robot is a subscriber to topics relating to commands and maintenance. The dashboard on the workstation can publish messages to the broker for the robot to perform tasks like turning off for maintenance or updating its firmware.

MQTT Message Structure

MQTT messages contain a mandatory fixed header, an optional variable header, and an optional payload. The fixed header comprises message type, flags, and message length. The variable header will depend on the fixed header's message type, but it might contain the topic name, flags, packet identifiers, and more. The content of the payload varies depending on the specific use case. Industrial sensor payloads might have status updates in JSON, XML, or binary format, whereas an IoT camera payload might be a JPEG in binary.  

Quality of Service Levels

There are three different QoS levels for MQTT. These levels determine the message delivery guarantee. They are:

• QoS 0: This level is also known as "at most once" because the message is sent only once. If the subscribers do not receive the message, it is lost, and the broker does not try to resend it. There is no acknowledgment of receipt or lack thereof.
• QoS 1: This level is also called "at least once" because the receiving device will get the message at least once, although it might receive duplicates.

QoS 1 consists of a two-part handshake between publisher and broker. The publisher sends a PUBLISH packet, and the broker sends back a PUBACK packet. If no PUBACK packet is received due to the PUBLISH or PUBACK packet dropping, the publisher continues sending the PUBLISH message with a DUP duplicate flag until it receives the acknowledgment packet. This process works the same when brokers send messages to subscribers.

• QoS 2: This level is also known as "exactly once." It works similarly to QoS 1 but instead has a four-part handshake between the sending and receiving parties to make sure each message is received only once. When publishers send a message to the broker, they do so with a PUBLISH control packet. If the transmission is successful, the broker sends back a PUBREC packet. The publisher then sends a PUBREL packet, letting the broker know they can pass the message along. The handshake ends when the broker replies with a PUBCOMP. If the initial PUBLISH packet isn't acknowledged with a PUBREC packet, it will be sent again with a DUP flag, indicating it's a duplicate. Like QoS1, this process is the same for brokers sending messages to subscribers.

Reliability Features

In addition to QoS, there are other ways to provide message reliability using the MQTT protocol in IoT.

Persistent Sessions and Clean Sessions

MQTT can facilitate clean or persistent sessions for individual clients via the cleanSession flag. When a client's cleanSession flag is set to true in their CONNECT message, their connection with the broker will be non-persistent. When set to false, it will be persistent. Persistent sessions allow a broker to retain a client's missed and queued QoS 1 and QoS 2 messages, received but unacknowledged QoS 2 messages, and its subscriptions. These sessions ensure that a client who needs reliable delivery receives every message. For instance, in IIoT settings, clients who monitor machines for critical status updates and can't miss any data points will likely use persistent sessions. When a client disconnects during a clean session, the broker discards queued information and messages. This session type is intended for clients who send but don't receive messages or who don't need to receive every message, like a temperature sensor that only publishes weather data.

LWT, Graceful Disconnects, and Ungraceful Disconnects

In MQTT, there are graceful and ungraceful disconnects. A disconnect is graceful when it is expected and includes a disconnect message informing the broker that it has disconnected. An ungraceful disconnect lacks this message because it happens unexpectedly. In such cases, developers can set a Last Will and Testament (LWT) message during the client's initial CONNECT message for the broker to store. This message will be broadcast to subscribers of a chosen topic in the case of an ungraceful disconnect.

How Do You Choose the Right MQTT Provider?

Since IoT takes many forms, including commercial and consumer use cases, MQTT implementations vary wildly. A manufacturing company with thousands of endpoints requires scalability and performance that simply isn't needed by an aspiring developer or IoT enthusiast working on a smart home project. Much of what your devices can do over MQTT will depend on your choice of broker and plan (if you choose a paid broker). If you're just an individual, you might use a simple Raspberry Pi with free, open-source software to handle an automated plant watering system or a wireless LED display. However, for MQTT implementations with a greater amount of devices, topics, and messages, you'll have to consider several factors when looking for the right broker, including:

• Performance and scalability: Broker and plan choice greatly impact how many messages you can send per minute, how many devices can send and receive them, and how much latency there will be in delivery.
• Security and compliance: MQTT has many options for encryption, authentication, and authorization, like TLS/SSL, virtual private clouds, and role-based access options. Available security features and compliance with relevant national like GDRP or industry laws like HIPAA will vary by both vendor and plan.
• Support: Most companies will offer some level of support, but the amount of hands-on help, resolution speed, and any uptime guarantees will depend on the plan.
• Billing and budget: Vendors often offer different billing options, such as pay-as-you-go, flat rate, and special enterprise plans. Some offer limited free plans for small-scale IoT projects. You may have to shop around to find a vendor that can meet your needs within budget as prices can vary from under $10/month to well over $10,000/month.

Consider two different companies using IoT systems.

One is a large automobile manufacturing plant with thousands of clients, including machinery, dashboards, and various sensors for things like temperature, battery status, and energy consumption. The other is a small retail outlet that uses movement sensors to track customer traffic throughout the store. The manufacturing plant works with a greater number of clients that are steadily sending and receiving messages. This plant will need a highly scalable and performant broker that can accommodate these messages and respond quickly if something fails, along with a plan that includes high uptime guarantees and responsive support. On the other hand, the store won't have such high scalability and performance requirements because the data their sensors report will be less critical and won't suffer from a bit of latency.

Best Practices for Using MQTT

Whether working on a project at home or improving your IoT systems at work, there are some MQTT best practices you should consider following for QoS, securing communication, topic structure, and testing.

Pick the Right QoS Levels

The three QoS levels each serve their own purpose, and you might use all of them at once if you're working with several clients.

QoS 0 is suited for situations where data will be reported often and losing some messages won't have a negative impact. This level would be a good choice for non-critical IoT devices reporting their battery percentage. QoS 1 works when a message must be received, but duplicates won't cause issues. A manufacturing plant's dashboard might subscribe to topics related to the performance reports of the IIoT machines; each report is important enough to need a delivery guarantee, but the dashboard can filter out duplicate messages. QoS 2 is for when messages must be received, and you cannot tolerate duplicates. Clients that send or receive important medical information or financial transaction data that cannot be repeated should use QoS 2. While it might seem like QoS 2 should be the default, the higher the QoS level, the higher the message overhead and resultant latency. This can worsen performance and increase costs, so pick the right level for each topic a client publishes and subscribes to.

Secure MQTT Communication

With potentially thousands of devices sending messages at once, IoT expands your attack surface and can leave networks vulnerable if unprotected. Thankfully, MQTT comes with several ways to protect your data and devices. By default, payloads are in plain text, so it's important to encrypt them. Even with free plans, most brokers will offer the ability to use Transport Layer Security (TLS). It's also possible to set it up yourself when self-hosting by obtaining your own TLS certificate and configuring the broker and each client to use TLS. Limit the authorization of clients by role via access control lists in the broker, with plugins, or with a system like Active Directory. You can control access by requiring a username and password for each client on the network, which you can set in the broker. If a hacker gets access to your system, a compromised temperature sensor that only has permission to publish to one topic does much less damage than one with unrestricted access. You should also employ general cybersecurity best practices, like using firewalls and VPNs and encrypting your password lists. Security options will depend on the broker and plan, so consult their documentation for a better idea of what the developers recommend.

Optimize Topic Structure

When setting the message topics that clients subscribe to, you should:

• Use the forward slash to separate topic levels only, never at the start or end of the topic, because the slash will add an unneeded level. location1/aisle8/motion is optimal with only three levels, whereas /location1/aisle8/motion/ has a total of five levels, including two unnecessary ones.
• Only use ASCII characters when creating topics. Using non-ASCII characters can cause display errors or other issues that can make it harder for clients to receive messages from or send messages to them.
• Be descriptive, consistent, logical, and to the point when creating your topic hierarchy. Remember our example topic mysmarthome/livingroom/thermostat/temperature? You know what messages to receive from what device and what part of the home from this hierarchy. It goes from the broadest category (smart home) to the most specific (temperature).
• Never include sensitive information in your topics, such as usernames or passwords.

When dealing with smaller IoT projects, following this advice might not matter much, but if you're working with thousands of clients, these tips prevent your implementation from becoming unwieldy.

Test Before Deploying

As with any other system, you should always test before you go live with it. You can test performance and scalability under high-stress loads and measure the latency with testing tools. Try to get past your own protective measures to see if there are any holes in your security posture, especially when dealing with sensitive data. You can start by running Wireshark on your packets to make sure they're encrypted and that no one can sniff out passwords in CONNECT messages. Read your plan conditions carefully and configure your clients properly to avoid unforeseen charges. Mismanaged devices sending high amounts of data can cause costs to skyrocket if they're not carefully monitored.

Know When to Use Alternatives to MQTT for IoT

MQTT is a great choice in many IoT scenarios, but sometimes it might not be enough or you may be unable to use it with some devices. In such situations, you might consider using one of the following alternatives:

• Hypertext Transfer Protocol (HTTP): When working with web servers or REST APIs, HTTP might be preferred over MQTT, as it is the standard communication protocol of the internet. 
• Constrained Application Protocol (CoAP): CoAP works well with low-powered devices, too, but it uses HTTP-like methods. It also uses UDP instead of TCP. It's the better choice for working with more heavily constrained equipment.
• Advanced Messaging Queuing Protocol (AMQP): AMQP is a popular protocol for financial and enterprise use cases due to its flexibility in messaging patterns, support for larger messages, and its greater granularity in routing compared to MQTT's simpler topic-based system. 
• Data Distribution Service (DDS): DDS also uses a publish/subscribe framework, but it lacks the need for brokers by being peer-to-peer. It has more defined QoS options, and it's geared for real-time, high-performance devices.

Frequently Asked Questions