Conference PaperPDF Available

Evaluating and Analyzing MQTT Brokers with Stress-testing

Authors:

Abstract and Figures

MQTT (MQ Telemetry Transport) is a simple, lightweight, open-source and widely used publish/subscribe type communication protocol for IoT systems. No matter which radio technology is used to deploy an IoT/Machine-to-Machine (M2M) network, all independent data generating end devices (sensors and actuators) must make their data available through the Internet for further processing, and send control information back. For this, they heavily rely on the special messaging protocols like MQTT designed for M2M communication within IoT applications. This study aims to evaluate the performance of several MQTT Broker implementations by putting them under stress-test. The evaluation of the servers is made in a realistic test scenario, and the comparison of the results is presented by different metrics (CPU, latency, message rates). We also provide a detailed discussion of the applied test conditions (QoS level, message throughput per client and message payload size, etc.). Our results showed that Mosquitto is the most efficient, optimized broker implementation, and Bevywise's MQTT Route is the second with respect to message processing capabilities under full CPU load in all QoS categories.
Content may be subject to copyright.
Evaluating and Analyzing MQTT Brokers with Stress-testing
Biswajeeban Mishra, Biswaranjan Mishra
Abstract: MQTT (MQ Telemetry Transport) is a simple, lightweight, open-source and widely
used publish/subscribe type communication protocol for IoT systems. No matter which radio
technology is used to deploy an IoT/Machine-to-Machine (M2M) network, all independent
data generating end devices (sensors and actuators) must make their data available through
the Internet for further processing, and send control information back. For this, they heavily
rely on the special messaging protocols like MQTT designed for M2M communication within
IoT applications. This study aims to evaluate the performance of several MQTT Broker im-
plementations by putting them under stress-test. The evaluation of the servers is made in a
realistic test scenario, and the comparison of the results is presented by different metrics (CPU,
latency, message rates). We also provide a detailed discussion of the applied test conditions
(QoS level, message throughput per client and message payload size, etc.). Our results showed
that Mosquitto is the most efficient, optimized broker implementation, and Bevywise’s MQTT
Route is the second with respect to message processing capabilities under full CPU load in all
QoS categories.
Keywords: Internet of Things, MQTT, MQTT Brokers, Performance Evaluation, Stress-testing
1 Introduction
Internet of Things devices are growing rapidly and increasingly becoming parts of our lives as
the costs of sensors and actuators are on a continuous decline. Today, the footprint of IoT is
significantly visible everywhere. It is rare to find any industry that does not get revolutionized
with the rise of IoT. IoT Networks use several radio technologies like WLAN, WPAN, etc. to
communicate at the lower level. No matter which radio technology is used to create an M2M
network, the end device or machine must make their data available to the Internet [1]. The
performance of M2M communication heavily depends on the underlying special Messaging
protocols designed for M2M communication within IoT applications. There are many M2M
data transfer protocols are available for IoT systems-MQTT, CoAP, AMQP, and HTTP, to name
but a few. Amongst these M2M protocols, characteristics like lightweight, open, simple and
easy to deploy, make MQTT an ideal protocol for communication in constraint environments
like IoT [2].
MQTT was created in 1999, and became an OASIS standard in 2014. It works on the top
of TCP/IP and uses 1883 and 8883 ports for unencrypted and encrypted communication re-
spectively. Its minimalistic designing principles surrounding less ’network bandwidth’ and
’device resource requirements’ makes MQTT able to transmit telemetry information between
constrained devices (devices having limited processing capabilities and memory) over low-
bandwidth, high-latency or unreliable networks. The MQTT protocol has two kinds of network
entities: a message broker(server) and client( a publisher or subscriber). MQTT based IoT de-
vices/applications(publishers) send or publish messages on a topic head to a server called an
MQTT broker; then the broker delivers the messages to those clients (subscribers) that have
previously subscribed to that topic. There are many MQTT server/broker options available
from different vendors. MQTT provides three Quality of service levels for delivering messages
to an MQTT Broker and any client (ranging from 0 to 2). At QoS 0, a message will not be ac-
knowledged by the receiver or stored and delivered by the sender. This is often called "fire and
forget." It is the minimal level and guarantees the best delivery effort. At QoS 1, acknowledg-
ment is assured. Data loss may occur. At least once delivery is guaranteed. At QoS 2, exactly
once delivery of a message is guaranteed.
The aim of this paper is to evaluate the performance of several MQTT Broker implementa-
tions put under stress. The evaluation of the servers is made in a realistic test scenario, and
the comparison of the results is presented by different metrics (CPU, latency, and message
rates). We also provide a detailed discussion of the applied test conditions (QoS level, message
throughput per client and message payload size, etc.). The remainder of this paper is orga-
nized as follows: Section 2 presents the test topology and details of the MQTT brokers put into
stress, Section 3 details the test scenario, and Section 4 summarizes test results. Finally, Section
5 concludes the paper highlighting the main points of our future work.
2 Test Environment and Test Topology
In this experiment we fire messages at very fast rates from a publisher machine to MQTT server
using a publishing script based on Paho Python MQTT library. The subscriber machine runs
’mosquitto_sub’ command line subscribers. ’mosquitto_sub’ is an MQTT client for subscribing
to topics and prints the messages that it receives. During this test we redirect mosquitto_sub
output to /dev/null to ensure resources are not consumed for printing MQTT messages on the
terminal. Also, we have configured each subscriber to subscribe to all published topics to make
sure a higher server load at reasonable message rates. Figure 1 presents the test topology.
Figure 1: Test Topology
In our local test environment, we have used an Ideapad 330-15ARR, as a publisher ma-
chine running publisher threads. With 8 hardware threads, it is capable of firing messages
at higher rates. At high message publishing rate with multiple publishers overall CPU usage
stays below 70% during the entire course of the experiment on this hardware. MQTT broker
was run on an Intel NUC(Intel Corporation, NUC7i5BNB) and subscribers run on an Intel Core
2 Duo machine(Toshiba, Satellite B40-A)Throughout the test. The CPU usage on the subscriber
side doesn’t exceed 80%. No swap usage observed on the publisher, broker, and subscriber
machines during the tests. Gigabit Ethernet (local network) was used to ensure there is no
network bottleneck. See table 1 for hardware details of test environment.
Table 1: Hardware details of the test environment
HW Details Publisher MQTT Broker Subscriber
CPU: 64 bit AMD Ryzen 5
2500U @3600 MHz
64 bit An Intel(R)
Core(TM) i5-7260U
CPU @ 2.20GHz
Intel(R) Pentium(R)
CPU 2020M
@ 2.40GHz
Memory:
8GB, SODIMM DDR4
Synchronous Unbuffered
(Unregistered)
2400 MHz (0.4 ns)
8GB, SODIMM DDR4
Synchronous Unbuffered
(Unregistered)
2400 MHz (0.4 ns)
2GB, SODIMM DDR3
Synchronous
1600 MHz (0.6 ns)
3 Test Scenario
This section sheds light on our test conditions and the number of brokers evaluated. Following
test conditions are remain constant for all the brokers throughout the experiment - (a) Number
of topics: 3 (via 3 publishers threads), (b) Number of publishers: 3, (c) Number of subscribers:15
(subscribing to all 3 topics), (d) Payload:64 bytes, (e) Topic names used to publish large number
of messages: topic/0’, ’topic/1’, ’topic/2’, (f) Topic used to calculate latency’topic/latency’. La-
tency is defined as the message transmission time from publisher to subscriber. We measured
message delivery time by publishing an MQTT message on a topic different from the topics
used to fire a large number of messages to stress the server. This is to ensure that the process-
ing time of a currently queued messages on broker on a specific topic head doesn’t affect the
processing time of another message published on another topic head. A good implementa-
tion should efficiently handle processing of all message irrespective of rate of messages/topic.
Moreover, this is close to the real world scenario of a broker experiencing extreme load condi-
tions while a client is trying to publish a message on a random topic. In MQTT, every message
gets transmitted as s single telemetry parameter.Hence in this experiment small payload size
is chosen not to overload server’s memory. during the entire test the message payload size is
fixed and set to 64 bytes. All the brokers(servers) were put under stress-test with default config-
uration settings. The evaluated brokers are Mosquitto 1.4.15[2], Active-MQ 5.15.8[4], HiveMQ
Community Edition 2020.2[5], and Bevywise MQTT Route 2.0[3].
4 Test Results: A comparative Analysis
This section presents a comparative analysis of the performance of MQTT brokers based on
the stress-test results. In our test, we found Mosquitto being a single-threaded implementation
beats all other brokers by a huge margin in message processing rate across all QoS categories.
It is the most efficient, optimized implementation with the least latency in QoS1 and QoS2 cat-
egory among all brokers we have tested so far. Bevywise MQTT Route occupies the second
position(after mosquitto) with respect to message processing capabilities @ 100% CPU load in
all QOS categories. We also observed that it has lower latency/message delivery time com-
pared to ActiveMQ and HiveMQ across all QOSes. It has shown better latency(less round the
trip time) than Mosquitto in QOS 0. In this experiment, we have tried to limit the CPU usage
around 100% for the process group to have a fair comparison with other brokers. The figure 2
presents a detailed comparative analysis of test-results.
The message-rate mentioned in the above figure indicates the number of messages/second
required to push the broker to approximate 100% total CPU usage. It is to be noted that some
broker implementations like HiveMQ CE, ActiveMQ can scale up automatically to utilize avail-
able resources on the system. These brokers create multiple threads or sub-processes to handle
the higher message load. The CPU utilization data presented in this table is for the process
group(consisting of all sub-processes/threads) of MQTT Server. So, CPU utilization percent
for these brokers can go up to 400% on a machine with 4 cores. However, we are not consid-
ering that in our experiment as other brokers in the test are single-threaded and don’t scale up
automatically.
5 Conclusion
M2M protocols are the backbone of communication in IoT systems. There are many M2M
communication protocols are available such as MQTT, CoAP, AMQP, and HTTP. In this paper,
we investigated and analyzed the performance of MQTT brokers in terms of projected message
rate @100% CPU usage and the average time taken for message transmission by putting them
under stress-test. Our results showed, Mosquitto being a single-threaded implementation beats
Figure 2: A comparative presentation of test results
all other brokers by a huge margin in message processing rate across all QoS categories. It is
found to be the most efficient, optimized MQTT broker implementation among all the brokers
we put into test. Bevywise MQTT Route occupies the second position(after Mosquitto) with
respect to message processing capabilities @~100\% CPU load in all QOS levels, and broker
implementations like HiveMQ CE, ActiveMQ can scale up automatically to utilize available
resources on the system. In the future, it would be interesting to observe the performance
of MQTT brokers in a cloud-deployed test environment. We plan to widen the scope of this
research by putting more brokers under stress-test and create a generic, easily configurable
message blaster for MQTT brokers.
References
[1] Nitin Naik, Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP
and HTTP. IEEE International Systems Engineering Symposium, Vienna, pp. 1-7, 2017.
[2] MQTT Version 5.OASIS Standard, https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-
v5.0.html. Accessed in March, 2020.
[3] Bevywise MQTT Route Developer’s Guide, https://www.bevywise.com/mqtt-
broker/developer-guide. Accessed in March, 2020.
[4] ActiveMQ 5 Documentation, https://activemq.apache.org/components/classic/documentation/.
Accessed in March, 2020.
[5] HiveMQ Editions, https://www.hivemq.com/developers/community/. Accessed in
March, 2020.
[6] Mosquitto man page, https://mosquitto.org/man/mosquitto-8.html. Accessed in March,
2020.
ResearchGate has not been able to resolve any citations for this publication.
Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP
  • Nitin Naik
Nitin Naik, Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP. IEEE International Systems Engineering Symposium, Vienna, pp. 1-7, 2017.