Cluster forming with Mosquitto broker
Original source of this answer: Horizontal scaling for brokers. I just added the plugin support.
Two functionalities of mosquitto broker combined could be used to setup a n node cluster.
- Mosquitto bridge support.
- Plugin support - mosquitto-auth-plug(1)
The bridge functionality is used ONLY to synchronize messages between all mosquitto brokers while the mosquitto-auth-plugin could be used to save authorization and ACL in a single database back end.
While setting up the bridge, notice the usage of in/out bridge topics to avoid forwarding loops(2). The pattern is of format
topic pattern [[[ out | in | both ] qos-level] local-prefix remote-prefix]
Quoting mosquitto.conf man page(3), For incoming topics, the bridge will prepend the pattern with the remote prefix and subscribe to the resulting topic on the remote broker. When a matching incoming message is received, the remote prefix will be removed from the topic and then the local prefix added. And vice versa for an outgoing topic
Sample mosquitto.conf for a 3 node cluster is below. To be load tested with mqtt-malaria(4) and more importantly the effect on clients connecting with clean_session flag to false, .
+-------------------+
+------------------> BRIDGE BROKER <------------+
| | 192.168.1.1 | |
| +--------^----------+ |
| | |
| | |
| | |
Broker A | Broker B | | Broker C
| | |
+----------------------+--+ +----------------+--------+ +--+----------------------+
| connection A | | connection B | | connection C |
| | | | | |
| address 192.168.1.1:1883| | address 192.168.1.1:1883| | address 192.168.1.1:1883|
| | | | | |
| topic # out 2 "" A/ | | topic # out 2 "" B/ | | topic # out 2 "" C/ |
| topic # in 2 "" B/ | | topic # in 2 "" A/ | | topic # in 2 "" A/ |
| topic # in 2 "" C/ | | topic # in 2 "" C/ | | topic # in 2 "" B/ |
+----------------------^--+ +----------------^--------+ +--+----------------------+
| | ^
| | |
| | |
| +-------+---------+ |
+-------------------+ HA PROXY +-------------+
+-----^--^--------+
| |
| |
+ +
There are 2 (for this time of writing) major open source projects that support large scale MQTT deployments and clustering
VerneMQ and EMQ are both written in Erlang, which is very suitable for distributed messaging, having all the replication as built in toolset part of the (Erlang) language.
VerneMQ has commercial enterprise support, while EMQ is planning to in the near future, also both projects have large companies backing them as sponsors.
- VerneMQ (https://vernemq.com)
- EMQ (http://emqtt.io)
You may also look into HiveMQ which is the oldest kid in the block, yet cannot be used without a payed license.
- HiveMQ (http://www.hivemq.com)
You can't do clustering with mosquitto.
Some other MQTT brokers out there which support clustering, including HiveMQ. HiveMQ has an elastic cluster ability with auto discovery and a distributed masterless architecture and works very well on cloud providers like AWS or Azure.
You can see a full list of all brokers which support clustering here.
Disclaimer: I'm a developer of HiveMQ, so this answer may be biased.
It's not on the list mentioned by Dominik but Solace Systems has an MQTT appliance that supports clustering. Each broker supports several hundred thousand concurrent client connections.
Disclaimer: I am a Systems Engineer at Solace