EMQ Clusters

Erlang MQTT or EMQ Broker is acclaimed as one of the best open source MQTT brokers available in the market to serve the needs of a large scale enterprise. It is powered by Erlang/OTP and can support millions of concurrent MQTT clients. To read more about Erlang MQTT Broker, follow this link.
Features inherent to EMQ that have earned it the right fame are:

• Distributed
• Fault tolerant
• Massively scalable
• Highly extensible

In this blog, I shall attempt to provide a list of simple steps to create an EMQ cluster with two nodes and then connect to the cluster using a MQTT subscriber and a publisher and route messages between them. I shall assume the reader's moderate acquaintance with MQTT, Oracle VM VirtualBox and Raspberry Pi 3. Here are the components I'll use:

• EMQ cluster with two nodes in a Ubuntu 16.04 Oracle Virtual Box VM
• A MQTT subscriber connected to the second node of the cluster
• A MQTT publisher connected to the first node of the cluster (from Raspberry Pi)

To add a little twist to the tale, I'll use Raspberry Pi 3 as the publisher and watch the messages sent from Pi connected to the first node of the EMQ cluster, getting received by the MQTT subscriber connected to the second node.
With that preamble in place, let's start creating the components.

Download and start the EMQ broker

Steps to download and run the EMQ broker in Ubuntu 16.04 
(Note: There are targeted EMQ packages for the specific OS you are using)

1. Login to Ubuntu (I am using a superuser- adrin)
2. Download EMQ package for Ubuntu 16.04 (emqttd-ubuntu16.04-v2.3-rc.2.zip) directly OR use wget command to download the package 
   
    
3. Unzip the archive to your favorite location using unzip command. Once unzipped you   should see a new folder named 'emqttd' created. A snapshot of the contents of this folder is given below 
   
4. Start the broker from within the bin directory by running emqttd with start option and then check the status of the running broker using emqttd_ctl 
   
5. By default, EMQ node listens to port 1883 for client requests and port 8883 for SSH requests. EMQ also provides an admin console which runs on port 18083. Let's take a peek into the dashboard of this console. The default username/password is admin/public. From this console, you can easily see that one EMQ node (named emq@127.0.0.1) is running in the localhost.
   

Create EMQ cluster

Every node in the EMQ cluster stores a topic trie and topic route table in Mnesia database. When a MQTT client subscribes to a topic, the node it is connected to, broadcasts to all the other nodes in the cluster that it has subscribed to that topic and updates the topic route table (a map that stores topic to nodes entries). Similarly, when a MQTT client publishes a message to a topic, the node it is connected with looks up the topic route table to get the list of nodes that this message needs to be forwarded to. So essentially the EMQ cluster, replicates the topic trie and topic route table across all the nodes. For more details about EMQ cluster please visit this link.

Creating a EMQ cluster is pretty straightforward since when a node is started, it is already part of a default cluster (cluster name= emqcl). Every EMQ node is part of this default cluster in which it runs. This cluster configuration is available in emq.conf file. 

Following are the steps to start the second node and create the cluster with two nodes:

1. Start another EMQ node to create a cluster. Now since we will be starting the second node in the same VM, we will have to use a different set of listen ports for this node. To start the second node, we copy the contents of 'emqttd' folder and name it as emqttd2. Use: $cp -R emqttd/ emqttd2/
2. Open emqttd2/etc/emq.conf and use your favorite editor to change the following properties (Note: If you start the second node in another machine, then tweaking the conf file to change the listening ports may not be required since there will be no chance of a port conflict):
• node.name = emq2@127.0.0.1
• listener.tcp.external = 0.0.0.0:1884
• listener.tcp.internal = 127.0.0.1:11884
• listener.ssl.external = 8884
• listener.ws.external = 8084
• listener.wss.external = 8085
• listener.api.mgmt = 127.0.0.1:8081
3. Now start the second node using the same emqttd command with start option, this time from within emqttd2/bin (Note: The name of this second node is emq2@127.0.0.1 which was specified in emq.conf file)
   
4. Check the cluster status using emqttd_ctl. You'd notice that there is only one node in the default cluster
   
5. Make the second node to join the cluster of the first node using the emqttd_ctl command. Once the second node joins the cluster, the cluster status shows two nodes in the cluster. You could actually do it in another way as well. That is, you could make the first node to join the cluster of the second node.
6. A peek in the admin console's dashboard shows the updated cluster status with two nodes running 
   

Run the subscriber

We could easily create a fully functional subscriber from the EMQ admin console, or your favorite programming language. However here I'll do something crazy and use pre-packaged clients of another MQTT broker called Mosquitto.
Following steps will set up the subscriber:

1. Use apt-get to install 'mosquitto-clients'. This will include a set of MQTT publisher and subscriber. In case this doesn;t work, add the mosquitto repository to apt-get and then try: $sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
2. Now run 'mosquitto_sub', a pre-packaged MQTT subscriber and connect with second node of the cluster. There are quite a few options that could be used along with the subscriber. However, here I will be using only a few of these options. (Note: In this example, the MQTT subscriber is run from the same VM that hosts the EMQ cluster, however this is not necessary)
• -d : Debug
• -t  : MQTT Topic name (One can use wildcards like +, # in topic names)
• -q : QOS (1 = At most Once/Fire and forget) 
• -h : Hostname
• -p : Listener port of the broker (second EMQ node)
  
  

Run the publisher

As already stated, I'll use Raspberry Pi 3 to create a MQTT publisher. However since this client will connect with the EMQ cluster from outside the Oracle VM VirtualBox, we need to take some measures to allow the traffic to reach the guest VM (Ubuntu 16.04). The easiest way to do this is to create a port forwarding in Oracle VM VirtualBox.
Right click on the VM, go to Settings, select 'Network' tab. Ensure that NAT network adapter is being used and make the following changes:

• In advanced section, click on Port Forwarding and in the 'Port Forwarding Rules' dialog, add the following rules and click on OK.This means that, any traffic coming to host machine's 1883 port will get forwarded to the guest machine's 1883 port. Do note that 1883 port of Ubuntu 16.04 VM is hosting the first EMQ node
  

Additionally, note the IP address of the host machine. In my case, I issued Window's ipconfig command to know the IP address of the host. This will have to be configured in the MQTT publisher hosted in Raspberry Pi 3.

Here are the steps to create a mosquitto publisher in Raspberry Pi 3 and connect with the EMQ broker (first node) hosted in a Oracle VM VirtualBox.

1. Use putty (SSH service must be running in Pi) or vnc-viewer (VNC service must be running in Pi) to connect to Pi 
   
2. Use apt-get package manager to install 'mosquitto-clients' in Pi 
   
3. Run a mosquitto publisher (mosquitto_pub) and connect it with the first EMQ node. Following options have been used along with mosquitto_pub:
• -d : Debug
• -t  : MQTT Topic name
• -q : QOS (2 = At least once) 
• -m: The message payload to be published
• -h : Hostname
• -p : Listener port of the broker (first EMQ node)
4. Now watch this message pop-up in the mosquitto subscriber which was connected to the second EMQ node (on port 1884)

This points to the fact that the replication mechanism (described in 'Create EMQ Cluster' section) is working between the EMQ nodes in the cluster. The details of this replication based on 'topic trie' and 'topic route table' is captured in this link.

In production, we should place a highly available load-balancer in front of the EMQ nodes for a scalable and fault tolerant solution. That way, the clients are shielded from the location details of the nodes and the traffic is balanced between the nodes. Additionally we need to look into the security aspects of the cluster.

In another blog post, I have tried to explain how to create EMQ cluster on AWS EC2 instances using ELB to balance the traffic.