Create an Cluster

With EMQX cluster, you can enjoy the benefits of fault tolerance and high availability by allowing the cluster to continue operating even if one or more nodes fail. You can create an EMQX cluster either manually or automatically. Before the actual clustering step, let’s first get familiar with 2 key concepts, node names and node discovery.

Concepts

Node Names

Before starting the cluster creation step, let’s first get familiar with the concept of node names in EMQX. EMQX nodes are identified by their names. A node name consists of two parts, node name and host, separated with @, for example, emqx@s1.emqx.io. The host part must either be the IP address or a fully qualified domain name (FQDN), such as myhost.example.tld, for example:

  • For EMQX node deployed on server s1.emqx.io, the node name should be emqx@s1.emqx.io;
  • If this server has a static IP (192.168.0.10), the node name should be emqx@192.168.0.10.

Tip

EMQX node names are immutable, as they are baked into the database schema and data files. Therefore, it is recommended to use static FQDNs for EMQX node names.

Node Discovery

EMQX’s clustering feature is based on the EkkaCreate an Cluster - 图1 (opens new window) library, a cluster management library developed for Erlang/OTP applications.

One of the crucial steps for EMQX clustering is node discovery, which enables individual EMQX nodes with different IP addresses or locations to discover and communicate with each other. EMQX supports multiple node discovery strategies:

StrategyDescription
manualCreate a cluster through manual command
staticCreate a cluster using a static node list
dnsCreate a cluster using DNS A and SRV records
etcdCreate a cluster via etcd
k8sCreate a cluster via Kubernetes service

Before You Begin

Before creating an EMQX cluster, the following prerequisites should first be met:

  1. All nodes are set with a unique node name in the format of name@host, where host must be an IP address or fully qualified domain name (FQDN). For more information on configuring node names, see Configure Node Names.

  2. If there is a firewall or security group between nodes, ensure the cluster communication port has been opened. For details, see Intra-cluster communication port.

  3. For security concerns, you should change the default cookie settings to a Secret cookie in emqx.conf on all nodes to join the cluster. Note: All nodes to join the cluster should use the same Secret cookie. For details about the magic cookie used, see Distributed Erlang - SecurityCreate an Cluster - 图2 (opens new window).

    1. node {
    2.   cookie = "<a Secret cookie>"
    3. }

:::

Now you can begin your journey to create an EMQX cluster.

Configure Node Names

Before creating a cluster, you need first to name the nodes to join the cluster. Suppose you want to create a cluster for 2 nodes deployed in s1.emqx.io and s2.emqx.io respectively, you can follow the steps below to create the cluster.

Configure the node name in the emqx.conf configuration file of the 1st node, for example:

  1. node.name = emqx@s1.emqx.io

You can also override the node name with an environment variable:

  1. env EMQX_NODE__NAME='emqx@s1.emqx.io' ./bin/emqx start

Repeat the above step for the other node to join the cluster.

Now you have named 2 nodes to join the cluster, emqx@s1.emqx.io and emqx@s2.emqx.io. You can create a cluster either manually or automatically.

Manual Clustering

Set Node Discovery Strategy

EMQX supports creating clusters manually and automatically. This section will introduce the manual clustering feature in EMQX.

Set node discovery strategy

Manual clustering is the method to configure an EMQX cluster by manually specifying which nodes should be part of the cluster. By default, EMQX adopts a manual clustering strategy, which can also be set in emqx.conf. Besides, you also need to configure the default

  1. cluster {
  2.     ## options: manual | static | dns | etcd | K8s
  3.     discovery_strategy  =  manual
  4. }

Configure Nodes to Join a Cluster

After the nodes are started, you can run the cluster join command on the node that you want to join the cluster. For example, you want emqx@s2.emqx.io to join emqx@s1.emqx.io, run the command below on emqx@s2.emqx.io：

  1. $ ./bin/emqx_ctl cluster join emqx@s1.emqx.io
  3. Join the cluster successfully.
  4. Cluster status: [{running_nodes,['emqx@s1.emqx.io','emqx@s2.emqx.io']}]

TIP

  1. This command must run on the node to join the cluster, that is, as a request rather than invite.

  2. After emqx@s2.emqx.io joins emqx@s1.emqx.io to form a cluster, it will clear the local data and synchronize the data in emqx@s1.emqx.io.

  3. If emqx@s2.emqx.io wants to join another cluster, it must first leave the current cluster. On how to leave the cluster, see Manage cluster nodes.

Now you have successfully created a cluster with two nodes, you can read the Query Cluster Status, Manage Cluster Nodes, and Configure Network Protocols sections on how to monitor the cluster status and how to manage the cluster.

Auto Clustering

Auto clustering in EMQX is another feature that allows multiple EMQX nodes to form a cluster automatically without manual configuration. Auto clustering simplifies the process of setting up an EMQX cluster and makes it easier to add or remove nodes from the cluster dynamically.

EMQX supports auto clustering based on static node list, DNS Record, etcd, and Kubernetes. Continue to read to learn how to work with these features.

Autocluster by Static Node List

In EMQX, autocluster by static node list is to use a pre-defined static node list on each node to join the cluster. After starting, the nodes will create a cluster automatically according to the node list.

Static clustering is the easiest way to create an EMQX cluster automatically with no dependencies on other network components or services. As long as each node can communicate with each other through the TCP protocol, they can form an EMQX cluster.

To enable this feature, configure the cluster mode and node list in emqx.conf:

Example code:

  1. cluster {
  2.     discovery_strategy = static
  3.     static {
  4.         seeds = ["emqx@s1.emqx.io", "emqx@s2.emqx.io"]
  5.     }
  6. }

Where,

  • discovery_strategy is the node discovery strategy, set it to static
  • seeds is an array, where you can add the node to join the cluster, multiple nodes can be separated with ,

After all nodes are started, the cluster will be automatically established.

Autocluster by DNS Records

EMQX supports auto clustering by DNS A records and DNS SRV records. Domain Name System (DNS) record is a type of record used to map domain names to IP addresses. Because multiple DNS records are allowed for one domain name, EMQX leverages this feature to implement auto clustering.

Configure DNS Services

Most public cloud services have DNS services. After assigning a domain name, you only need to add the IP address of each EMQX node to the A record of this domain to finish the configuration.

TIP

If EMQX is deployed in a private cloud or internal network, you will need to deploy your own DNS system, for example, with software BINDCreate an Cluster - 图3 (opens new window).

Configure Autocluster by DNS Records

After the DNS service is ready, you can add all nodes to join the cluster in emqx.conf with the cluster.dns configuration item:

Example code:

  1. cluster {
  2.     discovery_strategy = dns
  3.     dns {
  4.         name = "localhost"
  5.         ## support DNS A record and DNS SRV record
  6.         record_type = a
  7.     }
  8. }

Where,

  • discovery_strategy is the node discovery strategy, set it to dns
  • cluster.dns.name is a a string, input the localhost
  • cluster.dns.record_type is a enum, optional value: a or srv

After all nodes are started, the cluster will be automatically established.

Autocluster Using etcd

etcdCreate an Cluster - 图4 (opens new window) is an open-source project initiated by CoreOS. It is widely used in distributed systems for service discovery and connection establishing, which is exactly what EMQX auto clustering needs.

After you deploy an etcd server (cluster) in your network, EMQX can automatically create the cluster via etcd. For how to install and configure etcd, see etcd InstallCreate an Cluster - 图5 (opens new window).

To enable autocluster using etcd, you can work with the cluster.etcd configuration items in emqx.conf.

Example code:

  1. cluster {
  2.     discovery_strategy = etcd
  3.     etcd {
  4.         server = "http://127.0.0.1:2379"
  5.         prefix = emqxcl
  6.         node_ttl = 1m
  7.     }
  8. }

Where:

  • discovery_strategy is the node discovery strategy, set it to etcd
  • cluster.etcd.server is the server address, multiple nodes can be separated with ,
  • cluster.etcd.prefix is the etcd key prefix used for EMQX service discovery
  • cluster.etcd.node_ttl is a duration, indicating the expiration time of the etcd key associated with the node, default: 1m

After completing the configuration, you can start the EMQX nodes one by one, and use the etcdctl tool to observe the changes on the etcd server:

  1. $ etcdctl ls /emqxcl/emqxcl --recursive
  3. /emqxcl/emqxcl/nodes
  4. /emqxcl/emqxcl/nodes/emqx@s1.emqx.io
  5. /emqxcl/emqxcl/nodes/emqx@s2.emqx.io

The result shows that all nodes are started normally and joined the cluster automatically.

Autocluster on Kubernetes

The EMQX Kubernetes OperatorCreate an Cluster - 图6 (opens new window) helps you quickly create and manage EMQX clusters on a Kubernetes environment quickly, greatly simplifying the EMQX cluster deployment and management process by turning deployment and management efforts into a low-cost, labeled, repeatable job.

If you want to deploy and manage EMQX by yourself, you can still use Kubernetes API for node discovery and auto clustering. To use this feature, you need first to create RBAC for the EMQX Pod to allow EMQX to get cluster node information from the Kubernetes APIServer via the endpoints resource. On how to configure, see Using RBAC AuthorizationCreate an Cluster - 图7 (opens new window).

To enable EMQX autocluster on Kubernetes, you can work with the cluster.k8s configuration item in emqx.conf.

  1. cluster {
  2.     discovery_strategy = K8s
  3.     K8s {
  4.         apiserver = "http://10.110.111.204:8080"
  5.         service_name = emqx
  6.         address_type = ip
  7.         namespace = default
  8.     }
  9. }

Where:

  • discovery_strategy is the node discovery strategy, set it to K8s
  • cluster.K8s.apiserver is the Kubernetes API endpoint URL, default: http://10.110.111.204:8080
  • cluster.K8s.service_name is the EMQX service name, default: emqx
  • cluster.K8s.address_type is the address type to connect the discovered nodes, default: ip, optional values: ip, dns, hostname
  • [optional] cluster.K8s.suffix is the node name suffix, only needed when cluster.K8s.address_type is set to dns, default: pod.local
  • cluster.K8s.namespace is the Kubernetes namespace, it is a string object, default: default

Start all nodes one by one after the configuration, and the cluster will be automatically established.

TIP

When working EMQX autocluster on Kubernetes, CalicoCreate an Cluster - 图8 (opens new window) rather than Fannel plugin is recommended.

After the cluster is succefully created, you can refer to the section below on how to monitor the cluster status and how to manage cluster nodes.

Query Cluster Status

Run the command below on any cluster node to query the cluster status:

  1. $ ./bin/emqx_ctl cluster status
  3. Cluster status: [{running_nodes,['emqx@s1.emqx.io','emqx@s2.emqx.io']}]

Manage Cluster Nodes

You can remove a node from a cluster with cluster leave or cluster force-leave:

When an EMQX node issues the cluster leave command, it notifies the other nodes in the cluster that it intends to leave, and it stops participating in cluster operations, it will complete any ongoing tasks before leaving.

When an EMQX node issues the cluster force-leave <node@host> command, the target node will be forcefully removed from a cluster. This command is typically used when a node fails or becomes unresponsive.

For example, in the previously built cluster, if emqx@s2.emqx.io wants to leave the cluster, you can run the command below on emqx@s2.emqx.io:

  1. ./bin/emqx_ctl cluster leave

Or run the command below on emqx@s1.emqx.io to remove emqx@s2.emqx.io from the cluster:

  1. ./bin/emqx_ctl cluster force-leave emqx@s2.emqx.io

After the cluster is created, you can continue to set the network protocols for the nodes. EMQX supports connecting the nodes via TCP or TLS.

Configure Network Protocols

Each Erlang node can be connected via TCP or TLS, and the connection method can be configured in emqx.conf:

To use TCP IPv4 and TCP IPv6, you can set with the cluster.proto_dist in emqx.conf.

  • TCP IPv4: inet_tcp (Default)
  • TCP IPv6: inet6_tcp

To enable SSL, you first need to set the cluster.proto_dist to inet_tls, then configure the ssl_dist.conf file in the etc folder and specify the TLS certificate. For details, see Using TLS for Erlang DistributionCreate an Cluster - 图9 (opens new window).