Performance Test

After you deploy the EMQX either in a single mode or in an EMQX cluster, you can test the performance of your deployment to know the system capability. This section introduces how to install and use the eMQTT-BenchPerformance Test - 图1 (opens new window) to do the performance test. The eMQTT-Bench is a concise and powerful MQTT protocol benchmark tool written with Erlang. If you need testing services with large-scale scenarios and in-depth customization, the test service XMeterPerformance Test - 图2 (opens new window) is recommended.

Install eMQTT-Bench

There are three options for installing the eMQTT-Bench:

  • Run the docker image
  • Download and install the binary package
  • Build from the source code

Docker Image

You can install the benchmark tool by running the emqtt_bench docker image pushed to hub.docker.comPerformance Test - 图3 (opens new window). The :latest tag is updated with each new version:

  1. docker run -it emqx/emqtt-bench:latest
  2. Usage: emqtt_bench pub | sub | conn [--help]

Note that docker image name is using hyphen ‘-‘, while binary script name is with underscore ‘_‘.

Binary Package

You can download the released binary packages and install the emqtt_bench on the following platforms:

  • Amazon Linux 2
  • CentOS 7
  • Rocky Linux 8
  • Rocky Linux 9
  • Debian 9
  • Debain 10
  • Debain 11
  • Ubuntu 16.04
  • Ubuntu 18.04
  • Ubuntu 20.04
  • Ubuntu 22.04
  • MacOS 11
  • MacOS 12

For detailed information on each release, see ReleasesPerformance Test - 图4 (opens new window).

For example, here is how to install emqtt_bench on Ubuntu 20.04:

  1. mkdir emqtt_bench && cd emqtt_bench
  2. wget https://github.com/emqx/emqtt-bench/releases/download/0.4.11/emqtt-bench-0.4.11-ubuntu20.04-amd64.tar.gz
  3. tar xfz emqtt-bench-0.4.11-ubuntu20.04-amd64.tar.gz
  4. rm emqtt-bench-0.4.11-ubuntu20.04-amd64.tar.gz
  6. ./emqtt_bench
  7. Usage: emqtt_bench pub | sub | conn [--help]

Build from Source

emqtt_bench is written in Erlang and requires Erlang/OTPPerformance Test - 图5 (opens new window) 21.2 and above version to build it. The installation process of Erlang/OTP is skipped. For details, please refer to the online installation tutorials.

After the Erlang environment is installed, download the latest code of emqtt-bench and compile it:

  1. git clone https://github.com/emqx/emqtt-bench
  2. cd emqtt-bench
  4. make

After the compilation, an executable script named emqtt_bench will be generated in the current directory. Execute the following command to confirm that it can be used normally:

  1. ./emqtt_bench
  2. Usage: emqtt_bench pub | sub | conn [--help]

The output of the above content proves that emqtt_bench has been correctly installed on the host.

Test Performance Using eMQTT-Bench

There are three subcommands of emqtt_bench:

  1. pub: Create a large number of clients to perform the operation of publishing messages.
  2. sub: Create a large number of clients to subscribe to topics and receive messages.
  3. conn: Create a large number of connections.

Publish

When executing ./emqtt_bench pub --help, you get the available parameter output.

ParameterAbbreviationOptional ValueDefault ValueDescription
—host-h-localhostAddress of the MQTT server to connect
—port-p-1883MQTT service port
—version-V3
4
5		5MQTT protocol version used
—count-c-200Total number of clients
—startnumber-n-0Start number of clients
—interval-i-10Interval to create a client; unit: ms
—interval_of_msg-I-1000Interval to publish a message
—username-u-None; optionalClient username
—password-P-None; optionalClient password
—topic-t-None; requiredPublished topics; support placeholders:
%c: ClientId
%u: Username
%i: Client’s sequence number
—size-s-256Message Payload size; unit: bytes
—qos-q-0QoS level
—retain-rtrue
false		falseWhether the message sets the Retain flag
—keepalive-k-300Client keepalive time
—clean-Ctrue
false		trueWhether to establish a connection by cleaning the session
—ssl-Strue
false		falseWhether to enable SSL
—certfile--NoneClient SSL certificate
—keyfile--NoneClient SSL key file
—ws-true
false		falseWhether to establish a connection via WebSocket
—ifaddr--NoneSpecifies the local network card used by the client connection

For example, we start 10 connections and send 100 Qos0 messages to the topic t every second, where the size of each message payload is 16 bytes:

  1. ./emqtt_bench pub -t t -h emqx-server -s 16 -q 0 -c 10 -I 10

Subscribe

Execute ./emqtt_bench sub --help to get all available parameters of this subcommand. Their explanations have been included in the table above and are omitted here.

For example, we start 500 connections, and each subscribes to the t topic with Qos0:

  1. ./emqtt_bench sub -t t -h emqx-server -c 500

Connect

Execute ./emqtt_bench conn --help to get all available parameters of this subcommand. Their explanations have been included in the table above and are omitted here.

For example, we start 1000 connections:

  1. ./emqtt_bench conn -h emqx-server -c 1000

SSL Connection

emqtt_bench supports establishing a secure SSL connection and performing tests.

One-way certificate:

  1. ./emqtt_bench sub -c 100 -i 10 -t bench/%i -p 8883 -S
  2. ./emqtt_bench pub -c 100 -I 10 -t bench/%i -p 8883 -s 256 -S

Two-way certificate:

  1. ./emqtt_bench sub -c 100 -i 10 -t bench/%i -p 8883 --certfile path/to/client-cert.pem --keyfile path/to/client-key.pem
  2. ./emqtt_bench pub -c 100 -i 10 -t bench/%i -s 256 -p 8883 --certfile path/to/client-cert.pem --keyfile path/to/client-key.pem

Typical Stress Test Scenario

Scenario Description

Verify the use of the tool in 2 most typical scenarios:

  1. Connections: Use emqtt_bench to create millions of connections to EMQX.
  2. Throughput: Use emqtt_bench to create 10W / s Qos0 message throughput in EMQX.

Device and Deployment Topology

A total of three 8C16G servers need to be prepared, one for EMQX and two for client presses:

  • System: CentOS Linux release 7.7.1908 (Core)

  • CPU: Intel Xeon Processor (Skylake) Main frequency: 2693.670 MHZ

  • Server: emqx-centos7-v4.0.2.zip

  • Press: emqtt-bench v0.3.1

    • Each press is configured with 10 network cards, which are used to establish a large number of MQTT client connections in the connection test

The topology structure is as follows:

benchmark_topology

Tuning

Both the client’s press and the server’s machine need to perform system parameter tuning, refer to Tuning guide.

Connection Test

After performing system tuning, start the server:

  1. ./bin/emqx start

Then start 50 thousand connections on each network card on bench1, which is a total of 50w connections:

  2. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.100
  3. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.101
  4. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.102
  5. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.103
  6. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.104
  7. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.105
  8. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.106
  9. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.107
  10. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.108
  11. ./emqtt_bench -h 192.168.0.99 -c 50000 --ifaddr 192.168.0.109

Perform the same operation on bench2.

After all connections are established, execute ./bin/emqx_ctl listeners and find the following content to view the information about the number of connections in EMQX:

  1. listener on mqtt:tcp:0.0.0.0:1883
  2.   acceptors       : 8
  3.   max_conns       : 1024000
  4.   current_conn    : 1000000
  5.   shutdown_count  : []

Throughput Test

Similarly, first start the server:

  1. ./bin/emqx start

Start 500 subscription clients in bench1:

  1. ./emqtt_bench sub -t t -h 192.168.0.99 -c 500

Then start 20 publishers on bench2 and publish 10 messages per second:

  1. ./emqtt_bench pub -t t -h 192.168.0.99 -c 20 -I 100

Then, go back to the subscribing client on bench1, you can see the current rate of receiving messages:

  1. recv(28006): total=2102563, rate=99725(msg/sec)