Alarm

EMQX offers a built-in monitoring and alarm functionality for monitoring the internal state changes, such as CPU occupancy, system and process memory occupancy, number of processes, rule engine resource status, and cluster partition and healing. EMQX triggers and records these changes when they exceed a threshold or deviate from expectations, and removes them from the list once they are restored.

This page introduces which alarm information EMQX provides, how you can obtain and check the detailed alarm information, and how to configure the alarm settings and thresholds in EMQX. The monitoring and alarm function keeps you notified of potential problems during operation. By configuring alarms and setting appropriate thresholds, you can make sure that EMQX remains secure, stable, and reliable.

Alarm List

The following table lists the alarms that can be triggered to indicate potential problems during system monitoring.

TIP

Depending on the severance and impacts on the system, alarms can have 3 levels:

• Error: Errors caused by user presets. The client can perceive the error and retry.

• Warning: Occasional errors, and need to be taken seriously if they occur frequently.

• Critical: Irreversible data loss between the client and server, causing communication and business interruption.

The levels are defined from development perspectives and are only for a recommendation. You can define your own alarm levels according to the business needs.

Get Alarms

EMQX provides you with various ways to get alarms and check detailed alarm information. One way is to view the alarms on EMQX Dashboard, where you can view a list of active or historical alarms. However, it is only a central place for easy access to an overview of alarms that have been triggered. Another way is to subscribe to system topics through MQTT to receive real-time notifications of alarms with detailed alarm information. Alarms can also be accessed from the log or via REST API.

View Alarms on Dashboard

On EMQX Dashboard, click Monitoring -> Alarms. Select the Active or History tab, and you can see the list of currently active alarms and historical alarms.

Get Alarms via System Topic

EMQX will publish an MQTT message to system topics $SYS/brokers/<Node>/alarms/activate or $SYS/brokers/<Node>/alarms/deactivate when an alarm is triggered or cleared. Users can subscribe to the topic to receive alarm notifications.

The payload in the alarm notification message is in JSON format and contains the following fields:

Taking the alarm of high system memory usage as an example, you will receive an alarm message like below:

One system multifunction will be repeatedly reported. That is, if one alarm on high CPU usage is activated, the system will not generate another alarm of the same type. The generated alarm will be automatically deactivated when the monitored metric returns to normal, or you can manually deactivate the alarm.

Get Alarms from Log

The activation and deactivation of alarms can be written to log (console or file). When failures occur during message transmission or event processing, detailed information can be logged, and the logging system can also be used to capture alerts through log analysis. The following example shows the detailed alarm information printed in the log: The log level is warning, and the msg field is alarm_is_activated and alarm_is_deactivated.

Get Alarms via REST API

You can query and manage alarms through the API. Click Alarms on the left navigation menu on the UI to execute this API request. For how to work with EMQX API, see REST API.

Alarm Configuration

Alarm configuration includes configuring alarm settings and thresholds. Alarm settings determine how the alarm message is displayed and stored, while alarm thresholds establish limits or values that trigger the alarm when potential problems are detected. The alarm configuration feature allows you to customize the alarm settings and thresholds to meet your business needs.

Configure Alarm Settings

The settings for alarms can only be configured by modifying the configuration items in emqx.conf file. The following table lists the configuration items available for alarm setting configuration.

Configure Alarm Thresholds via Dashboard

Alarm thresholds can be configured on EMQX Dashboard. There are two ways to launch the Monitoring page for configuring the alarm thresholds:

1. On the Alarms page, click the Setting button and you will be led to the Monitoring page.
2. From the left navigation menu, click Management -> Monitoring.

On the Monitoring page, click the Erlang tab, you can configure the following items for the system performance of the Erlang Virtual Machine:

• Process limit check interval: Specify the time interval for checking the periodic process limit. The default value is 30 seconds.

• Process high watermark: Specify the threshold value of processes that can simultaneously exist at the local node. When the percentage exceeds the specified number, an alarm is raised. The default value is 80 percent.

• Process low watermark: Specify the threshold value of processes that can simultaneously exist at the local node. When the percentage is lowered to the specified number, an alarm is cleared. The default value is 60 percent.

• Enable Long GC monitoring: Disabled by default. When enabled, a warning-level log long_gc is emitted and an MQTT message is published to the system topic $SYS/sysmon/long_gc when an Erlang process spends long time performing garbage collection.

• Enable Long Schedule monitoring: Enabled by default, which means when the Erlang VM detects a task scheduled for too long, a warning level log long_schedule is emitted. You can set the proper time scheduled for a task in the text box. The default value is 240 milliseconds.

• Enable Large Heap monitoring: Enabled by default, which means when an Erlang process consumed a large amount of memory for its heap space, a warning level log large_heap is emitted, and an MQTT message is published to the system topic $SYS/sysmon/large_heap. You can set the limit of space bytesize in the text box. The default value is 32 MB.

• Enable Busy Distribution Port monitoring: Enabled by default, which means when the Remote Procedure Call (RPC) connection used to communicate with other nodes in the cluster is overloaded, a warning level log busy_dis_port log is emitted, and an MQTT message is published to system topic $SYS/sysmon/busy_dist_port.

• Enable Busy Port monitory: Enabled by default, which means when a port is overloaded, a warning level log busy_port is emitted, and an MQTT message is published to the system topic $SYS/sysmon/busy_port.

After you complete the configurations, click Save Changes.

Click the Operating System tab, you can configure the following items for the system performance:

• The time interval of the periodic CPU check: Specify the time interval for checking the CPU usage. The default value is 60 seconds.

• CPU high watermark: Specify the threshold value of how much system CPU can be used. When the percentage exceeds the specified value, a corresponding alarm is raised. The default value is 80 percent.

• CPU low watermark: Specify the threshold value of how much system CPU can be used. When the percentage is lowered to the specified value, a corresponding alarm is released. The default value is 60 percent.

• Mem check interval: Enabled by default. You can specify the time interval for periodic memory checks. The default value is 60 seconds.

• SysMem high watermark: Specify the threshold for how much system memory can be allocated. When the percentage exceeds the specified value, a corresponding alarm is raised. The default value is 70%.

• ProcMem high watermark: Specify the threshold for how much system memory can be allocated by one Erlang process. When the percentage exceeds the specified value, a corresponding alarm is raised. The default value is 5%.

After you complete the configurations, click Save Changes.

Configure Alarm Thresholds via Configuration Items

You can also configure alarm thresholds by modifying the configuration items for alarm thresholds. The following configuration items are currently available to be modified in emqx.conf file: