告警

概览

IoTDB 告警功能预计支持两种模式：

  • 写入触发：用户写入原始数据到原始时间序列，每插入一条数据都会触发 Trigger 的判断逻辑， 若满足告警要求则发送告警到下游数据接收器， 数据接收器再转发告警到外部终端。这种模式：

    • 适合需要即时监控每一条数据的场景。
    • 由于触发器中的运算会影响数据写入性能，适合对原始数据写入性能不敏感的场景。

  • 持续查询：用户写入原始数据到原始时间序列， ContinousQuery 定时查询原始时间序列，将查询结果写入新的时间序列， 每一次写入触发 Trigger 的判断逻辑， 若满足告警要求则发送告警到下游数据接收器， 数据接收器再转发告警到外部终端。这种模式：

    • 适合需要定时查询数据在某一段时间内的情况的场景。
    • 适合需要将原始数据降采样并持久化的场景。
    • 由于定时查询几乎不影响原始时间序列的写入，适合对原始数据写入性能敏感的场景。

随着 Trigger 模块的引入，可以实现写入触发模式的告警。

部署 AlertManager

安装与运行

二进制文件

预编译好的二进制文件可在 这里告警机制 - 图1open in new window 下载。

运行方法：

  1. ./alertmanager --config.file=<your_file>
Docker 镜像

可在 Quay.io告警机制 - 图2open in new window 或 Docker Hub告警机制 - 图3open in new window 获得。

运行方法：

  1. docker run --name alertmanager -d -p 127.0.0.1:9093:9093 quay.io/prometheus/alertmanager

配置

如下是一个示例，可以覆盖到大部分配置规则，详细的配置规则参见 这里告警机制 - 图4open in new window。

示例：

  1. # alertmanager.yml
  3. global:
  4.   # The smarthost and SMTP sender used for mail notifications.
  5.   smtp_smarthost: 'localhost:25'
  6.   smtp_from: 'alertmanager@example.org'
  8. # The root route on which each incoming alert enters.
  9. route:
  10.   # The root route must not have any matchers as it is the entry point for
  11.   # all alerts. It needs to have a receiver configured so alerts that do not
  12.   # match any of the sub-routes are sent to someone.
  13.   receiver: 'team-X-mails'
  15.   # The labels by which incoming alerts are grouped together. For example,
  16.   # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  17.   # be batched into a single group.
  18.   #
  19.   # To aggregate by all possible labels use '...' as the sole label name.
  20.   # This effectively disables aggregation entirely, passing through all
  21.   # alerts as-is. This is unlikely to be what you want, unless you have
  22.   # a very low alert volume or your upstream notification system performs
  23.   # its own grouping. Example: group_by: [...]
  24.   group_by: ['alertname', 'cluster']
  26.   # When a new group of alerts is created by an incoming alert, wait at
  27.   # least 'group_wait' to send the initial notification.
  28.   # This way ensures that you get multiple alerts for the same group that start
  29.   # firing shortly after another are batched together on the first
  30.   # notification.
  31.   group_wait: 30s
  33.   # When the first notification was sent, wait 'group_interval' to send a batch
  34.   # of new alerts that started firing for that group.
  35.   group_interval: 5m
  37.   # If an alert has successfully been sent, wait 'repeat_interval' to
  38.   # resend them.
  39.   repeat_interval: 3h
  41.   # All the above attributes are inherited by all child routes and can
  42.   # overwritten on each.
  44.   # The child route trees.
  45.   routes:
  46.   # This routes performs a regular expression match on alert labels to
  47.   # catch alerts that are related to a list of services.
  48.   - match_re:
  49.       service: ^(foo1|foo2|baz)$
  50.     receiver: team-X-mails
  52.     # The service has a sub-route for critical alerts, any alerts
  53.     # that do not match, i.e. severity != critical, fall-back to the
  54.     # parent node and are sent to 'team-X-mails'
  55.     routes:
  56.     - match:
  57.         severity: critical
  58.       receiver: team-X-pager
  60.   - match:
  61.       service: files
  62.     receiver: team-Y-mails
  64.     routes:
  65.     - match:
  66.         severity: critical
  67.       receiver: team-Y-pager
  69.   # This route handles all alerts coming from a database service. If there's
  70.   # no team to handle it, it defaults to the DB team.
  71.   - match:
  72.       service: database
  74.     receiver: team-DB-pager
  75.     # Also group alerts by affected database.
  76.     group_by: [alertname, cluster, database]
  78.     routes:
  79.     - match:
  80.         owner: team-X
  81.       receiver: team-X-pager
  83.     - match:
  84.         owner: team-Y
  85.       receiver: team-Y-pager
  87. # Inhibition rules allow to mute a set of alerts given that another alert is
  88. # firing.
  89. # We use this to mute any warning-level notifications if the same alert is
  90. # already critical.
  91. inhibit_rules:
  92. - source_match:
  93.     severity: 'critical'
  94.   target_match:
  95.     severity: 'warning'
  96.   # Apply inhibition if the alertname is the same.
  97.   # CAUTION: 
  98.   #   If all label names listed in `equal` are missing 
  99.   #   from both the source and target alerts,
  100.   #   the inhibition rule will apply!
  101.   equal: ['alertname']
  103. receivers:
  104. - name: 'team-X-mails'
  105.   email_configs:
  106.   - to: 'team-X+alerts@example.org, team-Y+alerts@example.org'
  108. - name: 'team-X-pager'
  109.   email_configs:
  110.   - to: 'team-X+alerts-critical@example.org'
  111.   pagerduty_configs:
  112.   - routing_key: <team-X-key>
  114. - name: 'team-Y-mails'
  115.   email_configs:
  116.   - to: 'team-Y+alerts@example.org'
  118. - name: 'team-Y-pager'
  119.   pagerduty_configs:
  120.   - routing_key: <team-Y-key>
  122. - name: 'team-DB-pager'
  123.   pagerduty_configs:
  124.   - routing_key: <team-DB-key>

在后面的示例中，我们采用的配置如下：

  1. # alertmanager.yml
  3. global: 
  4.   smtp_smarthost: ''
  5.   smtp_from: '' 
  6.   smtp_auth_username: '' 
  7.   smtp_auth_password: '' 
  8.   smtp_require_tls: false
  10. route:
  11.   group_by: ['alertname'] 
  12.   group_wait: 1m
  13.   group_interval: 10m
  14.   repeat_interval: 10h 
  15.   receiver: 'email'
  17. receivers:
  18.   - name: 'email'
  19.     email_configs: 
  20.     - to: '' 
  22. inhibit_rules:
  23.   - source_match:
  24.       severity: 'critical'
  25.     target_match:
  26.       severity: 'warning'
  27.     equal: ['alertname']

API

AlertManager API 分为 v1 和 v2 两个版本，当前 AlertManager API 版本为 v2 （配置参见 api/v2/openapi.yaml告警机制 - 图5open in new window)。

默认配置的前缀为 /api/v1 或 /api/v2， 发送告警的 endpoint 为 /api/v1/alerts 或 /api/v2/alerts。 如果用户指定了 --web.route-prefix， 例如 --web.route-prefix=/alertmanager/， 那么前缀将会变为 /alertmanager/api/v1 或 /alertmanager/api/v2， 发送告警的 endpoint 变为 /alertmanager/api/v1/alerts 或 /alertmanager/api/v2/alerts。

创建 trigger

编写 trigger 类

用户通过自行创建 Java 类、编写钩子中的逻辑来定义一个触发器。 具体配置流程参见 Trigger。

下面的示例创建了 org.apache.iotdb.trigger.ClusterAlertingExample 类， 其 alertManagerHandler 成员变量可发送告警至地址为 http://127.0.0.1:9093/ 的 AlertManager 实例。

当 value > 100.0 时，发送 severity 为 critical 的告警； 当 50.0 < value <= 100.0 时，发送 severity 为 warning 的告警。

  1. package org.apache.iotdb.trigger;
  3. import org.apache.iotdb.db.engine.trigger.sink.alertmanager.AlertManagerConfiguration;
  4. import org.apache.iotdb.db.engine.trigger.sink.alertmanager.AlertManagerEvent;
  5. import org.apache.iotdb.db.engine.trigger.sink.alertmanager.AlertManagerHandler;
  6. import org.apache.iotdb.trigger.api.Trigger;
  7. import org.apache.iotdb.trigger.api.TriggerAttributes;
  8. import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
  9. import org.apache.iotdb.tsfile.write.record.Tablet;
  10. import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
  12. import org.slf4j.Logger;
  13. import org.slf4j.LoggerFactory;
  15. import java.io.IOException;
  16. import java.util.HashMap;
  17. import java.util.List;
  19. public class ClusterAlertingExample implements Trigger {
  20.   private static final Logger LOGGER = LoggerFactory.getLogger(ClusterAlertingExample.class);
  22.   private final AlertManagerHandler alertManagerHandler = new AlertManagerHandler();
  24.   private final AlertManagerConfiguration alertManagerConfiguration =
  25.       new AlertManagerConfiguration("http://127.0.0.1:9093/api/v2/alerts");
  27.   private String alertname;
  29.   private final HashMap<String, String> labels = new HashMap<>();
  31.   private final HashMap<String, String> annotations = new HashMap<>();
  33.   @Override
  34.   public void onCreate(TriggerAttributes attributes) throws Exception {
  35.     alertname = "alert_test";
  37.     labels.put("series", "root.ln.wf01.wt01.temperature");
  38.     labels.put("value", "");
  39.     labels.put("severity", "");
  41.     annotations.put("summary", "high temperature");
  42.     annotations.put("description", "{{.alertname}}: {{.series}} is {{.value}}");
  44.     alertManagerHandler.open(alertManagerConfiguration);
  45.   }
  47.   @Override
  48.   public void onDrop() throws IOException {
  49.     alertManagerHandler.close();
  50.   }
  52.   @Override
  53.   public boolean fire(Tablet tablet) throws Exception {
  54.     List<MeasurementSchema> measurementSchemaList = tablet.getSchemas();
  55.     for (int i = 0, n = measurementSchemaList.size(); i < n; i++) {
  56.       if (measurementSchemaList.get(i).getType().equals(TSDataType.DOUBLE)) {
  57.         // for example, we only deal with the columns of Double type
  58.         double[] values = (double[]) tablet.values[i];
  59.         for (double value : values) {
  60.           if (value > 100.0) {
  61.             LOGGER.info("trigger value > 100");
  62.             labels.put("value", String.valueOf(value));
  63.             labels.put("severity", "critical");
  64.             AlertManagerEvent alertManagerEvent =
  65.                 new AlertManagerEvent(alertname, labels, annotations);
  66.             alertManagerHandler.onEvent(alertManagerEvent);
  67.           } else if (value > 50.0) {
  68.             LOGGER.info("trigger value > 50");
  69.             labels.put("value", String.valueOf(value));
  70.             labels.put("severity", "warning");
  71.             AlertManagerEvent alertManagerEvent =
  72.                 new AlertManagerEvent(alertname, labels, annotations);
  73.             alertManagerHandler.onEvent(alertManagerEvent);
  74.           }
  75.         }
  76.       }
  77.     }
  78.     return true;
  79.   }
  80. }

创建 trigger

如下的 sql 语句在 root.ln.wf01.wt01.temperature 时间序列上注册了名为 root-ln-wf01-wt01-alert、 运行逻辑由 org.apache.iotdb.trigger.ClusterAlertingExample 类定义的触发器。

  1.   CREATE STATELESS TRIGGER `root-ln-wf01-wt01-alert`
  2.   AFTER INSERT
  3.   ON root.ln.wf01.wt01.temperature
  4.   AS "org.apache.iotdb.trigger.ClusterAlertingExample"
  5.   USING URI 'http://jar/ClusterAlertingExample.jar'

写入数据

当我们完成 AlertManager 的部署和启动、Trigger 的创建， 可以通过向时间序列写入数据来测试告警功能。

  1. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (1, 0);
  2. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (2, 30);
  3. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (3, 60);
  4. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (4, 90);
  5. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (5, 120);

执行完上述写入语句后，可以收到告警邮件。由于我们的 AlertManager 配置中设定 severity 为 critical 的告警 会抑制 severity 为 warning 的告警，我们收到的告警邮件中只包含写入 (5, 120) 后触发的告警。

alerting