Alerting

Overview

The alerting of IoTDB is expected to support two modes:

  • Writing triggered: the user writes data to the original time series, and every time a piece of data is inserted, the judgment logic of trigger will be triggered. If the alerting requirements are met, an alert is sent to the data sink, The data sink then forwards the alert to the external terminal.

    • This mode is suitable for scenarios that need to monitor every piece of data in real time.
    • Since the operation in the trigger will affect the data writing performance, it is suitable for scenarios that are not sensitive to the original data writing performance.

  • Continuous query: the user writes data to the original time series, ContinousQuery periodically queries the original time series, and writes the query results into the new time series, Each write triggers the judgment logic of trigger, If the alerting requirements are met, an alert is sent to the data sink, The data sink then forwards the alert to the external terminal.

    • This mode is suitable for scenarios where data needs to be regularly queried within a certain period of time.
    • It is Suitable for scenarios where the original data needs to be down-sampled and persisted.
    • Since the timing query hardly affects the writing of the original time series, it is suitable for scenarios that are sensitive to the performance of the original data writing performance.

With the introduction of the Trigger into IoTDB, at present, users can use these two modules with AlertManager to realize the writing triggered alerting mode.

Deploying AlertManager

Installation

Precompiled binaries

The pre-compiled binary file can be downloaded hereAlerting - 图1open in new window.

Running command:

  1. ./alertmanager --config.file=<your_file>

Docker image

Available at Quay.ioAlerting - 图2open in new window or Docker HubAlerting - 图3open in new window.

Running command:

  1. docker run --name alertmanager -d -p 127.0.0.1:9093:9093 quay.io/prometheus/alertmanager

Configuration

The following is an example, which can cover most of the configuration rules. For detailed configuration rules, see hereAlerting - 图4open in new window.

Example:

  1. # alertmanager.yml
  3. global:
  4.   # The smarthost and SMTP sender used for mail notifications.
  5.   smtp_smarthost: 'localhost:25'
  6.   smtp_from: 'alertmanager@example.org'
  8. # The root route on which each incoming alert enters.
  9. route:
  10.   # The root route must not have any matchers as it is the entry point for
  11.   # all alerts. It needs to have a receiver configured so alerts that do not
  12.   # match any of the sub-routes are sent to someone.
  13.   receiver: 'team-X-mails'
  15.   # The labels by which incoming alerts are grouped together. For example,
  16.   # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  17.   # be batched into a single group.
  18.   #
  19.   # To aggregate by all possible labels use '...' as the sole label name.
  20.   # This effectively disables aggregation entirely, passing through all
  21.   # alerts as-is. This is unlikely to be what you want, unless you have
  22.   # a very low alert volume or your upstream notification system performs
  23.   # its own grouping. Example: group_by: [...]
  24.   group_by: ['alertname', 'cluster']
  26.   # When a new group of alerts is created by an incoming alert, wait at
  27.   # least 'group_wait' to send the initial notification.
  28.   # This way ensures that you get multiple alerts for the same group that start
  29.   # firing shortly after another are batched together on the first
  30.   # notification.
  31.   group_wait: 30s
  33.   # When the first notification was sent, wait 'group_interval' to send a batch
  34.   # of new alerts that started firing for that group.
  35.   group_interval: 5m
  37.   # If an alert has successfully been sent, wait 'repeat_interval' to
  38.   # resend them.
  39.   repeat_interval: 3h
  41.   # All the above attributes are inherited by all child routes and can
  42.   # overwritten on each.
  44.   # The child route trees.
  45.   routes:
  46.   # This routes performs a regular expression match on alert labels to
  47.   # catch alerts that are related to a list of services.
  48.   - match_re:
  49.       service: ^(foo1|foo2|baz)$
  50.     receiver: team-X-mails
  52.     # The service has a sub-route for critical alerts, any alerts
  53.     # that do not match, i.e. severity != critical, fall-back to the
  54.     # parent node and are sent to 'team-X-mails'
  55.     routes:
  56.     - match:
  57.         severity: critical
  58.       receiver: team-X-pager
  60.   - match:
  61.       service: files
  62.     receiver: team-Y-mails
  64.     routes:
  65.     - match:
  66.         severity: critical
  67.       receiver: team-Y-pager
  69.   # This route handles all alerts coming from a database service. If there's
  70.   # no team to handle it, it defaults to the DB team.
  71.   - match:
  72.       service: database
  74.     receiver: team-DB-pager
  75.     # Also group alerts by affected database.
  76.     group_by: [alertname, cluster, database]
  78.     routes:
  79.     - match:
  80.         owner: team-X
  81.       receiver: team-X-pager
  83.     - match:
  84.         owner: team-Y
  85.       receiver: team-Y-pager
  88. # Inhibition rules allow to mute a set of alerts given that another alert is
  89. # firing.
  90. # We use this to mute any warning-level notifications if the same alert is
  91. # already critical.
  92. inhibit_rules:
  93. - source_match:
  94.     severity: 'critical'
  95.   target_match:
  96.     severity: 'warning'
  97.   # Apply inhibition if the alertname is the same.
  98.   # CAUTION: 
  99.   #   If all label names listed in `equal` are missing 
  100.   #   from both the source and target alerts,
  101.   #   the inhibition rule will apply!
  102.   equal: ['alertname']
  105. receivers:
  106. - name: 'team-X-mails'
  107.   email_configs:
  108.   - to: 'team-X+alerts@example.org, team-Y+alerts@example.org'
  110. - name: 'team-X-pager'
  111.   email_configs:
  112.   - to: 'team-X+alerts-critical@example.org'
  113.   pagerduty_configs:
  114.   - routing_key: <team-X-key>
  116. - name: 'team-Y-mails'
  117.   email_configs:
  118.   - to: 'team-Y+alerts@example.org'
  120. - name: 'team-Y-pager'
  121.   pagerduty_configs:
  122.   - routing_key: <team-Y-key>
  124. - name: 'team-DB-pager'
  125.   pagerduty_configs:
  126.   - routing_key: <team-DB-key>

In the following example, we used the following configuration:

  1. # alertmanager.yml
  3. global: 
  4.   smtp_smarthost: ''
  5.   smtp_from: '' 
  6.   smtp_auth_username: '' 
  7.   smtp_auth_password: '' 
  8.   smtp_require_tls: false
  10. route:
  11.   group_by: ['alertname'] 
  12.   group_wait: 1m
  13.   group_interval: 10m
  14.   repeat_interval: 10h 
  15.   receiver: 'email'
  17. receivers:
  18.   - name: 'email'
  19.     email_configs: 
  20.     - to: '' 
  22. inhibit_rules:
  23.   - source_match:
  24.       severity: 'critical'
  25.     target_match:
  26.       severity: 'warning'
  27.     equal: ['alertname']

API

The AlertManager API is divided into two versions, v1 and v2. The current AlertManager API version is v2 (For configuration see api/v2/openapi.yamlAlerting - 图5open in new window).

By default, the prefix is /api/v1 or /api/v2 and the endpoint for sending alerts is /api/v1/alerts or /api/v2/alerts. If the user specifies --web.route-prefix, for example --web.route-prefix=/alertmanager/, then the prefix will become /alertmanager/api/v1 or /alertmanager/api/v2, and the endpoint that sends the alert becomes /alertmanager/api/v1/alerts or /alertmanager/api/v2/alerts.

Creating trigger

Writing the trigger class

The user defines a trigger by creating a Java class and writing the logic in the hook. Please refer to Triggers for the specific configuration process.

The following example creates the org.apache.iotdb.trigger.ClusterAlertingExample class, Its alertManagerHandler member variables can send alerts to the AlertManager instance at the address of http://127.0.0.1:9093/.

When value> 100.0, send an alert of critical severity; when 50.0 <value <= 100.0, send an alert of warning severity .

  1. /*
  2.  * Licensed to the Apache Software Foundation (ASF) under one
  3.  * or more contributor license agreements.  See the NOTICE file
  4.  * distributed with this work for additional information
  5.  * regarding copyright ownership.  The ASF licenses this file
  6.  * to you under the Apache License, Version 2.0 (the
  7.  * "License"); you may not use this file except in compliance
  8.  * with the License.  You may obtain a copy of the License at
  9.  *
  10.  *     http://www.apache.org/licenses/LICENSE-2.0
  11.  *
  12.  * Unless required by applicable law or agreed to in writing,
  13.  * software distributed under the License is distributed on an
  14.  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  15.  * KIND, either express or implied.  See the License for the
  16.  * specific language governing permissions and limitations
  17.  * under the License.
  18.  */
  20. package org.apache.iotdb.trigger;
  22. import org.apache.iotdb.db.engine.trigger.sink.alertmanager.AlertManagerConfiguration;
  23. import org.apache.iotdb.db.engine.trigger.sink.alertmanager.AlertManagerEvent;
  24. import org.apache.iotdb.db.engine.trigger.sink.alertmanager.AlertManagerHandler;
  25. import org.apache.iotdb.trigger.api.Trigger;
  26. import org.apache.iotdb.trigger.api.TriggerAttributes;
  27. import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
  28. import org.apache.iotdb.tsfile.write.record.Tablet;
  29. import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
  31. import org.slf4j.Logger;
  32. import org.slf4j.LoggerFactory;
  34. import java.io.IOException;
  35. import java.util.HashMap;
  36. import java.util.List;
  38. public class ClusterAlertingExample implements Trigger {
  39.   private static final Logger LOGGER = LoggerFactory.getLogger(ClusterAlertingExample.class);
  41.   private final AlertManagerHandler alertManagerHandler = new AlertManagerHandler();
  43.   private final AlertManagerConfiguration alertManagerConfiguration =
  44.       new AlertManagerConfiguration("http://127.0.0.1:9093/api/v2/alerts");
  46.   private String alertname;
  48.   private final HashMap<String, String> labels = new HashMap<>();
  50.   private final HashMap<String, String> annotations = new HashMap<>();
  52.   @Override
  53.   public void onCreate(TriggerAttributes attributes) throws Exception {
  54.     alertname = "alert_test";
  56.     labels.put("series", "root.ln.wf01.wt01.temperature");
  57.     labels.put("value", "");
  58.     labels.put("severity", "");
  60.     annotations.put("summary", "high temperature");
  61.     annotations.put("description", "{{.alertname}}: {{.series}} is {{.value}}");
  63.     alertManagerHandler.open(alertManagerConfiguration);
  64.   }
  66.   @Override
  67.   public void onDrop() throws IOException {
  68.     alertManagerHandler.close();
  69.   }
  71.   @Override
  72.   public boolean fire(Tablet tablet) throws Exception {
  73.     List<MeasurementSchema> measurementSchemaList = tablet.getSchemas();
  74.     for (int i = 0, n = measurementSchemaList.size(); i < n; i++) {
  75.       if (measurementSchemaList.get(i).getType().equals(TSDataType.DOUBLE)) {
  76.         // for example, we only deal with the columns of Double type
  77.         double[] values = (double[]) tablet.values[i];
  78.         for (double value : values) {
  79.           if (value > 100.0) {
  80.             LOGGER.info("trigger value > 100");
  81.             labels.put("value", String.valueOf(value));
  82.             labels.put("severity", "critical");
  83.             AlertManagerEvent alertManagerEvent =
  84.                 new AlertManagerEvent(alertname, labels, annotations);
  85.             alertManagerHandler.onEvent(alertManagerEvent);
  86.           } else if (value > 50.0) {
  87.             LOGGER.info("trigger value > 50");
  88.             labels.put("value", String.valueOf(value));
  89.             labels.put("severity", "warning");
  90.             AlertManagerEvent alertManagerEvent =
  91.                 new AlertManagerEvent(alertname, labels, annotations);
  92.             alertManagerHandler.onEvent(alertManagerEvent);
  93.           }
  94.         }
  95.       }
  96.     }
  97.     return true;
  98.   }
  99. }

Creating trigger

The following SQL statement registered the trigger named root-ln-wf01-wt01-alert on the root.ln.wf01.wt01.temperature time series, whose operation logic is defined by org.apache.iotdb.trigger.ClusterAlertingExample java class.

  1.   CREATE STATELESS TRIGGER `root-ln-wf01-wt01-alert`
  2.   AFTER INSERT
  3.   ON root.ln.wf01.wt01.temperature
  4.   AS "org.apache.iotdb.trigger.AlertingExample"
  5.   USING URI 'http://jar/ClusterAlertingExample.jar'

Writing data

When we finish the deployment and startup of AlertManager as well as the creation of Trigger, we can test the alerting by writing data to the time series.

  1. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (1, 0);
  2. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (2, 30);
  3. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (3, 60);
  4. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (4, 90);
  5. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (5, 120);

After executing the above writing statements, we can receive an alerting email. Because our AlertManager configuration above makes alerts of critical severity inhibit those of warning severity, the alerting email we receive only contains the alert triggered by the writing of (5, 120).

alerting