Configuration

Alertmanager is configured via command-line flags and a configuration file. While the command-line flags configure immutable system parameters, the configuration file defines inhibition rules, notification routing and notification receivers.

The visual editor can assist in building routing trees.

To view all available command-line flags, run alertmanager -h.

Alertmanager can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied and an error is logged. A configuration reload is triggered by sending a SIGHUP to the process or sending a HTTP POST request to the /-/reload endpoint.

Configuration file

To specify which configuration file to load, use the --config.file flag.

  1. ./alertmanager --config.file=alertmanager.yml

The file is written in the YAML format, defined by the scheme described below. Brackets indicate that a parameter is optional. For non-list parameters the value is set to the specified default.

Generic placeholders are defined as follows:

  • <duration>: a duration matching the regular expression ((([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?|0), e.g. 1d, 1h30m, 5m, 10s
  • <labelname>: a string matching the regular expression [a-zA-Z_][a-zA-Z0-9_]*
  • <labelvalue>: a string of unicode characters
  • <filepath>: a valid path in the current working directory
  • <boolean>: a boolean that can take the values true or false
  • <string>: a regular string
  • <secret>: a regular string that is a secret, such as a password
  • <tmpl_string>: a string which is template-expanded before usage
  • <tmpl_secret>: a string which is template-expanded before usage that is a secret
  • <int>: an integer value

The other placeholders are specified separately.

A provided valid example file shows usage in context.

The global configuration specifies parameters that are valid in all other configuration contexts. They also serve as defaults for other configuration sections.

  1. global:
  2.   # The default SMTP From header field.
  3.   [ smtp_from: <tmpl_string> ]
  4.   # The default SMTP smarthost used for sending emails, including port number.
  5.   # Port number usually is 25, or 587 for SMTP over TLS (sometimes referred to as STARTTLS).
  6.   # Example: smtp.example.org:587
  7.   [ smtp_smarthost: <string> ]
  8.   # The default hostname to identify to the SMTP server.
  9.   [ smtp_hello: <string> | default = "localhost" ]
  10.   # SMTP Auth using CRAM-MD5, LOGIN and PLAIN. If empty, Alertmanager doesn't authenticate to the SMTP server.
  11.   [ smtp_auth_username: <string> ]
  12.   # SMTP Auth using LOGIN and PLAIN.
  13.   [ smtp_auth_password: <secret> ]
  14.   # SMTP Auth using PLAIN.
  15.   [ smtp_auth_identity: <string> ]
  16.   # SMTP Auth using CRAM-MD5.
  17.   [ smtp_auth_secret: <secret> ]
  18.   # The default SMTP TLS requirement.
  19.   # Note that Go does not support unencrypted connections to remote SMTP endpoints.
  20.   [ smtp_require_tls: <bool> | default = true ]
  22.   # The API URL to use for Slack notifications.
  23.   [ slack_api_url: <secret> ]
  24.   [ slack_api_url_file: <filepath> ]
  25.   [ victorops_api_key: <secret> ]
  26.   [ victorops_api_url: <string> | default = "https://alert.victorops.com/integrations/generic/20131114/alert/" ]
  27.   [ pagerduty_url: <string> | default = "https://events.pagerduty.com/v2/enqueue" ]
  28.   [ opsgenie_api_key: <secret> ]
  29.   [ opsgenie_api_key_file: <filepath> ]
  30.   [ opsgenie_api_url: <string> | default = "https://api.opsgenie.com/" ]
  31.   [ wechat_api_url: <string> | default = "https://qyapi.weixin.qq.com/cgi-bin/" ]
  32.   [ wechat_api_secret: <secret> ]
  33.   [ wechat_api_corp_id: <string> ]
  34.   [ telegram_api_url: <string> | default = "https://api.telegram.org" ]
  35.   # The default HTTP client configuration
  36.   [ http_config: <http_config> ]
  38.   # ResolveTimeout is the default value used by alertmanager if the alert does
  39.   # not include EndsAt, after this time passes it can declare the alert as resolved if it has not been updated.
  40.   # This has no impact on alerts from Prometheus, as they always include EndsAt.
  41.   [ resolve_timeout: <duration> | default = 5m ]
  43. # Files from which custom notification template definitions are read.
  44. # The last component may use a wildcard matcher, e.g. 'templates/*.tmpl'.
  45. templates:
  46.   [ - <filepath> ... ]
  48. # The root node of the routing tree.
  49. route: <route>
  51. # A list of notification receivers.
  52. receivers:
  53.   - <receiver> ...
  55. # A list of inhibition rules.
  56. inhibit_rules:
  57.   [ - <inhibit_rule> ... ]
  59. # DEPRECATED: use time_intervals below.
  60. # A list of mute time intervals for muting routes.
  61. mute_time_intervals:
  62.   [ - <mute_time_interval> ... ]
  64. # A list of time intervals for muting/activating routes.
  65. time_intervals:
  66.   [ - <time_interval> ... ]

<route>

A route block defines a node in a routing tree and its children. Its optional configuration parameters are inherited from its parent node if not set.

Every alert enters the routing tree at the configured top-level route, which must match all alerts (i.e. not have any configured matchers). It then traverses the child nodes. If continue is set to false, it stops after the first matching child. If continue is true on a matching node, the alert will continue matching against subsequent siblings. If an alert does not match any children of a node (no matching child nodes, or none exist), the alert is handled based on the configuration parameters of the current node.

  1. [ receiver: <string> ]
  2. # The labels by which incoming alerts are grouped together. For example,
  3. # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  4. # be batched into a single group.
  5. #
  6. # To aggregate by all possible labels use the special value '...' as the sole label name, for example:
  7. # group_by: ['...']
  8. # This effectively disables aggregation entirely, passing through all
  9. # alerts as-is. This is unlikely to be what you want, unless you have
  10. # a very low alert volume or your upstream notification system performs
  11. # its own grouping.
  12. [ group_by: '[' <labelname>, ... ']' ]
  14. # Whether an alert should continue matching subsequent sibling nodes.
  15. [ continue: <boolean> | default = false ]
  17. # DEPRECATED: Use matchers below.
  18. # A set of equality matchers an alert has to fulfill to match the node.
  19. match:
  20.   [ <labelname>: <labelvalue>, ... ]
  22. # DEPRECATED: Use matchers below.
  23. # A set of regex-matchers an alert has to fulfill to match the node.
  24. match_re:
  25.   [ <labelname>: <regex>, ... ]
  27. # A list of matchers that an alert has to fulfill to match the node.
  28. matchers:
  29.   [ - <matcher> ... ]
  31. # How long to initially wait to send a notification for a group
  32. # of alerts. Allows to wait for an inhibiting alert to arrive or collect
  33. # more initial alerts for the same group. (Usually ~0s to few minutes.)
  34. [ group_wait: <duration> | default = 30s ]
  36. # How long to wait before sending a notification about new alerts that
  37. # are added to a group of alerts for which an initial notification has
  38. # already been sent. (Usually ~5m or more.)
  39. [ group_interval: <duration> | default = 5m ]
  41. # How long to wait before sending a notification again if it has already
  42. # been sent successfully for an alert. (Usually ~3h or more).
  43. [ repeat_interval: <duration> | default = 4h ]
  45. # Times when the route should be muted. These must match the name of a
  46. # mute time interval defined in the mute_time_intervals section.
  47. # Additionally, the root node cannot have any mute times.
  48. # When a route is muted it will not send any notifications, but
  49. # otherwise acts normally (including ending the route-matching process
  50. # if the `continue` option is not set.)
  51. mute_time_intervals:
  52.   [ - <string> ...]
  54. # Times when the route should be active. These must match the name of a
  55. # time interval defined in the time_intervals section. An empty value
  56. # means that the route is always active.
  57. # Additionally, the root node cannot have any active times.
  58. # The route will send notifications only when active, but otherwise
  59. # acts normally (including ending the route-matching process
  60. # if the `continue` option is not set).
  61. active_time_intervals:
  62.   [ - <string> ...]
  64. # Zero or more child routes.
  65. routes:
  66.   [ - <route> ... ]

Example

  1. # The root route with all parameters, which are inherited by the child
  2. # routes if they are not overwritten.
  3. route:
  4.   receiver: 'default-receiver'
  5.   group_wait: 30s
  6.   group_interval: 5m
  7.   repeat_interval: 4h
  8.   group_by: [cluster, alertname]
  9.   # All alerts that do not match the following child routes
  10.   # will remain at the root node and be dispatched to 'default-receiver'.
  11.   routes:
  12.   # All alerts with service=mysql or service=cassandra
  13.   # are dispatched to the database pager.
  14.   - receiver: 'database-pager'
  15.     group_wait: 10s
  16.     matchers:
  17.     - service=~"mysql|cassandra"
  18.   # All alerts with the team=frontend label match this sub-route.
  19.   # They are grouped by product and environment rather than cluster
  20.   # and alertname.
  21.   - receiver: 'frontend-pager'
  22.     group_by: [product, environment]
  23.     matchers:
  24.     - team="frontend"
  26.   # All alerts with the service=inhouse-service label match this sub-route.
  27.   # the route will be muted during offhours and holidays time intervals.
  28.   # even if it matches, it will continue to the next sub-route
  29.   - receiver: 'dev-pager'
  30.     matchers:
  31.       - service="inhouse-service"
  32.     mute_time_intervals:
  33.       - offhours
  34.       - holidays
  35.     continue: true
  37.     # All alerts with the service=inhouse-service label match this sub-route
  38.     # the route will be active only during offhours and holidays time intervals.
  39.   - receiver: 'on-call-pager'
  40.     matchers:
  41.       - service="inhouse-service"
  42.     active_time_intervals:
  43.       - offhours
  44.       - holidays

<time_interval>

A time_interval specifies a named interval of time that may be referenced in the routing tree to mute/activate particular routes for particular times of the day.

  1. name: <string>
  2. time_intervals:
  3.   [ - <time_interval> ... ]

<time_interval>

A time_interval contains the actual definition for an interval of time. The syntax supports the following fields:

  1. - times:
  2.   [ - <time_range> ...]
  3.   weekdays:
  4.   [ - <weekday_range> ...]
  5.   days_of_month:
  6.   [ - <days_of_month_range> ...]
  7.   months:
  8.   [ - <month_range> ...]
  9.   years:
  10.   [ - <year_range> ...]

All fields are lists. Within each non-empty list, at least one element must be satisfied to match the field. If a field is left unspecified, any value will match the field. For an instant of time to match a complete time interval, all fields must match. Some fields support ranges and negative indices, and are detailed below. All definitions are taken to be in UTC, no other timezones are currently supported.

time_range Ranges inclusive of the starting time and exclusive of the end time to make it easy to represent times that start/end on hour boundaries. For example, starttime: ‘17:00’ and endtime: ‘24:00’ will begin at 17:00 and finish immediately before 24:00. They are specified like so:

  1.     times:
  2.     - start_time: HH:MM
  3.       end_time: HH:MM

weekday_range: A list of days of the week, where the week begins on Sunday and ends on Saturday. Days should be specified by name (e.g. ‘Sunday’). For convenience, ranges are also accepted of the form : and are inclusive on both ends. For example: ['monday:wednesday','saturday', 'sunday']

days_of_month_range: A list of numerical days in the month. Days begin at 1. Negative values are also accepted which begin at the end of the month, e.g. -1 during January would represent January 31. For example: ['1:5', '-3:-1']. Extending past the start or end of the month will cause it to be clamped. E.g. specifying ['1:31'] during February will clamp the actual end date to 28 or 29 depending on leap years. Inclusive on both ends.

month_range: A list of calendar months identified by a case-insensitive name (e.g. ‘January’) or by number, where January = 1. Ranges are also accepted. For example, ['1:3', 'may:august', 'december']. Inclusive on both ends.

year_range: A numerical list of years. Ranges are accepted. For example, ['2020:2022', '2030']. Inclusive on both ends.

<inhibit_rule>

An inhibition rule mutes an alert (target) matching a set of matchers when an alert (source) exists that matches another set of matchers. Both target and source alerts must have the same label values for the label names in the equal list.

Semantically, a missing label and a label with an empty value are the same thing. Therefore, if all the label names listed in equal are missing from both the source and target alerts, the inhibition rule will apply.

To prevent an alert from inhibiting itself, an alert that matches both the target and the source side of a rule cannot be inhibited by alerts for which the same is true (including itself). However, we recommend to choose target and source matchers in a way that alerts never match both sides. It is much easier to reason about and does not trigger this special case.

  1. # DEPRECATED: Use target_matchers below.
  2. # Matchers that have to be fulfilled in the alerts to be muted.
  3. target_match:
  4.   [ <labelname>: <labelvalue>, ... ]
  5. # DEPRECATED: Use target_matchers below.
  6. target_match_re:
  7.   [ <labelname>: <regex>, ... ]
  9. # A list of matchers that have to be fulfilled by the target
  10. # alerts to be muted.
  11. target_matchers:
  12.   [ - <matcher> ... ]
  14. # DEPRECATED: Use source_matchers below.
  15. # Matchers for which one or more alerts have to exist for the
  16. # inhibition to take effect.
  17. source_match:
  18.   [ <labelname>: <labelvalue>, ... ]
  19. # DEPRECATED: Use source_matchers below.
  20. source_match_re:
  21.   [ <labelname>: <regex>, ... ]
  23. # A list of matchers for which one or more alerts have
  24. # to exist for the inhibition to take effect.
  25. source_matchers:
  26.   [ - <matcher> ... ]
  28. # Labels that must have an equal value in the source and target
  29. # alert for the inhibition to take effect.
  30. [ equal: '[' <labelname>, ... ']' ]

<http_config>

A http_config allows configuring the HTTP client that the receiver uses to communicate with HTTP-based API services.

  1. # Note that `basic_auth` and `authorization` options are mutually exclusive.
  3. # Sets the `Authorization` header with the configured username and password.
  4. # password and password_file are mutually exclusive.
  5. basic_auth:
  6.   [ username: <string> ]
  7.   [ password: <secret> ]
  8.   [ password_file: <string> ]
  10. # Optional the `Authorization` header configuration.
  11. authorization:
  12.   # Sets the authentication type.
  13.   [ type: <string> | default: Bearer ]
  14.   # Sets the credentials. It is mutually exclusive with
  15.   # `credentials_file`.
  16.   [ credentials: <secret> ]
  17.   # Sets the credentials with the credentials read from the configured file.
  18.   # It is mutually exclusive with `credentials`.
  19.   [ credentials_file: <filename> ]
  21. # Optional OAuth 2.0 configuration.
  22. # Cannot be used at the same time as basic_auth or authorization.
  23. oauth2:
  24.   [ <oauth2> ]
  26. # Optional proxy URL.
  27. [ proxy_url: <string> ]
  29. # Configure whether HTTP requests follow HTTP 3xx redirects.
  30. [ follow_redirects: <bool> | default = true ]
  32. # Configures the TLS settings.
  33. tls_config:
  34.   [ <tls_config> ]

oauth2

OAuth 2.0 authentication using the client credentials grant type. Alertmanager fetches an access token from the specified endpoint with the given client access and secret keys.

  1. client_id: <string>
  2. [ client_secret: <secret> ]
  4. # Read the client secret from a file.
  5. # It is mutually exclusive with `client_secret`.
  6. [ client_secret_file: <filename> ]
  8. # Scopes for the token request.
  9. scopes:
  10.   [ - <string> ... ]
  12. # The URL to fetch the token from.
  13. token_url: <string>
  15. # Optional parameters to append to the token URL.
  16. endpoint_params:
  17.   [ <string>: <string> ... ]

<tls_config>

A tls_config allows configuring TLS connections.

  1. # CA certificate to validate the server certificate with.
  2. [ ca_file: <filepath> ]
  4. # Certificate and key files for client cert authentication to the server.
  5. [ cert_file: <filepath> ]
  6. [ key_file: <filepath> ]
  8. # ServerName extension to indicate the name of the server.
  9. # http://tools.ietf.org/html/rfc4366#section-3.1
  10. [ server_name: <string> ]
  12. # Disable validation of the server certificate.
  13. [ insecure_skip_verify: <boolean> | default = false]

<receiver>

Receiver is a named configuration of one or more notification integrations.

Note: As part of lifting the past moratorium on new receivers it was agreed that, in addition to the existing requirements, new notification integrations will be required to have a committed maintainer with push access.

  1. # The unique name of the receiver.
  2. name: <string>
  4. # Configurations for several notification integrations.
  5. email_configs:
  6.   [ - <email_config>, ... ]
  7. opsgenie_configs:
  8.   [ - <opsgenie_config>, ... ]
  9. pagerduty_configs:
  10.   [ - <pagerduty_config>, ... ]
  11. pushover_configs:
  12.   [ - <pushover_config>, ... ]
  13. slack_configs:
  14.   [ - <slack_config>, ... ]
  15. sns_configs:
  16.   [ - <sns_config>, ... ]
  17. victorops_configs:
  18.   [ - <victorops_config>, ... ]
  19. webhook_configs:
  20.   [ - <webhook_config>, ... ]
  21. wechat_configs:
  22.   [ - <wechat_config>, ... ]
  23. telegram_configs:
  24.   [ - <telegram_config>, ... ]

<email_config>

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The email address to send notifications to.
  5. to: <tmpl_string>
  7. # The sender's address.
  8. [ from: <tmpl_string> | default = global.smtp_from ]
  10. # The SMTP host through which emails are sent.
  11. [ smarthost: <string> | default = global.smtp_smarthost ]
  13. # The hostname to identify to the SMTP server.
  14. [ hello: <string> | default = global.smtp_hello ]
  16. # SMTP authentication information.
  17. [ auth_username: <string> | default = global.smtp_auth_username ]
  18. [ auth_password: <secret> | default = global.smtp_auth_password ]
  19. [ auth_secret: <secret> | default = global.smtp_auth_secret ]
  20. [ auth_identity: <string> | default = global.smtp_auth_identity ]
  22. # The SMTP TLS requirement.
  23. # Note that Go does not support unencrypted connections to remote SMTP endpoints.
  24. [ require_tls: <bool> | default = global.smtp_require_tls ]
  26. # TLS configuration.
  27. tls_config:
  28.   [ <tls_config> ]
  30. # The HTML body of the email notification.
  31. [ html: <tmpl_string> | default = '{{ template "email.default.html" . }}' ]
  32. # The text body of the email notification.
  33. [ text: <tmpl_string> ]
  35. # Further headers email header key/value pairs. Overrides any headers
  36. # previously set by the notification implementation.
  37. [ headers: { <string>: <tmpl_string>, ... } ]

<opsgenie_config>

OpsGenie notifications are sent via the OpsGenie API.

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The API key to use when talking to the OpsGenie API.
  5. [ api_key: <secret> | default = global.opsgenie_api_key ]
  7. # The filepath to API key to use when talking to the OpsGenie API. Conflicts with api_key.
  8. [ api_key_file: <filepath> | default = global.opsgenie_api_key_file ]
  10. # The host to send OpsGenie API requests to.
  11. [ api_url: <string> | default = global.opsgenie_api_url ]
  13. # Alert text limited to 130 characters.
  14. [ message: <tmpl_string> | default = '{{ template "opsgenie.default.message" . }}' ]
  16. # A description of the alert.
  17. [ description: <tmpl_string> | default = '{{ template "opsgenie.default.description" . }}' ]
  19. # A backlink to the sender of the notification.
  20. [ source: <tmpl_string> | default = '{{ template "opsgenie.default.source" . }}' ]
  22. # A set of arbitrary key/value pairs that provide further detail
  23. # about the alert.
  24. # All common labels are included as details by default.
  25. [ details: { <string>: <tmpl_string>, ... } ]
  27. # List of responders responsible for notifications.
  28. responders:
  29.   [ - <responder> ... ]
  31. # Comma separated list of tags attached to the notifications.
  32. [ tags: <tmpl_string> ]
  34. # Additional alert note.
  35. [ note: <tmpl_string> ]
  37. # Priority level of alert. Possible values are P1, P2, P3, P4, and P5.
  38. [ priority: <tmpl_string> ]
  40. # Whether to update message and description of the alert in OpsGenie if it already exists
  41. # By default, the alert is never updated in OpsGenie, the new message only appears in activity log.
  42. [ update_alerts: <boolean> | default = false ]
  44. # Optional field that can be used to specify which domain alert is related to.
  45. [ entity: <tmpl_string> ]
  47. # Comma separated list of actions that will be available for the alert.
  48. [ actions: <tmpl_string> ]
  50. # The HTTP client's configuration.
  51. [ http_config: <http_config> | default = global.http_config ]

<responder>

  1. # Exactly one of these fields should be defined.
  2. [ id: <tmpl_string> ]
  3. [ name: <tmpl_string> ]
  4. [ username: <tmpl_string> ]
  6. # "team", "teams, "user", "escalation" or "schedule".
  7. type: <tmpl_string>

<pagerduty_config>

PagerDuty notifications are sent via the PagerDuty API. PagerDuty provides documentation on how to integrate. There are important differences with Alertmanager’s v0.11 and greater support of PagerDuty’s Events API v2.

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The following two options are mutually exclusive.
  5. # The PagerDuty integration key (when using PagerDuty integration type `Events API v2`).
  6. routing_key: <tmpl_secret>
  7. # The PagerDuty integration key (when using PagerDuty integration type `Prometheus`).
  8. service_key: <tmpl_secret>
  10. # The URL to send API requests to
  11. [ url: <string> | default = global.pagerduty_url ]
  13. # The client identification of the Alertmanager.
  14. [ client:  <tmpl_string> | default = '{{ template "pagerduty.default.client" . }}' ]
  15. # A backlink to the sender of the notification.
  16. [ client_url:  <tmpl_string> | default = '{{ template "pagerduty.default.clientURL" . }}' ]
  18. # A description of the incident.
  19. [ description: <tmpl_string> | default = '{{ template "pagerduty.default.description" .}}' ]
  21. # Severity of the incident.
  22. [ severity: <tmpl_string> | default = 'error' ]
  24. # A set of arbitrary key/value pairs that provide further detail
  25. # about the incident.
  26. [ details: { <string>: <tmpl_string>, ... } | default = {
  27.   firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
  28.   resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
  29.   num_firing:   '{{ .Alerts.Firing | len }}'
  30.   num_resolved: '{{ .Alerts.Resolved | len }}'
  31. } ]
  33. # Images to attach to the incident.
  34. images:
  35.   [ <image_config> ... ]
  37. # Links to attach to the incident.
  38. links:
  39.   [ <link_config> ... ]
  41. # The part or component of the affected system that is broken.
  42. [ component: <tmpl_string> ]
  44. # A cluster or grouping of sources.
  45. [ group: <tmpl_string> ]
  47. # The class/type of the event.
  48. [ class: <tmpl_string> ]
  50. # The HTTP client's configuration.
  51. [ http_config: <http_config> | default = global.http_config ]

<image_config>

The fields are documented in the PagerDuty API documentation.

  1. href: <tmpl_string>
  2. source: <tmpl_string>
  3. alt: <tmpl_string>

<link_config>

The fields are documented in the PagerDuty API documentation.

  1. href: <tmpl_string>
  2. text: <tmpl_string>

<pushover_config>

Pushover notifications are sent via the Pushover API.

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The recipient user's user key.
  5. user_key: <secret>
  7. # Your registered application's API token, see https://pushover.net/apps
  8. # You can also register a token by cloning this Prometheus app:
  9. # https://pushover.net/apps/clone/prometheus
  10. token: <secret>
  12. # Notification title.
  13. [ title: <tmpl_string> | default = '{{ template "pushover.default.title" . }}' ]
  15. # Notification message.
  16. [ message: <tmpl_string> | default = '{{ template "pushover.default.message" . }}' ]
  18. # A supplementary URL shown alongside the message.
  19. [ url: <tmpl_string> | default = '{{ template "pushover.default.url" . }}' ]
  21. # Priority, see https://pushover.net/api#priority
  22. [ priority: <tmpl_string> | default = '{{ if eq .Status "firing" }}2{{ else }}0{{ end }}' ]
  24. # How often the Pushover servers will send the same notification to the user.
  25. # Must be at least 30 seconds.
  26. [ retry: <duration> | default = 1m ]
  28. # How long your notification will continue to be retried for, unless the user
  29. # acknowledges the notification.
  30. [ expire: <duration> | default = 1h ]
  32. # The HTTP client's configuration.
  33. [ http_config: <http_config> | default = global.http_config ]

<slack_config>

Slack notifications are sent via Slack webhooks. The notification contains an attachment.

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The Slack webhook URL. Either api_url or api_url_file should be set.
  5. # Defaults to global settings if none are set here.
  6. [ api_url: <secret> | default = global.slack_api_url ]
  7. [ api_url_file: <filepath> | default = global.slack_api_url_file ]
  9. # The channel or user to send notifications to.
  10. channel: <tmpl_string>
  12. # API request data as defined by the Slack webhook API.
  13. [ icon_emoji: <tmpl_string> ]
  14. [ icon_url: <tmpl_string> ]
  15. [ link_names: <boolean> | default = false ]
  16. [ username: <tmpl_string> | default = '{{ template "slack.default.username" . }}' ]
  17. # The following parameters define the attachment.
  18. actions:
  19.   [ <action_config> ... ]
  20. [ callback_id: <tmpl_string> | default = '{{ template "slack.default.callbackid" . }}' ]
  21. [ color: <tmpl_string> | default = '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' ]
  22. [ fallback: <tmpl_string> | default = '{{ template "slack.default.fallback" . }}' ]
  23. fields:
  24.   [ <field_config> ... ]
  25. [ footer: <tmpl_string> | default = '{{ template "slack.default.footer" . }}' ]
  26. [ mrkdwn_in: '[' <string>, ... ']' | default = ["fallback", "pretext", "text"] ]
  27. [ pretext: <tmpl_string> | default = '{{ template "slack.default.pretext" . }}' ]
  28. [ short_fields: <boolean> | default = false ]
  29. [ text: <tmpl_string> | default = '{{ template "slack.default.text" . }}' ]
  30. [ title: <tmpl_string> | default = '{{ template "slack.default.title" . }}' ]
  31. [ title_link: <tmpl_string> | default = '{{ template "slack.default.titlelink" . }}' ]
  32. [ image_url: <tmpl_string> ]
  33. [ thumb_url: <tmpl_string> ]
  35. # The HTTP client's configuration.
  36. [ http_config: <http_config> | default = global.http_config ]

<action_config>

The fields are documented in the Slack API documentation for message attachments and interactive messages.

  1. text: <tmpl_string>
  2. type: <tmpl_string>
  3. # Either url or name and value are mandatory.
  4. [ url: <tmpl_string> ]
  5. [ name: <tmpl_string> ]
  6. [ value: <tmpl_string> ]
  8. [ confirm: <action_confirm_field_config> ]
  9. [ style: <tmpl_string> | default = '' ]

<action_confirm_field_config>

The fields are documented in the Slack API documentation.

  1. text: <tmpl_string>
  2. [ dismiss_text: <tmpl_string> | default '' ]
  3. [ ok_text: <tmpl_string> | default '' ]
  4. [ title: <tmpl_string> | default '' ]

<field_config>

The fields are documented in the Slack API documentation.

  1. title: <tmpl_string>
  2. value: <tmpl_string>
  3. [ short: <boolean> | default = slack_config.short_fields ]

<sns_config>

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The SNS API URL i.e. https://sns.us-east-2.amazonaws.com.
  5. #  If not specified, the SNS API URL from the SNS SDK will be used.
  6. [ api_url: <tmpl_string> ]
  8. # Configures AWS's Signature Verification 4 signing process to sign requests.
  9. sigv4:
  10.   [ <sigv4_config> ]
  12. # SNS topic ARN, i.e. arn:aws:sns:us-east-2:698519295917:My-Topic
  13. # If you don't specify this value, you must specify a value for the phone_number or target_arn.
  14. # If you are using a FIFO SNS topic you should set a message group interval longer than 5 minutes
  15. # to prevent messages with the same group key being deduplicated by the SNS default deduplication window
  16. [ topic_arn: <tmpl_string> ]
  18. # Subject line when the message is delivered to email endpoints.
  19. [ subject: <tmpl_string> | default = '{{ template "sns.default.subject" .}}' ]
  21. # Phone number if message is delivered via SMS in E.164 format.
  22. # If you don't specify this value, you must specify a value for the topic_arn or target_arn.
  23. [ phone_number: <tmpl_string> ]
  25. # The  mobile platform endpoint ARN if message is delivered via mobile notifications.
  26. # If you don't specify this value, you must specify a value for the topic_arn or phone_number.
  27. [ target_arn: <tmpl_string> ]
  29. # The message content of the SNS notification.
  30. [ message: <tmpl_string> | default = '{{ template "sns.default.message" .}}' ]
  32. # SNS message attributes.
  33. attributes:
  34.   [ <string>: <string> ... ]
  36. # The HTTP client's configuration.
  37. [ http_config: <http_config> | default = global.http_config ]

<sigv4_config>

  1. # The AWS region. If blank, the region from the default credentials chain is used.
  2. [ region: <string> ]
  4. # The AWS API keys. Both access_key and secret_key must be supplied or both must be blank.
  5. # If blank the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are used.
  6. [ access_key: <string> ]
  7. [ secret_key: <secret> ]
  9. # Named AWS profile used to authenticate.
  10. [ profile: <string> ]
  12. # AWS Role ARN, an alternative to using AWS API keys.
  13. [ role_arn: <string> ]

<matcher>

A matcher is a string with a syntax inspired by PromQL and OpenMetrics. The syntax of a matcher consists of three tokens:

  • A valid Prometheus label name.

  • One of =, !=, =~, or !~. = means equals, != means that the strings are not equal, =~ is used for equality of regex expressions and !~ is used for un-equality of regex expressions. They have the same meaning as known from PromQL selectors.

  • A UTF-8 string, which may be enclosed in double quotes. Before or after each token, there may be any amount of whitespace.

The 3rd token may be the empty string. Within the 3rd token, OpenMetrics escaping rules apply: \" for a double-quote, \n for a line feed, \\ for a literal backslash. Unescaped " must not occur inside the 3rd token (only as the 1st or last character). However, literal line feed characters are tolerated, as are single \ characters not followed by \, n, or ". They act as a literal backslash in that case.

Matchers are ANDed together, meaning that all matchers must evaluate to “true” when tested against the labels on a given alert. For example, an alert with these labels:

  1. {"alertname":"Watchdog","severity":"none"}

would NOT match this list of matchers:

  1. matchers:
  2.   - alertname = Watchdog
  3.   - severity =~ "warning|critical"

In the configuration, multiple matchers are combined in a YAML list. However, it is also possible to combine multiple matchers within a single YAML string, again using syntax inspired by PromQL. In such a string, a leading { and/or a trailing } is optional and will be trimmed before further parsing. Individual matchers are separated by commas outside of quoted parts of the string. Those commas may be surrounded by whitespace. Parts of the string inside unescaped double quotes "…" are considered quoted (and commas don’t act as separators there). If double quotes are escaped with a single backslash \, they are ignored for the purpose of identifying quoted parts of the input string. If the input string, after trimming the optional trailing }, ends with a comma, followed by optional whitespace, this comma and whitespace will be trimmed.

Here are some examples of valid string matchers:

  1. Shown below are two equality matchers combined in a long form YAML list.

    1. matchers:
    2.   - foo = bar
    3.   - dings !=bums

  2. Similar to example 1, shown below are two equality matchers combined in a short form YAML list.

    1. matchers: [ foo = bar, dings != bums ]

    As shown below, in the short-form, it’s generally better to quote the list elements to avoid problems with special characters like commas:

    1. matchers: [ "foo = bar,baz", "dings != bums" ]

  3. You can also put both matchers into one PromQL-like string. Single quotes for the whole string work best here.

    1. matchers: [ '{foo="bar",dings!="bums"}' ]

  4. To avoid any confusion about YAML string quoting and escaping, you can use YAML block quoting and then only worry about the OpenMetrics escaping inside the block. A complex example with a regular expression and different quotes inside the label value is shown below:

    1. matchers:
    2.   - |
    3.       {quote=~"She said: \"Hi, all!( How're you…)?\""}

<victorops_config>

VictorOps notifications are sent out via the VictorOps API

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The API key to use when talking to the VictorOps API.
  5. [ api_key: <secret> | default = global.victorops_api_key ]
  7. # The VictorOps API URL.
  8. [ api_url: <string> | default = global.victorops_api_url ]
  10. # A key used to map the alert to a team.
  11. routing_key: <tmpl_string>
  13. # Describes the behavior of the alert (CRITICAL, WARNING, INFO).
  14. [ message_type: <tmpl_string> | default = 'CRITICAL' ]
  16. # Contains summary of the alerted problem.
  17. [ entity_display_name: <tmpl_string> | default = '{{ template "victorops.default.entity_display_name" . }}' ]
  19. # Contains long explanation of the alerted problem.
  20. [ state_message: <tmpl_string> | default = '{{ template "victorops.default.state_message" . }}' ]
  22. # The monitoring tool the state message is from.
  23. [ monitoring_tool: <tmpl_string> | default = '{{ template "victorops.default.monitoring_tool" . }}' ]
  25. # The HTTP client's configuration.
  26. [ http_config: <http_config> | default = global.http_config ]

<webhook_config>

The webhook receiver allows configuring a generic receiver.

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The endpoint to send HTTP POST requests to.
  5. url: <string>
  7. # The HTTP client's configuration.
  8. [ http_config: <http_config> | default = global.http_config ]
  10. # The maximum number of alerts to include in a single webhook message. Alerts
  11. # above this threshold are truncated. When leaving this at its default value of
  12. # 0, all alerts are included.
  13. [ max_alerts: <int> | default = 0 ]

The Alertmanager will send HTTP POST requests in the following JSON format to the configured endpoint:

  1. {
  2.   "version": "4",
  3.   "groupKey": <string>,              // key identifying the group of alerts (e.g. to deduplicate)
  4.   "truncatedAlerts": <int>,          // how many alerts have been truncated due to "max_alerts"
  5.   "status": "<resolved|firing>",
  6.   "receiver": <string>,
  7.   "groupLabels": <object>,
  8.   "commonLabels": <object>,
  9.   "commonAnnotations": <object>,
  10.   "externalURL": <string>,           // backlink to the Alertmanager.
  11.   "alerts": [
  12.     {
  13.       "status": "<resolved|firing>",
  14.       "labels": <object>,
  15.       "annotations": <object>,
  16.       "startsAt": "<rfc3339>",
  17.       "endsAt": "<rfc3339>",
  18.       "generatorURL": <string>,      // identifies the entity that caused the alert
  19.       "fingerprint": <string>        // fingerprint to identify the alert
  20.     },
  21.     ...
  22.   ]
  23. }

There is a list of integrations with this feature.

<wechat_config>

WeChat notifications are sent via the WeChat API.

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The API key to use when talking to the WeChat API.
  5. [ api_secret: <secret> | default = global.wechat_api_secret ]
  7. # The WeChat API URL.
  8. [ api_url: <string> | default = global.wechat_api_url ]
  10. # The corp id for authentication.
  11. [ corp_id: <string> | default = global.wechat_api_corp_id ]
  13. # API request data as defined by the WeChat API.
  14. [ message: <tmpl_string> | default = '{{ template "wechat.default.message" . }}' ]
  15. # Type of the message type, supported values are `text` and `markdown`.
  16. [ message_type: <string> | default = 'text' ]
  17. [ agent_id: <string> | default = '{{ template "wechat.default.agent_id" . }}' ]
  18. [ to_user: <string> | default = '{{ template "wechat.default.to_user" . }}' ]
  19. [ to_party: <string> | default = '{{ template "wechat.default.to_party" . }}' ]
  20. [ to_tag: <string> | default = '{{ template "wechat.default.to_tag" . }}' ]

<telegram_config>

  1. # Whether to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The Telegram API URL i.e. https://api.telegram.org.
  5. # If not specified, default API URL will be used.
  6. [ api_url: <string> | default = global.telegram_api_url ]
  8. # Telegram bot token
  9. [ bot_token: <string> ]
  11. # ID of the chat where to send the messages.
  12. [ chat_id: <int> ]
  14. # Message template
  15. [ message: <tmpl_string> default = '{{ template "telegram.default.message" .}}' ]
  17. # Disable telegram notifications
  18. [ disable_notifications: <boolean> | default = false ]
  20. # Parse mode for telegram message, supported values are MarkdownV2, Markdown, HTML and empty string for plain text.
  21. [ parse_mode: <string> | default = "MarkdownV2" ]
  23. # The HTTP client's configuration.
  24. [ http_config: <http_config> | default = global.http_config ]