Configuration

Alertmanager is configured via command-line flags and a configuration file. While the command-line flags configure immutable system parameters, the configuration file defines inhibition rules, notification routing and notification receivers.

The visual editor can assist in building routing trees.

To view all available command-line flags, run alertmanager -h.

Alertmanager can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied and an error is logged. A configuration reload is triggered by sending a SIGHUP to the process or sending a HTTP POST request to the /-/reload endpoint.

Configuration file

To specify which configuration file to load, use the --config.file flag.

  1. ./alertmanager --config.file=alertmanager.yml

The file is written in the YAML format, defined by the scheme described below. Brackets indicate that a parameter is optional. For non-list parameters the value is set to the specified default.

Generic placeholders are defined as follows:

  • <duration>: a duration matching the regular expression [0-9]+(ms|[smhdwy])
  • <labelname>: a string matching the regular expression [a-zA-Z_][a-zA-Z0-9_]*
  • <labelvalue>: a string of unicode characters
  • <filepath>: a valid path in the current working directory
  • <boolean>: a boolean that can take the values true or false
  • <string>: a regular string
  • <secret>: a regular string that is a secret, such as a password
  • <tmpl_string>: a string which is template-expanded before usage
  • <tmpl_secret>: a string which is template-expanded before usage that is a secret
  • <int>: an integer value

The other placeholders are specified separately.

A provided valid example file shows usage in context.

The global configuration specifies parameters that are valid in all other configuration contexts. They also serve as defaults for other configuration sections.

  1. global:
  2.   # The default SMTP From header field.
  3.   [ smtp_from: <tmpl_string> ]
  4.   # The default SMTP smarthost used for sending emails, including port number.
  5.   # Port number usually is 25, or 587 for SMTP over TLS (sometimes referred to as STARTTLS).
  6.   # Example: smtp.example.org:587
  7.   [ smtp_smarthost: <string> ]
  8.   # The default hostname to identify to the SMTP server.
  9.   [ smtp_hello: <string> | default = "localhost" ]
  10.   # SMTP Auth using CRAM-MD5, LOGIN and PLAIN. If empty, Alertmanager doesn't authenticate to the SMTP server.
  11.   [ smtp_auth_username: <string> ]
  12.   # SMTP Auth using LOGIN and PLAIN.
  13.   [ smtp_auth_password: <secret> ]
  14.   # SMTP Auth using PLAIN.
  15.   [ smtp_auth_identity: <string> ]
  16.   # SMTP Auth using CRAM-MD5.
  17.   [ smtp_auth_secret: <secret> ]
  18.   # The default SMTP TLS requirement.
  19.   # Note that Go does not support unencrypted connections to remote SMTP endpoints.
  20.   [ smtp_require_tls: <bool> | default = true ]
  22.   # The API URL to use for Slack notifications.
  23.   [ slack_api_url: <secret> ]
  24.   [ victorops_api_key: <secret> ]
  25.   [ victorops_api_url: <string> | default = "https://alert.victorops.com/integrations/generic/20131114/alert/" ]
  26.   [ pagerduty_url: <string> | default = "https://events.pagerduty.com/v2/enqueue" ]
  27.   [ opsgenie_api_key: <secret> ]
  28.   [ opsgenie_api_url: <string> | default = "https://api.opsgenie.com/" ]
  29.   [ wechat_api_url: <string> | default = "https://qyapi.weixin.qq.com/cgi-bin/" ]
  30.   [ wechat_api_secret: <secret> ]
  31.   [ wechat_api_corp_id: <string> ]
  33.   # The default HTTP client configuration
  34.   [ http_config: <http_config> ]
  36.   # ResolveTimeout is the default value used by alertmanager if the alert does
  37.   # not include EndsAt, after this time passes it can declare the alert as resolved if it has not been updated.
  38.   # This has no impact on alerts from Prometheus, as they always include EndsAt.
  39.   [ resolve_timeout: <duration> | default = 5m ]
  41. # Files from which custom notification template definitions are read.
  42. # The last component may use a wildcard matcher, e.g. 'templates/*.tmpl'.
  43. templates:
  44.   [ - <filepath> ... ]
  46. # The root node of the routing tree.
  47. route: <route>
  49. # A list of notification receivers.
  50. receivers:
  51.   - <receiver> ...
  53. # A list of inhibition rules.
  54. inhibit_rules:
  55.   [ - <inhibit_rule> ... ]

<route>

A route block defines a node in a routing tree and its children. Its optional configuration parameters are inherited from its parent node if not set.

Every alert enters the routing tree at the configured top-level route, which must match all alerts (i.e. not have any configured matchers). It then traverses the child nodes. If continue is set to false, it stops after the first matching child. If continue is true on a matching node, the alert will continue matching against subsequent siblings. If an alert does not match any children of a node (no matching child nodes, or none exist), the alert is handled based on the configuration parameters of the current node.

  1. [ receiver: <string> ]
  2. # The labels by which incoming alerts are grouped together. For example,
  3. # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  4. # be batched into a single group.
  5. #
  6. # To aggregate by all possible labels use the special value '...' as the sole label name, for example:
  7. # group_by: ['...']
  8. # This effectively disables aggregation entirely, passing through all
  9. # alerts as-is. This is unlikely to be what you want, unless you have
  10. # a very low alert volume or your upstream notification system performs
  11. # its own grouping.
  12. [ group_by: '[' <labelname>, ... ']' ]
  14. # Whether an alert should continue matching subsequent sibling nodes.
  15. [ continue: <boolean> | default = false ]
  17. # A set of equality matchers an alert has to fulfill to match the node.
  18. match:
  19.   [ <labelname>: <labelvalue>, ... ]
  21. # A set of regex-matchers an alert has to fulfill to match the node.
  22. match_re:
  23.   [ <labelname>: <regex>, ... ]
  25. # How long to initially wait to send a notification for a group
  26. # of alerts. Allows to wait for an inhibiting alert to arrive or collect
  27. # more initial alerts for the same group. (Usually ~0s to few minutes.)
  28. [ group_wait: <duration> | default = 30s ]
  30. # How long to wait before sending a notification about new alerts that
  31. # are added to a group of alerts for which an initial notification has
  32. # already been sent. (Usually ~5m or more.)
  33. [ group_interval: <duration> | default = 5m ]
  35. # How long to wait before sending a notification again if it has already
  36. # been sent successfully for an alert. (Usually ~3h or more).
  37. [ repeat_interval: <duration> | default = 4h ]
  39. # Zero or more child routes.
  40. routes:
  41.   [ - <route> ... ]

Example

  1. # The root route with all parameters, which are inherited by the child
  2. # routes if they are not overwritten.
  3. route:
  4.   receiver: 'default-receiver'
  5.   group_wait: 30s
  6.   group_interval: 5m
  7.   repeat_interval: 4h
  8.   group_by: [cluster, alertname]
  9.   # All alerts that do not match the following child routes
  10.   # will remain at the root node and be dispatched to 'default-receiver'.
  11.   routes:
  12.   # All alerts with service=mysql or service=cassandra
  13.   # are dispatched to the database pager.
  14.   - receiver: 'database-pager'
  15.     group_wait: 10s
  16.     match_re:
  17.       service: mysql|cassandra
  18.   # All alerts with the team=frontend label match this sub-route.
  19.   # They are grouped by product and environment rather than cluster
  20.   # and alertname.
  21.   - receiver: 'frontend-pager'
  22.     group_by: [product, environment]
  23.     match:
  24.       team: frontend

<inhibit_rule>

An inhibition rule mutes an alert (target) matching a set of matchers when an alert (source) exists that matches another set of matchers. Both target and source alerts must have the same label values for the label names in the equal list.

Semantically, a missing label and a label with an empty value are the same thing. Therefore, if all the label names listed in equal are missing from both the source and target alerts, the inhibition rule will apply.

To prevent an alert from inhibiting itself, an alert that matches both the target and the source side of a rule cannot be inhibited by alerts for which the same is true (including itself). However, we recommend to choose target and source matchers in a way that alerts never match both sides. It is much easier to reason about and does not trigger this special case.

  1. # Matchers that have to be fulfilled in the alerts to be muted.
  2. target_match:
  3.   [ <labelname>: <labelvalue>, ... ]
  4. target_match_re:
  5.   [ <labelname>: <regex>, ... ]
  7. # Matchers for which one or more alerts have to exist for the
  8. # inhibition to take effect.
  9. source_match:
  10.   [ <labelname>: <labelvalue>, ... ]
  11. source_match_re:
  12.   [ <labelname>: <regex>, ... ]
  14. # Labels that must have an equal value in the source and target
  15. # alert for the inhibition to take effect.
  16. [ equal: '[' <labelname>, ... ']' ]

<http_config>

A http_config allows configuring the HTTP client that the receiver uses to communicate with HTTP-based API services.

  1. # Note that `basic_auth`, `bearer_token` and `bearer_token_file` options are
  2. # mutually exclusive.
  4. # Sets the `Authorization` header with the configured username and password.
  5. # password and password_file are mutually exclusive.
  6. basic_auth:
  7.   [ username: <string> ]
  8.   [ password: <secret> ]
  9.   [ password_file: <string> ]
  11. # Sets the `Authorization` header with the configured bearer token.
  12. [ bearer_token: <secret> ]
  14. # Sets the `Authorization` header with the bearer token read from the configured file.
  15. [ bearer_token_file: <filepath> ]
  17. # Configures the TLS settings.
  18. tls_config:
  19.   [ <tls_config> ]
  21. # Optional proxy URL.
  22. [ proxy_url: <string> ]

<tls_config>

A tls_config allows configuring TLS connections.

  1. # CA certificate to validate the server certificate with.
  2. [ ca_file: <filepath> ]
  4. # Certificate and key files for client cert authentication to the server.
  5. [ cert_file: <filepath> ]
  6. [ key_file: <filepath> ]
  8. # ServerName extension to indicate the name of the server.
  9. # http://tools.ietf.org/html/rfc4366#section-3.1
  10. [ server_name: <string> ]
  12. # Disable validation of the server certificate.
  13. [ insecure_skip_verify: <boolean> | default = false]

<receiver>

Receiver is a named configuration of one or more notification integrations.

We’re not actively adding new receivers, we recommend implementing custom notification integrations via the webhook receiver.

  1. # The unique name of the receiver.
  2. name: <string>
  4. # Configurations for several notification integrations.
  5. email_configs:
  6.   [ - <email_config>, ... ]
  7. pagerduty_configs:
  8.   [ - <pagerduty_config>, ... ]
  9. pushover_configs:
  10.   [ - <pushover_config>, ... ]
  11. slack_configs:
  12.   [ - <slack_config>, ... ]
  13. opsgenie_configs:
  14.   [ - <opsgenie_config>, ... ]
  15. webhook_configs:
  16.   [ - <webhook_config>, ... ]
  17. victorops_configs:
  18.   [ - <victorops_config>, ... ]
  19. wechat_configs:
  20.   [ - <wechat_config>, ... ]

<email_config>

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The email address to send notifications to.
  5. to: <tmpl_string>
  7. # The sender address.
  8. [ from: <tmpl_string> | default = global.smtp_from ]
  10. # The SMTP host through which emails are sent.
  11. [ smarthost: <string> | default = global.smtp_smarthost ]
  13. # The hostname to identify to the SMTP server.
  14. [ hello: <string> | default = global.smtp_hello ]
  16. # SMTP authentication information.
  17. [ auth_username: <string> | default = global.smtp_auth_username ]
  18. [ auth_password: <secret> | default = global.smtp_auth_password ]
  19. [ auth_secret: <secret> | default = global.smtp_auth_secret ]
  20. [ auth_identity: <string> | default = global.smtp_auth_identity ]
  22. # The SMTP TLS requirement.
  23. # Note that Go does not support unencrypted connections to remote SMTP endpoints.
  24. [ require_tls: <bool> | default = global.smtp_require_tls ]
  26. # TLS configuration.
  27. tls_config:
  28.   [ <tls_config> ]
  30. # The HTML body of the email notification.
  31. [ html: <tmpl_string> | default = '{{ template "email.default.html" . }}' ]
  32. # The text body of the email notification.
  33. [ text: <tmpl_string> ]
  35. # Further headers email header key/value pairs. Overrides any headers
  36. # previously set by the notification implementation.
  37. [ headers: { <string>: <tmpl_string>, ... } ]

<pagerduty_config>

PagerDuty notifications are sent via the PagerDuty API. PagerDuty provides documentation on how to integrate. There are important differences with Alertmanager’s v0.11 and greater support of PagerDuty’s Events API v2.

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The following two options are mutually exclusive.
  5. # The PagerDuty integration key (when using PagerDuty integration type `Events API v2`).
  6. routing_key: <tmpl_secret>
  7. # The PagerDuty integration key (when using PagerDuty integration type `Prometheus`).
  8. service_key: <tmpl_secret>
  10. # The URL to send API requests to
  11. [ url: <string> | default = global.pagerduty_url ]
  13. # The client identification of the Alertmanager.
  14. [ client:  <tmpl_string> | default = '{{ template "pagerduty.default.client" . }}' ]
  15. # A backlink to the sender of the notification.
  16. [ client_url:  <tmpl_string> | default = '{{ template "pagerduty.default.clientURL" . }}' ]
  18. # A description of the incident.
  19. [ description: <tmpl_string> | default = '{{ template "pagerduty.default.description" .}}' ]
  21. # Severity of the incident.
  22. [ severity: <tmpl_string> | default = 'error' ]
  24. # A set of arbitrary key/value pairs that provide further detail
  25. # about the incident.
  26. [ details: { <string>: <tmpl_string>, ... } | default = {
  27.   firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
  28.   resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
  29.   num_firing:   '{{ .Alerts.Firing | len }}'
  30.   num_resolved: '{{ .Alerts.Resolved | len }}'
  31. } ]
  33. # Images to attach to the incident.
  34. images:
  35.   [ <image_config> ... ]
  37. # Links to attach to the incident.
  38. links:
  39.   [ <link_config> ... ]
  41. # The HTTP client's configuration.
  42. [ http_config: <http_config> | default = global.http_config ]

<image_config>

The fields are documented in the PagerDuty API documentation.

  1. href: <tmpl_string>
  2. source: <tmpl_string>
  3. alt: <tmpl_string>

<link_config>

The fields are documented in the PagerDuty API documentation.

  1. href: <tmpl_string>
  2. text: <tmpl_string>

<pushover_config>

Pushover notifications are sent via the Pushover API.

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The recipient user’s user key.
  5. user_key: <secret>
  7. # Your registered application’s API token, see https://pushover.net/apps
  8. token: <secret>
  10. # Notification title.
  11. [ title: <tmpl_string> | default = '{{ template "pushover.default.title" . }}' ]
  13. # Notification message.
  14. [ message: <tmpl_string> | default = '{{ template "pushover.default.message" . }}' ]
  16. # A supplementary URL shown alongside the message.
  17. [ url: <tmpl_string> | default = '{{ template "pushover.default.url" . }}' ]
  19. # Priority, see https://pushover.net/api#priority
  20. [ priority: <tmpl_string> | default = '{{ if eq .Status "firing" }}2{{ else }}0{{ end }}' ]
  22. # How often the Pushover servers will send the same notification to the user.
  23. # Must be at least 30 seconds.
  24. [ retry: <duration> | default = 1m ]
  26. # How long your notification will continue to be retried for, unless the user
  27. # acknowledges the notification.
  28. [ expire: <duration> | default = 1h ]
  30. # The HTTP client's configuration.
  31. [ http_config: <http_config> | default = global.http_config ]

<slack_config>

Slack notifications are sent via Slack webhooks. The notification contains an attachment.

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The Slack webhook URL.
  5. [ api_url: <secret> | default = global.slack_api_url ]
  7. # The channel or user to send notifications to.
  8. channel: <tmpl_string>
  10. # API request data as defined by the Slack webhook API.
  11. [ icon_emoji: <tmpl_string> ]
  12. [ icon_url: <tmpl_string> ]
  13. [ link_names: <boolean> | default = false ]
  14. [ username: <tmpl_string> | default = '{{ template "slack.default.username" . }}' ]
  15. # The following parameters define the attachment.
  16. actions:
  17.   [ <action_config> ... ]
  18. [ callback_id: <tmpl_string> | default = '{{ template "slack.default.callbackid" . }}' ]
  19. [ color: <tmpl_string> | default = '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' ]
  20. [ fallback: <tmpl_string> | default = '{{ template "slack.default.fallback" . }}' ]
  21. fields:
  22.   [ <field_config> ... ]
  23. [ footer: <tmpl_string> | default = '{{ template "slack.default.footer" . }}' ]
  24. [ mrkdwn_in: '[' <string>, ... ']' | default = ["fallback", "pretext", "text"] ]
  25. [ pretext: <tmpl_string> | default = '{{ template "slack.default.pretext" . }}' ]
  26. [ short_fields: <boolean> | default = false ]
  27. [ text: <tmpl_string> | default = '{{ template "slack.default.text" . }}' ]
  28. [ title: <tmpl_string> | default = '{{ template "slack.default.title" . }}' ]
  29. [ title_link: <tmpl_string> | default = '{{ template "slack.default.titlelink" . }}' ]
  30. [ image_url: <tmpl_string> ]
  31. [ thumb_url: <tmpl_string> ]
  33. # The HTTP client's configuration.
  34. [ http_config: <http_config> | default = global.http_config ]

<action_config>

The fields are documented in the Slack API documentation for message attachments and interactive messages.

  1. text: <tmpl_string>
  2. type: <tmpl_string>
  3. # Either url or name and value are mandatory.
  4. [ url: <tmpl_string> ]
  5. [ name: <tmpl_string> ]
  6. [ value: <tmpl_string> ]
  8. [ confirm: <action_confirm_field_config> ]
  9. [ style: <tmpl_string> | default = '' ]

<action_confirm_field_config>

The fields are documented in the Slack API documentation.

  1. text: <tmpl_string>
  2. [ dismiss_text: <tmpl_string> | default '' ]
  3. [ ok_text: <tmpl_string> | default '' ]
  4. [ title: <tmpl_string> | default '' ]

<field_config>

The fields are documented in the Slack API documentation.

  1. title: <tmpl_string>
  2. value: <tmpl_string>
  3. [ short: <boolean> | default = slack_config.short_fields ]

<opsgenie_config>

OpsGenie notifications are sent via the OpsGenie API.

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The API key to use when talking to the OpsGenie API.
  5. [ api_key: <secret> | default = global.opsgenie_api_key ]
  7. # The host to send OpsGenie API requests to.
  8. [ api_url: <string> | default = global.opsgenie_api_url ]
  10. # Alert text limited to 130 characters.
  11. [ message: <tmpl_string> ]
  13. # A description of the incident.
  14. [ description: <tmpl_string> | default = '{{ template "opsgenie.default.description" . }}' ]
  16. # A backlink to the sender of the notification.
  17. [ source: <tmpl_string> | default = '{{ template "opsgenie.default.source" . }}' ]
  19. # A set of arbitrary key/value pairs that provide further detail
  20. # about the incident.
  21. [ details: { <string>: <tmpl_string>, ... } ]
  23. # List of responders responsible for notifications.
  24. responders:
  25.   [ - <responder> ... ]
  27. # Comma separated list of tags attached to the notifications.
  28. [ tags: <tmpl_string> ]
  30. # Additional alert note.
  31. [ note: <tmpl_string> ]
  33. # Priority level of alert. Possible values are P1, P2, P3, P4, and P5.
  34. [ priority: <tmpl_string> ]
  36. # The HTTP client's configuration.
  37. [ http_config: <http_config> | default = global.http_config ]

<responder>

  1. # Exactly one of these fields should be defined.
  2. [ id: <tmpl_string> ]
  3. [ name: <tmpl_string> ]
  4. [ username: <tmpl_string> ]
  6. # "team", "user", "escalation" or schedule".
  7. type: <tmpl_string>

<victorops_config>

VictorOps notifications are sent out via the VictorOps API

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The API key to use when talking to the VictorOps API.
  5. [ api_key: <secret> | default = global.victorops_api_key ]
  7. # The VictorOps API URL.
  8. [ api_url: <string> | default = global.victorops_api_url ]
  10. # A key used to map the alert to a team.
  11. routing_key: <tmpl_string>
  13. # Describes the behavior of the alert (CRITICAL, WARNING, INFO).
  14. [ message_type: <tmpl_string> | default = 'CRITICAL' ]
  16. # Contains summary of the alerted problem.
  17. [ entity_display_name: <tmpl_string> | default = '{{ template "victorops.default.entity_display_name" . }}' ]
  19. # Contains long explanation of the alerted problem.
  20. [ state_message: <tmpl_string> | default = '{{ template "victorops.default.state_message" . }}' ]
  22. # The monitoring tool the state message is from.
  23. [ monitoring_tool: <tmpl_string> | default = '{{ template "victorops.default.monitoring_tool" . }}' ]
  25. # The HTTP client's configuration.
  26. [ http_config: <http_config> | default = global.http_config ]

<webhook_config>

The webhook receiver allows configuring a generic receiver.

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The endpoint to send HTTP POST requests to.
  5. url: <string>
  7. # The HTTP client's configuration.
  8. [ http_config: <http_config> | default = global.http_config ]
  10. # The maximum number of alerts to include in a single webhook message. Alerts
  11. # above this threshold are truncated. When leaving this at its default value of
  12. # 0, all alerts are included.
  13. [ max_alerts: <int> | default = 0 ]

The Alertmanager will send HTTP POST requests in the following JSON format to the configured endpoint:

  1. {
  2.   "version": "4",
  3.   "groupKey": <string>,              // key identifying the group of alerts (e.g. to deduplicate)
  4.   "truncatedAlerts": <int>,          // how many alerts have been truncated due to "max_alerts"
  5.   "status": "<resolved|firing>",
  6.   "receiver": <string>,
  7.   "groupLabels": <object>,
  8.   "commonLabels": <object>,
  9.   "commonAnnotations": <object>,
  10.   "externalURL": <string>,           // backlink to the Alertmanager.
  11.   "alerts": [
  12.     {
  13.       "status": "<resolved|firing>",
  14.       "labels": <object>,
  15.       "annotations": <object>,
  16.       "startsAt": "<rfc3339>",
  17.       "endsAt": "<rfc3339>",
  18.       "generatorURL": <string>       // identifies the entity that caused the alert
  19.     },
  20.     ...
  21.   ]
  22. }

There is a list of integrations with this feature.

<wechat_config>

WeChat notifications are sent via the WeChat API.

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The API key to use when talking to the WeChat API.
  5. [ api_secret: <secret> | default = global.wechat_api_secret ]
  7. # The WeChat API URL.
  8. [ api_url: <string> | default = global.wechat_api_url ]
  10. # The corp id for authentication.
  11. [ corp_id: <string> | default = global.wechat_api_corp_id ]
  13. # API request data as defined by the WeChat API.
  14. [ message: <tmpl_string> | default = '{{ template "wechat.default.message" . }}' ]
  15. [ agent_id: <string> | default = '{{ template "wechat.default.agent_id" . }}' ]
  16. [ to_user: <string> | default = '{{ template "wechat.default.to_user" . }}' ]
  17. [ to_party: <string> | default = '{{ template "wechat.default.to_party" . }}' ]
  18. [ to_tag: <string> | default = '{{ template "wechat.default.to_tag" . }}' ]