Notification Template Examples

The following are all different examples of alerts and corresponding Alertmanager configuration file setups (alertmanager.yml). Each use the Go templating system.

Customizing Slack notifications

In this example we’ve customised our Slack notification to send a URL to our organisation’s wiki on how to deal with the particular alert that’s been sent.

  1. global:
  2.   # Also possible to place this URL in a file.
  3.   # Ex: `slack_api_url_file: '/etc/alertmanager/slack_url'`
  4.   slack_api_url: '<slack_webhook_url>'
  6. route:
  7.   receiver: 'slack-notifications'
  8.   group_by: [alertname, datacenter, app]
  10. receivers:
  11. - name: 'slack-notifications'
  12.   slack_configs:
  13.   - channel: '#alerts'
  14.     text: 'https://internal.myorg.net/wiki/alerts/{{ .GroupLabels.app }}/{{ .GroupLabels.alertname }}'

Accessing annotations in CommonAnnotations

In this example we again customize the text sent to our Slack receiver accessing the summary and description stored in the CommonAnnotations of the data sent by the Alertmanager.

Alert

  1. groups:
  2. - name: Instances
  3.   rules:
  4.   - alert: InstanceDown
  5.     expr: up == 0
  6.     for: 5m
  7.     labels:
  8.       severity: page
  9.     # Prometheus templates apply here in the annotation and label fields of the alert.
  10.     annotations:
  11.       description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
  12.       summary: 'Instance {{ $labels.instance }} down'

Receiver

  1. - name: 'team-x'
  2.   slack_configs:
  3.   - channel: '#alerts'
  4.     # Alertmanager templates apply here.
  5.     text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"

Ranging over all received Alerts

Finally, assuming the same alert as the previous example, we customize our receiver to range over all of the alerts received from the Alertmanager, printing their respective annotation summaries and descriptions on new lines.

Receiver

  1. - name: 'default-receiver'
  2.   slack_configs:
  3.   - channel: '#alerts'
  4.     title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
  5.     text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

Defining reusable templates

Going back to our first example, we can also provide a file containing named templates which are then loaded by Alertmanager in order to avoid complex templates that span many lines. Create a file under /alertmanager/template/myorg.tmpl and create a template in it named “slack.myorg.txt”:

  1. {{ define "slack.myorg.text" }}https://internal.myorg.net/wiki/alerts/{{ .GroupLabels.app }}/{{ .GroupLabels.alertname }}{{ end}}

The configuration now loads the template with the given name for the “text” field and we provide a path to our custom template file:

  1. global:
  2.   slack_api_url: '<slack_webhook_url>'
  4. route:
  5.   receiver: 'slack-notifications'
  6.   group_by: [alertname, datacenter, app]
  8. receivers:
  9. - name: 'slack-notifications'
  10.   slack_configs:
  11.   - channel: '#alerts'
  12.     text: '{{ template "slack.myorg.text" . }}'
  14. templates:
  15. - '/etc/alertmanager/templates/myorg.tmpl'

This example is explained in further detail in this blogpost.