Template examples

Prometheus supports templating in the annotations and labels of alerts, as well as in served console pages. Templates have the ability to run queries against the local database, iterate over data, use conditionals, format data, etc. The Prometheus templating language is based on the Go templating system.

Simple alert field templates

  1. alert: InstanceDown
  2. expr: up == 0
  3. for: 5m
  4. labels:
  5.   severity: page
  6. annotations:
  7.   summary: "Instance {{$labels.instance}} down"
  8.   description: "{{$labels.instance}} of job {{$labels.job}} has been down for more than 5 minutes."

Alert field templates will be executed during every rule iteration for each alert that fires, so keep any queries and templates lightweight. If you have a need for more complicated templates for alerts, it is recommended to link to a console instead.

Simple iteration

This displays a list of instances, and whether they are up:

  1. {{ range query "up" }}
  2.   {{ .Labels.instance }} {{ .Value }}
  3. {{ end }}

The special . variable contains the value of the current sample for each loop iteration.

Display one value

  1. {{ with query "some_metric{instance='someinstance'}" }}
  2.   {{ . | first | value | humanize }}
  3. {{ end }}

Go and Go’s templating language are both strongly typed, so one must check that samples were returned to avoid an execution error. For example this could happen if a scrape or rule evaluation has not run yet, or a host was down.

The included prom_query_drilldown template handles this, allows for formatting of results, and linking to the expression browser.

Using console URL parameters

  1. {{ with printf "node_memory_MemTotal{job='node',instance='%s'}" .Params.instance | query }}
  2.   {{ . | first | value | humanize1024 }}B
  3. {{ end }}

If accessed as console.html?instance=hostname, .Params.instance will evaluate to hostname.

Advanced iteration

  1. <table>
  2. {{ range printf "node_network_receive_bytes{job='node',instance='%s',device!='lo'}" .Params.instance | query | sortByLabel "device"}}
  3.   <tr><th colspan=2>{{ .Labels.device }}</th></tr>
  4.   <tr>
  5.     <td>Received</td>
  6.     <td>{{ with printf "rate(node_network_receive_bytes{job='node',instance='%s',device='%s'}[5m])" .Labels.instance .Labels.device | query }}{{ . | first | value | humanize }}B/s{{end}}</td>
  7.   </tr>
  8.   <tr>
  9.     <td>Transmitted</td>
  10.     <td>{{ with printf "rate(node_network_transmit_bytes{job='node',instance='%s',device='%s'}[5m])" .Labels.instance .Labels.device | query }}{{ . | first | value | humanize }}B/s{{end}}</td>
  11.   </tr>{{ end }}
  12. </table>

Here we iterate over all network devices and display the network traffic for each.

As the range action does not specify a variable, .Params.instance is not available inside the loop as . is now the loop variable.

Defining reusable templates

Prometheus supports defining templates that can be reused. This is particularly powerful when combined with console library support, allowing sharing of templates across consoles.

  1. {{/* Define the template */}}
  2. {{define "myTemplate"}}
  3.   do something
  4. {{end}}
  6. {{/* Use the template */}}
  7. {{template "myTemplate"}}

Templates are limited to one argument. The args function can be used to wrap multiple arguments.

  1. {{define "myMultiArgTemplate"}}
  2.   First argument: {{.arg0}}
  3.   Second argument: {{.arg1}}
  4. {{end}}
  5. {{template "myMultiArgTemplate" (args 1 2)}}