Console templates

Console templates allow for creation of arbitrary consoles using the Gotemplating language. These are servedfrom the Prometheus server.

Console templates are the most powerful way to create templates that can beeasily managed in source control. There is a learning curve though, so users newto this style of monitoring should try outGrafana first.

Getting started

Prometheus comes with an example set of consoles to get you going. These can befound at /consoles/index.html.example on a running Prometheus and willdisplay Node Exporter consoles if Prometheus is scraping Node Exporters with ajob="node" label.

The example consoles have 5 parts:

• A navigation bar on top
• A menu on the left
• Time controls on the bottom
• The main content in the center, usually graphs
• A table on the rightThe navigation bar is for links to other systems, such as other Prometheis¹,documentation, and whatever else makes sense to you. The menu is for navigationinside the same Prometheus server, which is very useful to be able to quicklyopen a console in another tab to correlate information. Both are configured inconsole_libraries/menu.lib.

The time controls allow changing of the duration and range of the graphs.Console URLs can be shared and will show the same graphs for others.

The main content is usually graphs. There is a configurable JavaScript graphinglibrary provided that will handle requesting data from Prometheus, and renderingit via Rickshaw.

Finally, the table on the right can be used to display statistics in a morecompact form than graphs.

Example Console

This is a basic console. It shows the number of tasks, how many of them are up,the average CPU usage, and the average memory usage in the right-hand-sidetable. The main content has a queries-per-second graph.

{{template "head" .}}
{{template "prom_right_table_head"}}
<tr>
  <th>MyJob</th>
  <th>{{ template "prom_query_drilldown" (args "sum(up{job='myjob'})") }}
      / {{ template "prom_query_drilldown" (args "count(up{job='myjob'})") }}
  </th>
</tr>
<tr>
  <td>CPU</td>
  <td>{{ template "prom_query_drilldown" (args
      "avg by(job)(rate(process_cpu_seconds_total{job='myjob'}[5m]))"
      "s/s" "humanizeNoSmallPrefix") }}
  </td>
</tr>
<tr>
  <td>Memory</td>
  <td>{{ template "prom_query_drilldown" (args
       "avg by(job)(process_resident_memory_bytes{job='myjob'})"
       "B" "humanize1024") }}
  </td>
</tr>
{{template "prom_right_table_tail"}}
{{template "prom_content_head" .}}
<h1>MyJob</h1>
<h3>Queries</h3>
<div id="queryGraph"></div>
<script>
new PromConsole.Graph({
  node: document.querySelector("#queryGraph"),
  expr: "sum(rate(http_query_count{job='myjob'}[5m]))",
  name: "Queries",
  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
  yUnits: "/s",
  yTitle: "Queries"
})
</script>
{{template "prom_content_tail" .}}
{{template "tail"}}

The prom_right_table_head and prom_right_table_tail templates contain theright-hand-side table. This is optional.

prom_query_drilldown is a template that will evaluate the expression passed to it, format it,and link to the expression in the expression browser. The firstargument is the expression. The second argument is the unit to use. The thirdargument is how to format the output. Only the first argument is required.

Valid output formats for the third argument to prom_query_drilldown:

• Not specified: Default Go display output.
• humanize: Display the result using metric prefixes.
• humanizeNoSmallPrefix: For absolute values greater than 1, display theresult using metric prefixes. Forabsolute values less than 1, display 3 significant digits. This is usefulto avoid units such as milliqueries per second that can be produced byhumanize.
• humanize1024: Display the humanized result using a base of 1024 rather than 1000.This is usually used with B as the second argument to produce units such as KiB and MiB.
• printf.3g: Display 3 significant digits.Custom formats can be defined. Seeprom.lib for examples.

Graph Library

The graph library is invoked as:

<div id="queryGraph"></div>
<script>
new PromConsole.Graph({
  node: document.querySelector("#queryGraph"),
  expr: "sum(rate(http_query_count{job='myjob'}[5m]))"
})
</script>

The head template loads the required Javascript and CSS.

Parameters to the graph library:

If both expr and name are lists, they must be of the same length. The namewill be applied to the plots for the corresponding expression.

Valid options for the yAxisFormatter and yHoverFormatter:

• PromConsole.NumberFormatter.humanize: Format using metric prefixes.
• PromConsole.NumberFormatter.humanizeNoSmallPrefix: For absolute valuesgreater than 1, format using using metric prefixes.For absolute values less than 1, format with 3 significant digits. This isuseful to avoid units such as milliqueries per second that can be produced byPromConsole.NumberFormatter.humanize.
• PromConsole.NumberFormatter.humanize1024: Format the humanized result using a base of 1024 rather than 1000.