Prometheus 2.0 migration guide

Prometheus 2.0 migration guide

In line with our stability promise,the Prometheus 2.0 release contains a number of backwards incompatible changes.This document offers guidance on migrating from Prometheus 1.8 to Prometheus 2.0.

Flags

The format of the Prometheus command line flags has changed. Instead of asingle dash, all flags now use a double dash. Common flags (—config.file,—web.listen-address and —web.external-url) are still the same but beyondthat, almost all the storage-related flags have been removed.

Some notable flags which have been removed:

-alertmanager.url In Prometheus 2.0, the command line flags for configuringa static Alertmanager URL have been removed. Alertmanager must now bediscovered via service discovery, see Alertmanager service discovery.
-log.format In Prometheus 2.0 logs can only be streamed to standard error.
-query.staleness-delta has been renamed to —query.lookback-delta; Prometheus2.0 introduces a new mechanism for handling staleness, see staleness.
-storage.local.* Prometheus 2.0 introduces a new storage engine, as such allflags relating to the old engine have been removed. For information on thenew engine, see Storage.
-storage.remote.* Prometheus 2.0 has removed the already deprecated remotestorage flags, and will fail to start if they are supplied. To write toInfluxDB, Graphite, or OpenTSDB use the relevant storage adapter.

Alertmanager service discovery

Alertmanager service discovery was introduced in Prometheus 1.4, allowing Prometheusto dynamically discover Alertmanager replicas using the same mechanism as scrapetargets. In Prometheus 2.0, the command line flags for static Alertmanager confighave been removed, so the following command line flag:

./prometheus -alertmanager.url=http://alertmanager:9093/

Would be replaced with the following in the prometheus.yml config file:

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager:9093

You can also use all the usual Prometheus service discovery integrations andrelabeling in your Alertmanager configuration. This snippet instructsPrometheus to search for Kubernetes pods, in the default namespace, with thelabel name: alertmanager and with a non-empty port.

alerting:
  alertmanagers:
  - kubernetes_sd_configs:
      - role: pod
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_name]
      regex: alertmanager
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      regex: default
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      regex:
      action: drop

Recording rules and alerts

The format for configuring alerting and recording rules has been changed to YAML.An example of a recording rule and alert in the old format:

job:request_duration_seconds:histogram_quantile99 =
  histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m])) by (le, job))
ALERT FrontendRequestLatency
  IF job:request_duration_seconds:histogram_quantile99{job="frontend"} > 0.1
  FOR 5m
  ANNOTATIONS {
    summary = "High frontend request latency",
  }

Would look like this:

groups:
- name: example.rules
  rules:
  - record: job:request_duration_seconds:histogram_quantile99
    expr: histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m]))
      BY (le, job))
  - alert: FrontendRequestLatency
    expr: job:request_duration_seconds:histogram_quantile99{job="frontend"} > 0.1
    for: 5m
    annotations:
      summary: High frontend request latency

To help with the change, the promtool tool has a mode to automate the rules conversion. Given a .rules file, it will output a .rules.yml file in thenew format. For example:

$ promtool update rules example.rules

Note that you will need to use promtool from 2.0, not 1.8.

Storage

The data format in Prometheus 2.0 has completely changed and is not backwardscompatible with 1.8. To retain access to your historic monitoring data werecommend you run a non-scraping Prometheus instance running at least version1.8.1 in parallel with your Prometheus 2.0 instance, and have the new serverread existing data from the old one via the remote read protocol.

Your Prometheus 1.8 instance should be started with the following flags and anconfig file containing only the external_labels setting (if any):

$ ./prometheus-1.8.1.linux-amd64/prometheus -web.listen-address ":9094" -config.file old.yml

Prometheus 2.0 can then be started (on the same machine) with the following flags:

$ ./prometheus-2.0.0.linux-amd64/prometheus --config.file prometheus.yml

Where prometheus.yml contains in addition to your full existing configuration, the stanza:

remote_read:
  - url: "http://localhost:9094/api/v1/read"

PromQL

The following features have been removed from PromQL:

drop_common_labels function - the without aggregation modifier should be usedinstead.
keep_common aggregation modifier - the by modifier should be used instead.
count_scalar function - use cases are better handled by absent() or correctpropagation of labels in operations.See issue #3060 for moredetails.

Miscellaneous

Prometheus non-root user

The Prometheus Docker image is now built to run Prometheusas a non-root user. If youwant the Prometheus UI/API to listen on a low port number (say, port 80), you'llneed to override it. For Kubernetes, you would use the following YAML:

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo-2
spec:
  securityContext:
    runAsUser: 0
...

See Configure a Security Context for a Pod or Containerfor more details.

If you're using Docker, then the following snippet would be used:

docker run -u root -p 80:80 prom/prometheus:v2.0.0-rc.2  --web.listen-address :80

Prometheus lifecycle

If you use the Prometheus /-/reload HTTP endpoint to automatically reload yourPrometheus config when it changes,these endpoints are disabled by default for security reasons in Prometheus 2.0.To enable them, set the —web.enable-lifecycle flag.

Migration