Securing Prometheus API and UI endpoints using TLS encryption

Prometheus does not directly support Transport Layer Security (TLS) encryption for connections to Prometheus instances (i.e. to the expression browser or HTTP API). If you would like to enforce TLS for those connections, we recommend using Prometheus in conjunction with a reverse proxy and applying TLS at the proxy layer. You can use any reverse proxy you like with Prometheus, but in this guide we’ll provide an nginx example.

NOTE: Although TLS connections to Prometheus instances are not supported, TLS is supported for connections from Prometheus instances to scrape targets.

nginx example

Let’s say that you want to run a Prometheus instance behind an nginx server available at the example.com domain (which you own), and for all Prometheus endpoints to be available via the /prometheus endpoint. The full URL for Prometheus’ /metrics endpoint would thus be:

  1. https://example.com/prometheus/metrics

Let’s also say that you’ve generated the following using OpenSSL or an analogous tool:

  • an SSL certificate at /root/certs/example.com/example.com.crt
  • an SSL key at /root/certs/example.com/example.com.key

You can generate a self-signed certificate and private key using this command:

  1. mkdir -p /root/certs/example.com && cd /root/certs/example.com
  2. openssl req \
  3.   -x509 \
  4.   -newkey rsa:4096 \
  5.   -nodes \
  6.   -keyout example.com.key \
  7.   -out example.com.crt

Fill out the appropriate information at the prompts, and make sure to enter example.com at the Common Name prompt.

nginx configuration

Below is an example nginx.conf configuration file. With this configuration, nginx will:

  • enforce TLS encryption using your provided certificate and key
  • proxy all connections to the /prometheus endpoint to a Prometheus server running on the same host (while removing the /prometheus from the URL)
  1. http {
  2.     server {
  3.         listen              443 ssl;
  4.         server_name         example.com;
  5.         ssl_certificate     /root/certs/example.com/example.com.crt;
  6.         ssl_certificate_key /root/certs/example.com/example.com.key;
  8.         location /prometheus/ {
  9.             proxy_pass http://localhost:9090/;
  10.         }
  11.     }
  12. }
  14. events {}

Start nginx as root (since nginx will need to bind to port 443):

  1. sudo nginx -c /usr/local/etc/nginx/nginx.conf

NOTE: This example uses /usr/local/etc/nginx as the location of the nginx configuration file, but this will vary based on the installation. Other common nginx config directories include /usr/local/nginx/conf and /etc/nginx.

Prometheus configuration

When running Prometheus behind the nginx proxy, you’ll need to set the external URL to https://example.com/prometheus and the route prefix to /:

  1. prometheus \
  2.   --config.file=/path/to/prometheus.yml \
  3.   --web.external-url=https://example.com/prometheus \
  4.   --web.route-prefix="/"

Testing

If you’d like to test out the nginx proxy locally using the example.com domain, you can add an entry to your /etc/hosts file that re-routes example.com to localhost:

  1. 127.0.0.1     example.com

You can then use cURL to interact with your local nginx/Prometheus setup:

  1. curl --cacert /root/certs/example.com/example.com.crt \
  2.   https://example.com/prometheus/api/v1/label/job/values

You can connect to the nginx server without specifying certs using the --insecure or -k flag:

  1. curl -k https://example.com/prometheus/api/v1/label/job/values