Bring your own Prometheus instance

Even though Linkerd comes with its own Prometheus instance, there can be cases where using an external instance makes more sense for various reasons.

Note

Note that this approach requires you to manually add and maintain additional scrape configuration in your Prometheus configuration. If you prefer to use the default Linkerd Prometheus add-on, you can export the metrics to your existing monitoring infrastructure following the instructions at https://linkerd.io/2/tasks/exporting-metrics/

This tutorial shows how to configure an external Prometheus instance to scrape both the control plane as well as the proxy’s metrics in a format that is consumable both by a user as well as Linkerd control plane components like web, etc.

There are two important points to tackle here.

  • Configuring external Prometheus instance to get the Linkerd metrics.
  • Configuring the Linkerd control plane components to use that Prometheus.

Prometheus Scrape Configuration

The following scrape configuration has to be applied to the external Prometheus instance.

Note

The below scrape configuration is a subset of linkerd-prometheus scrape configuration.

Before applying, it is important to replace templated values (present in {{}}) with direct values for the below configuration to work.

  1.     - job_name: 'linkerd-controller'
  3.       scrape_interval: 10s
  4.       scrape_timeout: 10s
  6.       kubernetes_sd_configs:
  7.       - role: pod
  8.         namespaces:
  9.           names: ['{{.Values.global.namespace}}']
  10.       relabel_configs:
  11.       - source_labels:
  12.         - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
  13.         - __meta_kubernetes_pod_container_port_name
  14.         action: keep
  15.         regex: (.*);admin-http$
  16.       - source_labels: [__meta_kubernetes_pod_container_name]
  17.         action: replace
  18.         target_label: component
  20.     - job_name: 'linkerd-service-mirror'
  22.       scrape_interval: 10s
  23.       scrape_timeout: 10s
  25.       kubernetes_sd_configs:
  26.       - role: pod
  27.       relabel_configs:
  28.       - source_labels:
  29.         - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
  30.         - __meta_kubernetes_pod_container_port_name
  31.         action: keep
  32.         regex: linkerd-service-mirror;admin-http$
  33.       - source_labels: [__meta_kubernetes_pod_container_name]
  34.         action: replace
  35.         target_label: component
  37.     - job_name: 'linkerd-proxy'
  39.       scrape_interval: 10s
  40.       scrape_timeout: 10s
  42.       kubernetes_sd_configs:
  43.       - role: pod
  44.       relabel_configs:
  45.       - source_labels:
  46.         - __meta_kubernetes_pod_container_name
  47.         - __meta_kubernetes_pod_container_port_name
  48.         - __meta_kubernetes_pod_label_linkerd_io_control_plane_ns
  49.         action: keep
  50.         regex: ^{{default .Values.global.proxyContainerName "linkerd-proxy" .Values.global.proxyContainerName}};linkerd-admin;{{.Values.global.namespace}}$
  51.       - source_labels: [__meta_kubernetes_namespace]
  52.         action: replace
  53.         target_label: namespace
  54.       - source_labels: [__meta_kubernetes_pod_name]
  55.         action: replace
  56.         target_label: pod
  57.       # special case k8s' "job" label, to not interfere with prometheus' "job"
  58.       # label
  59.       # __meta_kubernetes_pod_label_linkerd_io_proxy_job=foo =>
  60.       # k8s_job=foo
  61.       - source_labels: [__meta_kubernetes_pod_label_linkerd_io_proxy_job]
  62.         action: replace
  63.         target_label: k8s_job
  64.       # drop __meta_kubernetes_pod_label_linkerd_io_proxy_job
  65.       - action: labeldrop
  66.         regex: __meta_kubernetes_pod_label_linkerd_io_proxy_job
  67.       # __meta_kubernetes_pod_label_linkerd_io_proxy_deployment=foo =>
  68.       # deployment=foo
  69.       - action: labelmap
  70.         regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
  71.       # drop all labels that we just made copies of in the previous labelmap
  72.       - action: labeldrop
  73.         regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
  74.       # __meta_kubernetes_pod_label_linkerd_io_foo=bar =>
  75.       # foo=bar
  76.       - action: labelmap
  77.         regex: __meta_kubernetes_pod_label_linkerd_io_(.+)
  78.       # Copy all pod labels to tmp labels
  79.       - action: labelmap
  80.         regex: __meta_kubernetes_pod_label_(.+)
  81.         replacement: __tmp_pod_label_$1
  82.       # Take `linkerd_io_` prefixed labels and copy them without the prefix
  83.       - action: labelmap
  84.         regex: __tmp_pod_label_linkerd_io_(.+)
  85.         replacement:  __tmp_pod_label_$1
  86.       # Drop the `linkerd_io_` originals
  87.       - action: labeldrop
  88.         regex: __tmp_pod_label_linkerd_io_(.+)
  89.       # Copy tmp labels into real labels
  90.       - action: labelmap
  91.         regex: __tmp_pod_label_(.+)

The running configuration of the builtin prometheus can be used as a reference.

  1. kubectl -n linkerd  get configmap linkerd-prometheus-config -o yaml

Control Plane Components Configuration

Linkerd’s control plane components like public-api, etc depend on the Prometheus instance to power the dashboard and CLI.

The global.prometheusUrl field gives you a single place through which all these components can be configured to an external Prometheus URL. This is allowed both through the CLI and Helm.

CLI

This can be done by passing a file with the above field to the config flag, which is available both through linkerd install and linkerd upgrade commands

  1. global:
  2.   prometheusUrl: existing-prometheus.xyz:9090

Once applied, this configuration is persistent across upgrades, without having the user passing it again. The same can be overwritten as needed.

When using an external Prometheus and configuring the global.prometheusUrl field, Linkerd’s Prometheus will still be included in installation.

If you wish to disable this included Prometheus, be sure to include the following configuration as well:

  1. prometheus:
  2.   enabled: false

Helm

The same configuration can be applied through values.yaml when using Helm. Once applied, Helm makes sure that the configuration is persistent across upgrades.

More information on installation through Helm can be found here