Collect metrics with OpenTelemetry

Kuma provides integration with OpenTelemetry. You can collect and push dataplane proxy and application metrics to OpenTelemetry collector. Which opens up lots of possibilities of processing and exporting metrics to multiple ecosystems like Datadog, Grafana cloud, Honeycomb and more.

Prerequisites

  • Completed quickstart to set up a zone control plane with demo application

Install Kuma observability stack

To start we need to install Kuma observability stack which is build on top of Prometheus and Grafana.

  1. kumactl install observability | kubectl apply -f-

We will use it to scrape metrics from OpenTelemetry collector and visualise them on Kuma dashboards.

Since quickstart guide have really restrictive MeshTrafficPermissions we need to allow traffic in mesh-observability namespace:

  1. echo "apiVersion: kuma.io/v1alpha1
  2. kind: MeshTrafficPermission
  3. metadata:
  4.   namespace: mesh-observability
  5.   name: allow-observability
  6. spec:
  7.   from:
  8.     - targetRef:
  9.         kind: Mesh
  10.       default:
  11.         action: Allow" | kubectl apply -f -

Install OpenTelemetry collector

First we need an OpenTelemetry collector configuration. Save it by running:

  1. echo "
  2. mode: deployment
  3. config:
  4.   exporters:
  5.     prometheus:
  6.       endpoint: \${env:MY_POD_IP}:8889
  7.   extensions:
  8.     health_check:
  9.       endpoint: \${env:MY_POD_IP}:13133
  10.   processors:
  11.     batch: {}
  12.   receivers:
  13.     otlp:
  14.       protocols:
  15.         grpc:
  16.           endpoint: \${env:MY_POD_IP}:4317
  17.   service:
  18.     extensions:
  19.       - health_check
  20.     pipelines:
  21.       metrics:
  22.         receivers: [otlp]
  23.         exporters: [prometheus]
  24.         processors: [batch]
  25. ports:
  26.   otlp:
  27.     enabled: true
  28.     containerPort: 4317
  29.     servicePort: 4317
  30.     hostPort: 4317
  31.     protocol: TCP
  32.     appProtocol: grpc
  33.   prometheus:
  34.     enabled: true
  35.     containerPort: 8889
  36.     servicePort: 8889
  37.     protocol: TCP
  38. image:
  39.   repository: 'otel/opentelemetry-collector-contrib'
  40. resources:
  41.   limits:
  42.     cpu: 250m
  43.     memory: 512Mi
  44. " > values-otel.yaml

This is the Helm chart configuration we will be using. This will configure OpenTelemetry collector to listen on grpc port 4317 for metrics pushed by dataplane proxy, process and expose collected metrics in Prometheus format on port 8889. In the next step we will configure Prometheus to scrape these metrics. Our configuration relies on the contrib distribution of opentelemetry-collector so we set this in the values.

Most important in this configuration is pipelines section:

  1. pipelines:
  2.   metrics:
  3.     receivers: [otlp]
  4.     exporters: [prometheus]

In this basic guide we will focus only on collecting metrics, but this can be also easily configured to collect traces and logs. We use otlp receiver to accept metrics pushed from dataplane proxies.

Then we have basic recommended processors to limit memory usage and to process metrics in batch. You can filter, modify and do more with available processors.

Last part is exporters section. You can export metrics to multiple destination like Prometheus, Datadog, Grafana Cloud and more. Full list of available exporters can be found here. We will use Prometheus exporter for now.

With configuration in place we can install OpenTelemetry collector:

  1. helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
  2. helm install --namespace mesh-observability opentelemetry-collector open-telemetry/opentelemetry-collector -f values-otel.yaml

Configure Prometheus to scrape metrics from OpenTelemetry collector

We need to update prometheus-server ConfigMap and add scrape_configs entry:

  1. - job_name: "opentelemetry-collector"
  2.   scrape_interval: 15s
  3.   static_configs:
  4.     - targets: ["opentelemetry-collector.mesh-observability.svc:8889"]

Prometheus will automatically pick up this config and start scraping OpenTelemetry collector. To check if config was applied properly you can go to Prometheus GUI:

  1. kubectl port-forward svc/prometheus-server -n mesh-observability 9090:80

Now go to http://localhost:9090/targets. You should see new target opentelemetry-collector:

Prometheus OpenTelemetry source

Enable OpenTelemetry metrics and check results

By now we have installed and configured all needed observability tools: OpenTelemetry collector, Prometheus and Grafana. We can now apply MeshMetric policy:

  1. echo 'apiVersion: kuma.io/v1alpha1
  2. kind: MeshMetric
  3. metadata:
  4.   name: otel-metrics
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   default:
  10.     backends:
  11.       - type: OpenTelemetry
  12.         openTelemetry:
  13.           endpoint: opentelemetry-collector.mesh-observability.svc:4317' | kubectl apply -f -

This policy will configure all dataplane proxies in default Mesh to collect and push metrics to OpenTelemetry collector.

To check results we need to log into Grafana. First enable port forward to Grafana GUI:

  1. kubectl port-forward svc/grafana -n mesh-observability 3000:80

Then navigate to browser http://localhost:3000 and check Dataplane dashboard. You should see something similar to:

Dataplane Grafana dashboard

Next steps