Collect metrics with OpenTelemetry

Kuma provides integration with OpenTelemetry. You can collect and push dataplane proxy and application metrics to OpenTelemetry collector. Which opens up lots of possibilities of processing and exporting metrics to multiple ecosystems like Datadog, Grafana cloud, Honeycomb and more.

Prerequisites

  • Completed quickstart to set up a zone control plane with demo application

Install Kuma observability stack

To start we need to install Kuma observability stack which is build on top of Prometheus and Grafana.

  1. kumactl install observability | kubectl apply -f-

We will use it to scrape metrics from OpenTelemetry collector and visualise them on Kuma dashboards.

Install OpenTelemetry collector

First we need an OpenTelemetry collector configuration. Save it by running:

  1. echo "
  2. mode: deployment
  3. config:
  4.   exporters:
  5.     prometheus:
  6.       endpoint: \${env:MY_POD_IP}:8889
  7.   extensions:
  8.     health_check:
  9.       endpoint: \${env:MY_POD_IP}:13133
  10.     memory_ballast: {}
  11.   processors:
  12.     batch: {}
  13.     memory_limiter: null
  14.   receivers:
  15.     otlp:
  16.       protocols:
  17.         grpc:
  18.           endpoint: \${env:MY_POD_IP}:4317
  19.   service:
  20.     extensions:
  21.       - health_check
  22.       - memory_ballast
  23.     pipelines:
  24.       metrics:
  25.         receivers: [otlp]
  26.         processors: [memory_limiter, batch]
  27.         exporters: [prometheus]
  28. ports:
  29.   otlp:
  30.     enabled: true
  31.     containerPort: 4317
  32.     servicePort: 4317
  33.     hostPort: 4317
  34.     protocol: TCP
  35.     appProtocol: grpc
  36.   prometheus:
  37.     enabled: true
  38.     containerPort: 8889
  39.     servicePort: 8889
  40.     protocol: TCP
  41. image:
  42.   repository: 'otel/opentelemetry-collector-contrib'
  43. " > values-otel.yaml

This is the Helm chart configuration we will be using. This will configure OpenTelemetry collector to listen on grpc port 4317 for metrics pushed by dataplane proxy, process and expose collected metrics in Prometheus format on port 8889. In the next step we will configure Prometheus to scrape these metrics. Our configuration relies on the contrib distribution of opentelemetry-collector so we set this in the values.

Most important in this configuration is pipelines section:

  1. pipelines:
  2.   metrics:
  3.     receivers: [otlp]
  4.     processors: [memory_limiter, batch]
  5.     exporters: [prometheus]

In this basic guide we will focus only on collecting metrics, but this can be also easily configured to collect traces and logs. We use otlp receiver to accept metrics pushed from dataplane proxies.

Then we have basic recommended processors to limit memory usage and to process metrics in batch. You can filter, modify and do more with available processors.

Last part is exporters section. You can export metrics to multiple destination like Prometheus, Datadog, Grafana Cloud and more. Full list of available exporters can be found here. We will use Prometheus exporter for now.

With configuration in place we can install OpenTelemetry collector:

  1. helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
  2. helm install --namespace mesh-observability opentelemetry-collector open-telemetry/opentelemetry-collector -f values-otel.yaml

Configure Prometheus to scrape metrics from OpenTelemetry collector

We need to update prometheus-server ConfigMap and add scrape_configs entry:

  1. - job_name: "opentelemetry-collector"
  2.   scrape_interval: 15s
  3.   static_configs:
  4.     - targets: ["opentelemetry-collector.mesh-observability.svc:8889"]

Prometheus will automatically pick up this config and start scraping OpenTelemetry collector. To check if config was applied properly you can go to Prometheus GUI:

  1. kubectl port-forward svc/prometheus-server -n mesh-observability 9090:80

Now go to http://localhost:9090/targets. You should see new target opentelemetry-collector:

Prometheus OpenTelemetry source

Enable OpenTelemetry metrics and check results

By now we have installed and configured all needed observability tools: OpenTelemetry collector, Prometheus and Grafana. We can now apply MeshMetric policy:

  1. echo '
  2. apiVersion: kuma.io/v1alpha1
  3. kind: MeshMetric
  4. metadata:
  5.   name: otel-metrics
  6.   namespace: kuma-system
  7.   labels:
  8.     kuma.io/mesh: default
  9. spec:
  10.   targetRef:
  11.     kind: Mesh
  12.   default:
  13.     backends:
  14.       - type: OpenTelemetry
  15.         openTelemetry:
  16.           endpoint: opentelemetry-collector.mesh-observability.svc:4317
  17. ' | kubectl apply -f -

This policy will configure all dataplane proxies in default Mesh to collect and push metrics to OpenTelemetry collector.

To check results we need to log into Grafana. First enable port forward to Grafana GUI:

  1. kubectl port-forward svc/grafana -n mesh-observability 3000:80

Then navigate to browser http://localhost:3000 and check Dataplane dashboard. You should see something similar to:

Dataplane Grafana dashboard

Next steps