MeshTrace (beta)

This policy uses new policy matching algorithm and is in beta state, it should not be mixed with TrafficTrace.

This policy enables publishing traces to a third party tracing solution.

Tracing is supported over HTTP, HTTP2, and gRPC protocols. You must explicitly specify the protocol for each service and data plane proxy you want to enable tracing for.

Kuma currently supports the following trace exposition formats:

Services still need to be instrumented to preserve the trace chain across requests made across different services.

You can instrument with a language library of your choice (for zipkin and for datadog). For HTTP you can also manually forward the following headers:

  • x-request-id
  • x-b3-traceid
  • x-b3-parentspanid
  • x-b3-spanid
  • x-b3-sampled
  • x-b3-flags

TargetRef support matrix

TargetRef typetop leveltofrom
Mesh✅❌❌
MeshSubset✅❌❌
MeshService✅❌❌
MeshServiceSubset✅❌❌

To learn more about the information in this table, see the matching docs.

Configuration

Sampling

Most of the time setting only overall is sufficient. random and client are for advanced use cases.

You can configure sampling settings equivalent to Envoy’s:

The value is always a percentage and is between 0 and 100.

Example:

  1. sampling:
  2.   overall: 80
  3.   random: 60
  4.   client: 40

Tags

You can add tags to trace metadata by directly supplying the value (literal) or by taking it from a header (header).

Example:

  1. tags:
  2.   - name: team
  3.     literal: core
  4.   - name: env
  5.     header:
  6.       name: x-env
  7.       default: prod
  8.   - name: version
  9.     header:
  10.       name: x-version

If a value is missing for header, default is used. If default isn’t provided, then the tag won’t be added.

Backends

Datadog

You can configure a Datadog backend with a url and splitService.

Example:

  1. datadog:
  2.   url: http://my-agent:8080 # Required. The url to reach a running datadog agent
  3.   splitService: true # Default to false. If true, it will split inbound and outbound requests in different services in Datadog

The splitService property determines if Datadog service names should be split based on traffic direction and destination. For example, with splitService: true and a backend service that communicates with a couple of databases, you would get service names like backend_INBOUND, backend_OUTBOUND_db1, and backend_OUTBOUND_db2 in Datadog.

Zipkin

In most cases the only field you’ll want to set in url.

Example:

  1. zipkin:
  2.   url: http://jaeger-collector:9411/api/v2/spans # Required. The url to a zipkin collector to send traces to 
  3.   traceId128bit: false # Default to false which will expose a 64bits traceId. If true, the id of the trace is 128bits
  4.   apiVersion: httpJson # Default to httpJson. It can be httpJson, httpProto and is the version of the zipkin API
  5.   sharedSpanContext: false # Default to true. If true, the inbound and outbound traffic will share the same span.

OpenTelemetry

The only field you can set is endpoint.

Example:

  1. openTelemetry:
  2.   endpoint: otel-collector:4317 # Required. Address of OpenTelemetry collector

Examples

Zipkin

Simple example:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshTrace
  3. metadata:
  4.   name: default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if unset
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     backends:
  13.       - type: Zipkin
  14.         zipkin:
  15.           url: http://jaeger-collector.mesh-observability:9411/api/v2/spans

Full example:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshTrace
  3. metadata:
  4.   name: default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if unset
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     backends:
  13.       - type: Zipkin
  14.         zipkin:
  15.           url: http://jaeger-collector.mesh-observability:9411/api/v2/spans
  16.           apiVersion: httpJson
  17.     tags:
  18.       - name: team
  19.         literal: core
  20.       - name: env
  21.         header:
  22.           name: x-env
  23.           default: prod
  24.       - name: version
  25.         header:
  26.           name: x-version
  27.     sampling:
  28.       overall: 80
  29.       random: 60
  30.       client: 40

Apply the configuration with kubectl apply -f [..].

Simple example:

  1. type: MeshTrace
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     backends:
  9.       - type: Zipkin
  10.         zipkin:
  11.           url: http://jaeger-collector:9411/api/v2/spans

Full example:

  1. type: MeshTrace
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     backends:
  9.       - type: Zipkin
  10.         zipkin:
  11.           url: http://jaeger-collector:9411/api/v2/spans
  12.           apiVersion: httpJson
  13.     tags:
  14.       - name: team
  15.         literal: core
  16.       - name: env
  17.         header:
  18.           name: x-env
  19.           default: prod
  20.       - name: version
  21.         header:
  22.           name: x-version
  23.     sampling:
  24.       overall: 80
  25.       random: 60
  26.       client: 40

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Datadog

This assumes a Datadog agent is configured and running. If you haven’t already check the Datadog observability page.

Simple Example:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshTrace
  3. metadata:
  4.   name: default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if unset
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     backends:
  13.       - type: Datadog
  14.         datadog:
  15.           url: http://trace-svc.default.svc.cluster.local:8126

Full Example:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshTrace
  3. metadata:
  4.   name: default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if unset
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     backends:
  13.       - type: Datadog
  14.         datadog:
  15.           url: http://trace-svc.default.svc.cluster.local:8126
  16.           splitService: true
  17.     tags:
  18.       - name: team
  19.         literal: core
  20.       - name: env
  21.         header:
  22.           name: x-env
  23.           default: prod
  24.       - name: version
  25.         header:
  26.           name: x-version
  27.     sampling:
  28.       overall: 80
  29.       random: 60
  30.       client: 40

where trace-svc is the name of the Kubernetes Service you specified when you configured the Datadog APM agent.

Apply the configuration with kubectl apply -f [..].

Simple example:

  1. type: MeshTrace
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     backends:
  9.       - type: Datadog
  10.         datadog:
  11.           url: http://127.0.0.1:8126

Full example:

  1. type: MeshTrace
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     backends:
  9.       - type: Datadog
  10.         datadog:
  11.           url: http://127.0.0.1:8126
  12.           splitService: true
  13.     tags:
  14.       - name: team
  15.         literal: core
  16.       - name: env
  17.         header:
  18.           name: x-env
  19.           default: prod
  20.       - name: version
  21.         header:
  22.           name: x-version
  23.     sampling:
  24.       overall: 80
  25.       random: 60
  26.       client: 40

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

OpenTelemetry

This assumes a OpenTelemetry collector is configured and running. If you haven’t already check the OpenTelementry operator.

Simple Example:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshTrace
  3. metadata:
  4.   name: default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if unset
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     backends:
  13.       - type: OpenTelemetry
  14.         openTelemetry:
  15.           endpoint: otel-collector:4317

Full example:

  1. type: MeshTrace
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     backends:
  9.       - type: OpenTelemetry
  10.         openTelemetry:
  11.           endpoint: otel-collector:4317
  12.     tags:
  13.       - name: team
  14.         literal: core
  15.       - name: env
  16.         header:
  17.           name: x-env
  18.           default: prod
  19.       - name: version
  20.         header:
  21.           name: x-version
  22.     sampling:
  23.       overall: 80
  24.       random: 60
  25.       client: 40

where otel-collector is the name of the Kubernetes Service for OTel exporter.

Apply the configuration with kubectl apply -f [..].

Simple Example:

  1. type: MeshTrace
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     backends:
  9.       - type: OpenTelemetry
  10.         openTelemetry:
  11.           endpoint: my-otel-collector.com:4317

Full example:

  1. type: MeshTrace
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     backends:
  9.       - type: OpenTelemetry
  10.         openTelemetry:
  11.           endpoint: my-otel-collector.com:4317
  12.     tags:
  13.       - name: team
  14.         literal: core
  15.       - name: env
  16.         header:
  17.           name: x-env
  18.           default: prod
  19.       - name: version
  20.         header:
  21.           name: x-version
  22.     sampling:
  23.       overall: 80
  24.       random: 60
  25.       client: 40

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Targeting parts of the infrastructure

While usually you want all the traces to be sent to the same tracing backend, you can target parts of a Mesh by using a finer-grained targetRef and a designated backend to trace different paths of our service traffic. This is especially useful when you want traces to never leave a world region, or a cloud, for example.

In this example, we have two zones east and west, each of these with their own Zipkin collector: east.zipkincollector:9411/api/v2/spans and west.zipkincollector:9411/api/v2/spans. We want dataplane proxies in each zone to only send traces to their local collector.

To do this, we use a TargetRef kind value of MeshSubset to filter which dataplane proxy a policy applies to.

West only policy:

  1. type: MeshTrace
  2. name: trace-west
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     tags:
  8.       kuma.io/zome: west
  9.   default:
  10.     backends:
  11.       - type: Zipkin
  12.         zipkin:
  13.           url: http://west.zipkincollector:9411/api/v2/spans

East only policy:

  1. type: MeshTrace
  2. name: trace-east
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     tags:
  8.       kuma.io/zome: east
  9.   default:
  10.     backends:
  11.       - type: Zipkin
  12.         zipkin:
  13.           url: http://east.zipkincollector:9411/api/v2/spans

West only policy:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshTrace
  3. metadata:
  4.     name: trace-west
  5.     namespace: kuma-system
  6. spec:
  7.   targetRef:
  8.     kind: MeshSubset
  9.     tags:
  10.       kuma.io/zome: west
  11.   default:
  12.     backends:
  13.       - type: Zipkin
  14.         zipkin:
  15.           url: http://west.zipkincollector:9411/api/v2/spans

East only policy:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshTrace
  3. metadata:
  4.   name: trace-east
  5.   namespace: kuma-system
  6. spec:
  7.   targetRef:
  8.     kind: MeshSubset
  9.     tags:
  10.       kuma.io/zome: east
  11.   default:
  12.     backends:
  13.       - type: Zipkin
  14.         zipkin:
  15.           url: http://east.zipkincollector:9411/api/v2/spans

All policy options