Traffic Trace

This policy enables tracing logging to a third party tracing solution. Tracing is supported on any HTTP traffic in a Mesh, and will only work with data planes and services that have the Kuma protocol: http tag defined.

In order to enable tracing there are two steps that have to be taken:

On Kubernetes we can run kumactl install tracing | kubectl apply -f - to deploy Jaeger automatically in a kuma-tracing namespace.

Add a tracing backend

A tracing backend must be first specified in a Mesh resource. Once added, the tracing backend will be available for use in the TrafficTrace resource.

While most commonly we want all the traces to be sent to the same tracing backend, we can optionally create multiple tracing backends in a Mesh resource and store traces for different paths of our service traffic in different backends (by leveraging Kuma Tags). This is especially useful when we want traces to never leave a world region, or a cloud, among other examples.

The types supported are:

  • zipkin. You can also use this with JaegerTraffic Trace - 图1 since it supports Zipkin compatible traces.

To add a new tracing backend we must create a new tracing property in a Mesh resource:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   namespace: default
  5.   name: default
  6. spec:
  7.   tracing:
  8.     defaultBackend: jaeger-collector
  9.     backends:
  10.     - name: jaeger-collector
  11.       type: zipkin
  12.       sampling: 100.0
  13.       conf:
  14.         url: http://jaeger-collector.kuma-tracing:9411/api/v2/spans

We will apply the configuration with kubectl apply -f [..].

  1. type: Mesh
  2. name: default
  3. tracing:
  4.   defaultBackend: jaeger-collector
  5.   backends:
  6.   - name: jaeger-collector
  7.     type: zipkin
  8.     sampling: 100.0
  9.     conf:
  10.       url: http://jaeger-collector.kuma-tracing:9411/api/v2/spans

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

We can also specify a defaultBackend property that will be used if any TrafficTrace resource doesn’t explictly specify a tracing backend.

Add a TrafficTrace resource

Once we have added a tracing backend, we can now create TrafficTrace resources that will determine how we are going to collecting traces, and what backend we should be using to store them.

  1. apiVersion: kuma.io/v1alpha1
  2. kind: TrafficTrace
  3. mesh: default
  4. metadata:
  5.   name: trace-all-traffic
  6. spec:
  7.   selectors:
  8.   - match:
  9.       service: '*'
  10.   conf:
  11.     backend: jaeger-collector

We will apply the configuration with kubectl apply -f [..].

  1. type: TrafficTrace
  2. name: trace-all-traffic
  3. mesh: default
  4. selectors:
  5. - match:
  6.     service: '*'
  7. conf:
  8.   backend: jager-collector

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

We can use Kuma Tags to apply the TrafficTrace resource in a more target way to a subset of data plane proxies as opposed to all of them (like we do in the example by using the service: '*' selector),

It is important that we instrument our services to preserve the trace chain between requests that are made across different services. We can either use a library in the language of our choice, or we can manually pass the following headers:

  • x-request-id
  • x-b3-traceid
  • x-b3-parentspanid
  • x-b3-spanid
  • x-b3-sampled
  • x-b3-flags

As noted before, Envoy’s Zipkin tracer is also compatible with Jaeger through Zipkin V1 HTTP API.Traffic Trace - 图2.

On the current version of Kuma, because of a limitation of Envoy, every time we create, update or delete a TrafficTrace resource we need to restart the data plane proxies in order to successfully apply the tracing configuration.

A fix has already been merged in Envoy and we are waiting for its new release (most likely v1.15) before we can update Kuma to remove this limitation.