Configure tracing with Telemetry API
Istio provides the ability to configure advanced tracing options, such as sampling rate and adding custom tags to reported spans. This task shows you how to customize the tracing options with Telemetry API.
Before you begin
Ensure that your applications propagate tracing headers as described here.
Follow the tracing installation guide located under Integrations based on your preferred tracing backend to install the appropriate addon and configure your Istio proxies to send traces to the tracing deployment.
Installation
In this example, we will send tracing to zipkin so make sure it is installed:
$ cat <<EOF > ./tracing.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
defaultConfig:
tracing: {} # disabled MeshConfig tracing options
extensionProviders:
# add zipkin provider
- name: zipkin
zipkin:
service: zipkin.istio-system.svc.cluster.local
port: 9411
EOF
$ istioctl install -f ./tracing.yaml --skip-confirmation
Enable tracing for mesh
Enable tracing by applying the following configuration:
$ kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
tracing:
- providers:
- name: "zipkin"
EOF
Customizing Trace sampling
The sampling rate option can be used to control what percentage of requests get reported to your tracing system. This should be configured based upon your traffic in the mesh and the amount of tracing data you want to collect. The default rate is 1%.
$ kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
tracing:
- providers:
- name: "zipkin"
randomSamplingPercentage: 100.00
EOF
Customizing tracing tags
Custom tags can be added to spans based on literals, environmental variables and client request headers in order to provide additional information in spans specific to your environment.
There is no limit on the number of custom tags that you can add, but tag names must be unique.
You can customize the tags using any of the three supported options below.
Literal represents a static value that gets added to each span.
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
tracing:
- providers:
- name: "zipkin"
randomSamplingPercentage: 100.00
customTags:
"provider":
literal:
value: "zipkin"
Environmental variables can be used where the value of the custom tag is populated from a workload proxy environment variable.
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
tracing:
- providers:
- name: "zipkin"
randomSamplingPercentage: 100.00
customTags:
"cluster_id":
environment:
name: ISTIO_META_CLUSTER_ID
defaultValue: Kubernetes # optional
In order to add custom tags based on environmental variables, you must modify the
istio-sidecar-injector
ConfigMap in your root Istio system namespace.Client request header option can be used to populate tag value from an incoming client request header.
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
tracing:
- providers:
- name: "zipkin"
randomSamplingPercentage: 100.00
custom_tags:
my_tag_header:
header:
name: <CLIENT-HEADER>
defaultValue: <VALUE> # optional
Customizing tracing tag length
By default, the maximum length for the request path included as part of the HttpUrl
span tag is 256. To modify this maximum length, add the following to your tracing.yaml
file.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
defaultConfig:
tracing: {} # disabled tracing options via `MeshConfig`
extensionProviders:
# add zipkin provider
- name: zipkin
zipkin:
service: zipkin.istio-system.svc.cluster.local
port: 9411
maxTagLength: <VALUE>
Verify the results
You can verify the results with Zipkin UI.