Propagate Zipkin distributed tracing spans, and report spans to a Zipkin server.

Configuration Reference

This plugin is compatible with DB-less mode.

In DB-less mode, you configure Kong Gateway declaratively. Therefore, the Admin API is mostly read-only. The only tasks it can perform are all related to handling the declarative config, including:

  • Setting a target’s health status in the load balancer
  • Validating configurations against schemas
  • Uploading the declarative configuration using the /config endpoint

Enable the plugin on a service

Admin API

Kubernetes

Declarative (YAML)

Kong Manager

For example, configure this plugin on a service by making the following request:

  1. curl -X POST http://{HOST}:8001/services/{SERVICE}/plugins \
  2.     --data "name=zipkin"  \
  3.     --data "config.local_service_name=kong" \
  4.     --data "config.http_endpoint=http://your.zipkin.collector:9411/api/v2/spans" \
  5.     --data "config.sample_ratio=0.001" \
  6.     --data "config.include_credential=true" \
  7.     --data "config.traceid_byte_count=16" \
  8.     --data "config.header_type=preserve" \
  9.     --data "config.default_header_type=b3" \
  10.     --data "config.tags_header=Zipkin-Tags"

SERVICE is the id or name of the service that this plugin configuration will target.

First, create a KongPlugin resource:

  1. apiVersion: configuration.konghq.com/v1
  2. kind: KongPlugin
  3. metadata:
  4.   name: <zipkin-example>
  5. config: 
  6.   local_service_name: kong
  7.   http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  8.   sample_ratio: 0.001
  9.   include_credential: true
  10.   traceid_byte_count: 16
  11.   header_type: preserve
  12.   default_header_type: b3
  13.   tags_header: Zipkin-Tags
  14. plugin: zipkin

Next, apply the KongPlugin resource to a Service by annotating the Service as follows:

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: {SERVICE}
  5.   labels:
  6.     app: {SERVICE}
  7.   annotations:
  8.     konghq.com/plugins: <zipkin-example>
  9. spec:
  10.   ports:
  11.   - port: 80
  12.     targetPort: 80
  13.     protocol: TCP
  14.     name: {SERVICE}
  15.   selector:
  16.     app: {SERVICE}

{SERVICE} is the id or name of the service that this plugin configuration will target.

Note: The KongPlugin resource only needs to be defined once and can be applied to any service, consumer, or route in the namespace. If you want the plugin to be available cluster-wide, create the resource as a KongClusterPlugin instead of KongPlugin.

For example, configure this plugin on a service by adding this section to your declarative configuration file:

  1. plugins:
  2. - name: zipkin
  3.   service: {SERVICE}
  4.   config: 
  5.     local_service_name: kong
  6.     http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  7.     sample_ratio: 0.001
  8.     include_credential: true
  9.     traceid_byte_count: 16
  10.     header_type: preserve
  11.     default_header_type: b3
  12.     tags_header: Zipkin-Tags

SERVICE is the id or name of the service that this plugin configuration will target.

Configure this plugin on a service:

  1. In Kong Manager, select the workspace.
  2. From the Dashboard, scroll down to Services and click View for the service row.
  3. Scroll down to plugins and click Add Plugin.

  4. Find and select the Zipkin plugin.

    Note: If the plugin is greyed out, then it is not available for your product tier. See Kong Gateway tiers.

  5. If the option is available, select Scoped.

  6. Add the service name and ID to the Service field if it is not already prefilled.

  7. Enter the following parameters, updating the default or sample values as needed:

    • Config.Local Service Name: kong
    • Config.Include Credential: select checkbox
    • Config.Traceid Byte Count: 16
    • Config.Header Type: preserve
    • Config.Default Header Type: b3
    • Config.Tags Header: Zipkin-Tags
  8. Click Create.

Enable the plugin on a route

Admin API

Kubernetes

Declarative (YAML)

Kong Manager

For example, configure this plugin on a route with:

  1. $ curl -X POST http://{HOST}:8001/routes/{ROUTE}/plugins \
  2.     --data "name=zipkin"  \
  3.     --data "config.local_service_name=kong" \
  4.     --data "config.http_endpoint=http://your.zipkin.collector:9411/api/v2/spans" \
  5.     --data "config.sample_ratio=0.001" \
  6.     --data "config.include_credential=true" \
  7.     --data "config.traceid_byte_count=16" \
  8.     --data "config.header_type=preserve" \
  9.     --data "config.default_header_type=b3" \
  10.     --data "config.tags_header=Zipkin-Tags"

ROUTE is the id or name of the route that this plugin configuration will target.

First, create a KongPlugin resource:

  1. apiVersion: configuration.konghq.com/v1
  2. kind: KongPlugin
  3. metadata:
  4.   name: <zipkin-example>
  5. config: 
  6.   local_service_name: kong
  7.   http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  8.   sample_ratio: 0.001
  9.   include_credential: true
  10.   traceid_byte_count: 16
  11.   header_type: preserve
  12.   default_header_type: b3
  13.   tags_header: Zipkin-Tags
  14. plugin: zipkin

Then, apply it to an ingress (Route or Routes) by annotating the ingress as follows:

  1. apiVersion: networking/v1beta1
  2. kind: Ingress
  3. metadata:
  4.   name: {ROUTE}
  5.   annotations:
  6.     kubernetes.io/ingress.class: kong
  7.     konghq.com/plugins: <zipkin-example>
  8. spec:
  9.   rules:
  10.   - host: examplehostname.com
  11.     http:
  12.       paths:
  13.       - path: /bar
  14.         backend:
  15.           serviceName: echo
  16.           servicePort: 80

ROUTE is the id or name of the route that this plugin configuration will target.

Note: The KongPlugin resource only needs to be defined once and can be applied to any service, consumer, or route in the namespace. If you want the plugin to be available cluster-wide, create the resource as a KongClusterPlugin instead of KongPlugin.

For example, configure this plugin on a route by adding this section to your declarative configuration file:

  1. plugins:
  2. - name: zipkin
  3.   route: <route>
  4.   config: 
  5.     local_service_name: kong
  6.     http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  7.     sample_ratio: 0.001
  8.     include_credential: true
  9.     traceid_byte_count: 16
  10.     header_type: preserve
  11.     default_header_type: b3
  12.     tags_header: Zipkin-Tags

ROUTE is the id or name of the route that this plugin configuration will target.

Configure this plugin on a route:

  1. In Kong Manager, select the workspace.
  2. From the Dashboard, select Routes in the left navigation.
  3. Click View for the route row.
  4. Scroll down to plugins and click Add Plugin.

  5. Find and select the Zipkin plugin.

    Note: If the plugin is greyed out, then it is not available for your product tier. See Kong Gateway tiers.

  6. If the option is available, select Scoped.

  7. Add the Route ID if it is not already prefilled.

  8. Enter the following parameters, updating the default or sample values as needed:

    • Config.Local Service Name: kong
    • Config.Include Credential: select checkbox
    • Config.Traceid Byte Count: 16
    • Config.Header Type: preserve
    • Config.Default Header Type: b3
    • Config.Tags Header: Zipkin-Tags
  9. Click Create.

Enabling the plugin on a consumer

Admin API

Kubernetes

Declarative (YAML)

Kong Manager

For example, configure this plugin on a consumer with:

  1. $ curl -X POST http://{HOST}:8001/consumers/{CONSUMER}/plugins \
  2.     --data "name=zipkin"  \
  3.     --data "config.local_service_name=kong" \
  4.     --data "config.http_endpoint=http://your.zipkin.collector:9411/api/v2/spans" \
  5.     --data "config.sample_ratio=0.001" \
  6.     --data "config.include_credential=true" \
  7.     --data "config.traceid_byte_count=16" \
  8.     --data "config.header_type=preserve" \
  9.     --data "config.default_header_type=b3" \
  10.     --data "config.tags_header=Zipkin-Tags"

CONSUMER is the id or username of the consumer that this plugin configuration will target.

You can combine consumer.id, service.id, or route.id in the same request, to further narrow the scope of the plugin.

First, create a KongPlugin resource:

  1. apiVersion: configuration.konghq.com/v1
  2. kind: KongPlugin
  3. metadata:
  4.   name: <zipkin-example>
  5. config: 
  6.   local_service_name: kong
  7.   http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  8.   sample_ratio: 0.001
  9.   include_credential: true
  10.   traceid_byte_count: 16
  11.   header_type: preserve
  12.   default_header_type: b3
  13.   tags_header: Zipkin-Tags
  14. plugin: zipkin

Then, apply it to a consumer by annotating the KongConsumer resource as follows:

  1. apiVersion: configuration.konghq.com/v1
  2. kind: KongConsumer
  3. metadata:
  4.   name: {CONSUMER}
  5.   annotations:
  6.     konghq.com/plugins: <zipkin-example>
  7.     kubernetes.io/ingress.class: kong

CONSUMER is the id or username of the consumer that this plugin configuration will target.

Note: The KongPlugin resource only needs to be defined once and can be applied to any Service, Consumer, or Route in the namespace. If you want the plugin to be available cluster-wide, create the resource as a KongClusterPlugin instead of KongPlugin.

For example, configure this plugin on a consumer by adding this section to your declarative configuration file:

  1. plugins:
  2. - name: zipkin
  3.   consumer: {CONSUMER}
  4.   config: 
  5.     local_service_name: kong
  6.     http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  7.     sample_ratio: 0.001
  8.     include_credential: true
  9.     traceid_byte_count: 16
  10.     header_type: preserve
  11.     default_header_type: b3
  12.     tags_header: Zipkin-Tags

CONSUMER is the id or username of the consumer that this plugin configuration will target.

Configure this plugin on a consumer:

  1. In Kong Manager, select the workspace.
  2. From the Dashboard, scroll down to Consumers and click View for the consumer row.
  3. Select the Plugins tab.
  4. Click Add Plugin.

  5. Find and select the Zipkin plugin.

    Note: If the plugin is greyed out, then it is not available for your product tier. See Kong Gateway tiers.

  6. If the option is available, select Global.

  7. Enter the following parameters, updating the default or sample values as needed:

    • Config.Local Service Name: kong
    • Config.Include Credential: select checkbox
    • Config.Traceid Byte Count: 16
    • Config.Header Type: preserve
    • Config.Default Header Type: b3
    • Config.Tags Header: Zipkin-Tags
  8. Click Create.

Enable the plugin globally

A plugin which is not associated to any service, route, or consumer is considered global, and will be run on every request. Read the Plugin Reference and the Plugin Precedence sections for more information.

Admin API

Kubernetes

Declarative (YAML)

Kong Manager

For example, configure this plugin globally with:

  1. $ curl -X POST http://{HOST}:8001/plugins/ \
  2.     --data "name=zipkin"  \
  3.     --data "config.local_service_name=kong" \
  4.     --data "config.http_endpoint=http://your.zipkin.collector:9411/api/v2/spans" \
  5.     --data "config.sample_ratio=0.001" \
  6.     --data "config.include_credential=true" \
  7.     --data "config.traceid_byte_count=16" \
  8.     --data "config.header_type=preserve" \
  9.     --data "config.default_header_type=b3" \
  10.     --data "config.tags_header=Zipkin-Tags"

Create a KongClusterPlugin resource and label it as global:

  1. apiVersion: configuration.konghq.com/v1
  2. kind: KongClusterPlugin
  3. metadata:
  4.   name: <global-zipkin>
  5.   annotations:
  6.     kubernetes.io/ingress.class: kong
  7.   labels:
  8.     global: \"true\"
  9. config: 
  10.   local_service_name: kong
  11.   http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  12.   sample_ratio: 0.001
  13.   include_credential: true
  14.   traceid_byte_count: 16
  15.   header_type: preserve
  16.   default_header_type: b3
  17.   tags_header: Zipkin-Tags
  18. plugin: zipkin

For example, configure this plugin using the plugins: entry in the declarative configuration file:

  1. plugins:
  2. - name: zipkin
  3.   config: 
  4.     local_service_name: kong
  5.     http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  6.     sample_ratio: 0.001
  7.     include_credential: true
  8.     traceid_byte_count: 16
  9.     header_type: preserve
  10.     default_header_type: b3
  11.     tags_header: Zipkin-Tags

Configure this plugin globally:

  1. In Kong Manager, select the workspace.
  2. From the Dashboard, select Plugins in the left navigation.
  3. Click New Plugin.

  4. Find and select the Zipkin plugin.

    Note: If the plugin is greyed out, then it is not available for your product tier. See Kong Gateway tiers.

  5. If the option is available, set the plugin scope to Global.

  6. Enter the following parameters, updating the default/sample values as needed:

    • Config.Local Service Name: kong
    • Config.Include Credential: select checkbox
    • Config.Traceid Byte Count: 16
    • Config.Header Type: preserve
    • Config.Default Header Type: b3
    • Config.Tags Header: Zipkin-Tags
  7. Click Create.

Parameters

Here’s a list of all the parameters which can be used in this plugin’s configuration:

Form ParameterDescription
name
required

Type: string		The name of the plugin, in this case zipkin.
service.id

Type: string		The ID of the Service the plugin targets.
route.id

Type: string		The ID of the Route the plugin targets.
consumer.id

Type: string		The ID of the Consumer the plugin targets.
enabled
required

Type: boolean

Default value: true		Whether this plugin will be applied.
config.local_service_name
required

Type: string

Default value: kong

The name of the service as displayed in Zipkin. Customize this name to tell your Kong Gateway services apart in Zipkin request traces.

config.http_endpoint
optional

Type: string

Default value:

The full HTTP(S) endpoint to which Zipkin spans should be sent by Kong. If not specified, the Zipkin plugin will only act as a tracing header generator/transmitter.

config.sample_ratio
optional

Type: number

Default value: 0.001

How often to sample requests that do not contain trace IDs. Set to 0 to turn sampling off, or to 1 to sample all requests. The value must be between zero (0) and one (1), inclusive.

config.include_credential
required

Type: boolean

Default value: true

Specify whether the credential of the currently authenticated consumer should be included in metadata sent to the Zipkin server.

config.traceid_byte_count
required

Type: integer

Default value: 16

The length in bytes of each request’s Trace ID. The value can be either 8 or 16.

config.header_type
required

Type: string

Default value: preserve

All HTTP requests going through the plugin are tagged with a tracing HTTP request. This property codifies what kind of tracing header the plugin expects on incoming requests.

Possible values: b3, b3-single, w3c, preserve, jaeger, ot, or ignore.

  • b3: Expects Zipkin’s B3 multiple headers on incoming requests, and will add them to the transmitted requests if the headers are missing from those requests.
  • b3-single: Expects or adds Zipkin’s B3 single-header tracing headers.
  • w3c: Expects or adds W3C’s traceparent tracing header.
  • preserve: Does not expect any format, and will transmit whatever header is recognized or present, with a default of b3 if none is found. In case of a mismatch between the expected and incoming tracing headers (for example, when header_type is set to b3 but a w3c-style tracing header is found in the incoming request), then the plugin will add both kinds of tracing headers to the request and generate a mismatch warning in the logs.
  • jaeger: Expects or adds Jaeger-style tracing headers (uber-trace-id).
  • ot: Expects or adds OpenTelemetry tracing headers of the form ot-tracer-*.
  • ignore: Does not read any tracing headers from the incoming request. Starts a new request using the default_header_type value, or falls back to b3 if there is no default_header_type value set.

config.default_header_type
required

Type: string

Default value: b3

Allows specifying the type of header to be added to requests with no pre-existing tracing headers and when config.header_type is set to “preserve”. When header_type is set to any other value, default_header_type is ignored.

Possible values are b3, b3-single, w3c, jaeger, or ot. See the entry for header_type for value definitions.

config.tags_header
required

Type: string

Default value: Zipkin-Tags

The Zipkin plugin will add extra headers to the tags associated with any HTTP requests that come with a header named as configured by this property. The format is name_of_tag=value_of_tag, separated by commas.

For example: with the default value, a request with the header Zipkin-Tags: fg=blue, bg=red will generate a trace with the tag fg with value blue, and another tag called bg with value red.

config.static_tags
optional

Type: array of string tags

Default value: []

The tags specified on this property will be added to the generated request traces. For example: [ { “name”: “color”, “value”: “red” } ].

How it Works

When enabled, this plugin traces requests in a way compatible with zipkin.

The code is structured around an opentracing core using the opentracing-lua library to collect timing data of a request in each of Kong’s phases. The plugin uses opentracing-lua compatible extractor, injector, and reporters to implement Zipkin’s protocols.

Reporter

An opentracing “reporter” is how tracing data is reported to another system. This plugin records tracing data for a given request, and sends it as a batch to a Zipkin server using the Zipkin v2 API. Note that zipkin version 1.31 or higher is required.

The http_endpoint configuration variable must contain the full uri including scheme, host, port and path sections (i.e. your uri likely ends in /api/v2/spans).

Spans

The plugin does request sampling. For each request which triggers the plugin, a random number between 0 and 1 is chosen.

If the number is greater than the configured sample_ratio, then a trace with several spans will be generated. If sample_ratio is set to 1, then all requests will generate a trace (this might be very noisy).

For each request that gets traced, the following spans are produced:

  • Request span: 1 per request. Encompasses the whole request in kong (kind: SERVER). The Proxy and Balancer spans are children of this span. It contains the following logs/annotations for the rewrite phase:

    • krs - kong.rewrite.start
    • krf - kong.rewrite.finish

    The Request span has the following tags:

    • lc: Hardcoded to kong.
    • kong.service: The uuid of the service matched when processing the request, if any.
    • kong.service_name: The name of the service matched when processing the request, if service exists and has a name attribute.
    • kong.route: The uuid of the route matched when processing the request, if any (it can be nil on non-matched requests).
    • kong.route_name: The name of the route matched when processing the request, if route exists and has a name attribute.
    • http.method: The HTTP method used on the original request (only for HTTP requests).
    • http.path: The path of the request (only for HTTP requests).
    • If the plugin tags_header config option is set, and the request contains headers with the appropriate name and correct encoding tags, then the trace will include the tags.
    • If the plugin static_tags config option is set, then the tags in the config option will be included in the trace.
  • Proxy span: 1 per request, encompassing most of Kong’s internal processing of a request (kind: CLIENT). Contains the following logs/annotations for the start/finish of the of the Kong plugin phases:
    • kas - kong.access.start
    • kaf - kong.access.finish
    • kbs - kong.body_filter.start
    • kbf - kong.body_filter.finish
    • khs - kong.header_filter.start
    • khf - kong.header_filter.finish
    • kps - kong.preread.start (only for stream requests)
    • kpf - kong.preread.finish (only for stream requests)
  • Balancer span(s): 0 or more per request, each encompassing one balancer attempt (kind: CLIENT). Contains the following tags specific to load balancing:
    • kong.balancer.try: A number indicating the attempt (1 for the first load-balancing attempt, 2 for the second, and so on).
    • peer.ipv4 or peer.ipv6 for the balancer IP.
    • peer.port for the balanced port.
    • error: Set to true if the balancing attempt was unsuccessful, otherwise unset.
    • http.status_code: The HTTP status code received, in case of error.
    • kong.balancer.state: An NGINX-specific description of the error, next/failed for HTTP failures, or 0 for stream failures. Equivalent to state_name in OpenResty’s balancer’s get_last_failure function.

See also

For more information, read the Kong blog post.

Changelog

1.5.x

  • Added a new parameter: local_service_name
  • Added a new ignore option for the header_type parameter