opentelemetry

Description

The opentelemetry Plugin can be used to report tracing data according to the OpenTelemetry specification.

The Plugin only supports binary-encoded OLTP over HTTP.

Attributes

NameTypeRequiredDefaultValid valuesDescription
samplerobjectFalseSampling configuration.
sampler.namestringFalsealways_off[“always_on”, “always_off”, “trace_id_ratio”, “parent_base”]Sampling strategy. always_on: always samples, always_off: never samples, trace_id_ratio: random sampling result based on given sampling probability, parent_base: use parent decision if given, else determined by the root sampler.
sampler.optionsobjectFalse{fraction = 0, root = {name = “always_off”}}Parameters for sampling strategy.
sampler.options.fractionnumberFalse0[0, 1]Sampling probability for trace_id_ratio.
sampler.options.rootobjectFalse{name = “always_off”, options = {fraction = 0}}Root sampler for parent_base strategy.
sampler.options.root.namestringFalsealways_off[“always_on”, “always_off”, “trace_id_ratio”]Root sampling strategy.
sampler.options.root.optionsobjectFalse{fraction = 0}Root sampling strategy parameters.
sampler.options.root.options.fractionnumberFalse0[0, 1]Root sampling probability for trace_id_ratio.
additional_attributesarray[string]FalseAdditional attributes appended to the trace span. Support built-in NGINX or APISIX variables in values, such as http_header or route_id.
additional_header_prefix_attributesarray[string]FalseHeaders or header prefixes appended to the trace span’s attributes. For example, use x-my-header” or x-my-headers-* to include all headers with the prefix x-my-headers-.

Configuring the collector

You can set up the collector by configuring it in you configuration file (conf/config.yaml):

NameTypeDefaultDescription
trace_id_sourceenumx-request-idSource of the trace ID. Valid values are random or x-request-id. When set to x-request-id, the value of the x-request-id header will be used as trace ID. Make sure that it matches the regex pattern [0-9a-f]{32}.
resourceobjectAdditional resource appended to the trace.
collectorobject{address = “127.0.0.1:4318”, request_timeout = 3}OpenTelemetry Collector configuration.
collector.addressstring127.0.0.1:4318Collector address. If the collector serves on https, use https://127.0.0.1:4318 as the address.
collector.request_timeoutinteger3Report request timeout in seconds.
collector.request_headersobjectReport request HTTP headers.
batch_span_processorobjectTrace span processor.
batch_span_processor.drop_on_queue_fullbooleanfalseWhen set to true, drops the span when queue is full. Otherwise, force process batches.
batch_span_processor.max_queue_sizeinteger1024Maximum queue size for buffering spans for delayed processing.
batch_span_processor.batch_timeoutnumber2Maximum time in seconds for constructing a batch.
batch_span_processor.max_export_batch_sizeinteger16Maximum number of spans to process in a single batch.
batch_span_processor.inactive_timeoutnumber1Time interval in seconds between processing batches.
opentelemetry - 图1note

If you find a bad argument #1 to '?' (invalid value) error triggered by the hex2bytes function in error log, it’s essential to verify if your traceId matches the specified regex pattern [0-9a-f]{32}, as required by opentelemetry’s traceId format.

For instance, a possible scenario occurs when the plugin attribute trace_id_source is configured as x-request-id, and requests include an x-request-id header generated by Envoy. Envoy typically uses a UUID to create this header by default. When the opentelemetry plugin adopts this UUID as the traceId, the presence of hyphens in the UUID can cause issues. Since the UUID format with hyphens does not comply with the expected traceId format, it results in errors when attempting to push traces to the collector.

You can configure these as shown below in your configuration file (conf/config.yaml):

conf/config.yaml

  1. plugin_attr:
  2.   opentelemetry:
  3.     resource:
  4.       service.name: APISIX
  5.       tenant.id: business_id
  6.     collector:
  7.       address: 192.168.8.211:4318
  8.       request_timeout: 3
  9.       request_headers:
  10.         foo: bar
  11.     batch_span_processor:
  12.       drop_on_queue_full: false
  13.       max_queue_size: 6
  14.       batch_timeout: 2
  15.       inactive_timeout: 1
  16.       max_export_batch_size: 2

Variables

The following nginx variables are set by OpenTelemetry:

  • opentelemetry_context_traceparent - W3C trace context, e.g.: 00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01
  • opentelemetry_trace_id - Trace Id of the current span
  • opentelemetry_span_id - Span Id of the current span

How to use variables? you have to add it to your configuration file (conf/config.yaml):

conf/config.yaml

  1. http:
  2.     enable_access_log: true
  3.     access_log: "/dev/stdout"
  4.     access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr","uri": "$uri"}'
  5.     access_log_format_escape: json
  6. plugins:
  7.   - opentelemetry
  8. plugin_attr:
  9.   opentelemetry:
  10.     set_ngx_var: true

Enable Plugin

To enable the Plugin, you have to add it to your configuration file (conf/config.yaml):

conf/config.yaml

  1. plugins:
  2.   - ...
  3.   - opentelemetry

Now, you can enable the Plugin on a specific Route:

opentelemetry - 图2note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

  1. admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
  1. curl http://127.0.0.1:9180/apisix/admin/routes/1  -H "X-API-KEY: $admin_key" -X PUT -d '
  2. {
  3.     "methods": ["GET"],
  4.     "uris": [
  5.         "/uid/*"
  6.     ],
  7.     "plugins": {
  8.         "opentelemetry": {
  9.             "sampler": {
  10.                 "name": "always_on"
  11.             }
  12.         }
  13.     },
  14.     "upstream": {
  15.         "type": "roundrobin",
  16.         "nodes": {
  17.             "127.0.0.1:1980": 1
  18.         }
  19.     }
  20. }'

Delete Plugin

To remove the opentelemetry Plugin, you can delete the corresponding JSON configuration from the Plugin configuration. APISIX will automatically reload and you do not have to restart for this to take effect.

  1. curl http://127.0.0.1:9180/apisix/admin/routes/1  -H "X-API-KEY: $admin_key" -X PUT -d '
  2. {
  3.     "methods": ["GET"],
  4.     "uris": [
  5.         "/uid/*"
  6.     ],
  7.     "plugins": {
  8.     },
  9.     "upstream": {
  10.         "type": "roundrobin",
  11.         "nodes": {
  12.             "127.0.0.1:1980": 1
  13.         }
  14.     }
  15. }'