MeshAccessLog

With the MeshAccessLog policy you can easily set up access logs on every data plane proxy in a mesh.

This policy uses a new policy matching algorithm. Do not combine with TrafficLog.

This guide assumes you have already configured your observability tools to work with Kuma. If you haven’t, see the observability docs.

targetRef support matrix

targetRefAllowed kinds
targetRef.kindMesh, MeshSubset, MeshService, MeshServiceSubset
to[].targetRef.kindMesh, MeshService
from[].targetRef.kindMesh
targetRefAllowed kinds
targetRef.kindMesh, MeshGateway, MeshGateway with tags
to[].targetRef.kindMesh, MeshService
from[].targetRef.kindMesh
targetRefAllowed kinds
targetRef.kindMesh, MeshSubset, MeshService, MeshServiceSubset
to[].targetRef.kindMesh, MeshService

To learn more about the information in this table, see the matching docs.

Configuration

Format

Kuma gives you full control over the format of the access logs.

The shape of a single log record is defined by a template string that uses command operators to extract and format data about a TCP connection or an HTTP request.

For example:

  1. %START_TIME% %KUMA_SOURCE_SERVICE% => %KUMA_DESTINATION_SERVICE% %DURATION%

%START_TIME% and %KUMA_SOURCE_SERVICE% are examples of available command operators.

All command operators defined by Envoy are supported, along with additional command operators defined by Kuma:

Command OperatorDescription
%KUMA_MESH%Name of the mesh in which traffic is flowing.
%KUMA_SOURCE_SERVICE%Name of a service that is the source of traffic.
%KUMA_DESTINATION_SERVICE%Name of a service that is the destination of traffic.
%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%Address of a Dataplane that is the source of traffic.
%KUMA_TRAFFIC_DIRECTION%Direction of the traffic, INBOUND, OUTBOUND, or UNSPECIFIED.

All additional access log command operators are valid to use with both TCP and HTTP traffic.

Internally, Kuma determines traffic protocol based on the value of kuma.io/protocol tag on the inbound interface of a destination Dataplane.

There are two types of format, plain and json.

Plain accepts a string with command operators and produces a string output.

JSON accepts a list of key-value pairs that produces a valid JSON object.

It is up to the user to decide which format type to use. Some system will automatically parse JSON logs and allow you to filter and query based on available keys.

If a command operator is specific to HTTP traffic, such as %REQ(X?Y):Z% or %RESP(X?Y):Z%, in the case of TCP traffic it will be replaced by a symbol “-” for plain and a null value for json. You can set the format.omitEmptyValues boolean option to change this to "" for plain and omit them entirely for json.

Plain

The default format string for TCP traffic is:

  1. [%START_TIME%] %RESPONSE_FLAGS% %KUMA_MESH% %KUMA_SOURCE_ADDRESS_WITHOUT_PORT%(%KUMA_SOURCE_SERVICE%)->%UPSTREAM_HOST%(%KUMA_DESTINATION_SERVICE%) took %DURATION%ms, sent %BYTES_SENT% bytes, received: %BYTES_RECEIVED% bytes

The default format string for HTTP traffic is:

  1. [%START_TIME%] %KUMA_MESH% "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%KUMA_SOURCE_SERVICE%" "%KUMA_DESTINATION_SERVICE%" "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%" "%UPSTREAM_HOST%"

Example configuration:

  1. format:
  2.   type: Plain
  3.   plain: '[%START_TIME%] %BYTES_RECEIVED%'

Example output:

  1. [2016-04-15T20:17:00.310Z] 154

JSON

Example configuration:

  1. format:
  2.   type: Json
  3.   json:
  4.     - key: "start_time"
  5.       value: "%START_TIME%"
  6.     - key: "bytes_received"
  7.       value: "%BYTES_RECEIVED%"

Example output:

  1. {
  2.   "start_time": "2016-04-15T20:17:00.310Z",
  3.   "bytes_received": "154"
  4. }

TCP configuration with default fields:

  1. format:
  2.   type: Json
  3.   json:
  4.     - key: "start_time"
  5.       value: "%START_TIME%"
  6.     - key: "response_flags"
  7.       value: "%RESPONSE_FLAGS%"
  8.     - key: "kuma_mesh"
  9.       value: "%KUMA_MESH%"
  10.     - key: "kuma_source_address_without_port"
  11.       value: "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%"
  12.     - key: "kuma_source_service"
  13.       value: "%KUMA_SOURCE_SERVICE%"
  14.     - key: "upstream_host"
  15.       value: "%UPSTREAM_HOST%"
  16.     - key: "kuma_destination_service"
  17.       value: "%KUMA_DESTINATION_SERVICE%"
  18.     - key: "duration_ms"
  19.       value: "%DURATION%"
  20.     - key: "bytes_sent"
  21.       value: "%BYTES_SENT%"
  22.     - key: "bytes_received"
  23.       value: "%BYTES_RECEIVED%"

HTTP configuration with default fields:

  1. format:
  2.   type: Json
  3.   json:
  4.     - key: "start_time"
  5.       value: "%START_TIME%"
  6.     - key: "kuma_mesh"
  7.       value: "%KUMA_MESH%"
  8.     - key: 'method'
  9.       value: '"%REQ(:METHOD)%'
  10.     - key: "path"
  11.       value: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
  12.     - key: 'protocol'
  13.       value: '%PROTOCOL%'
  14.     - key: "response_code"
  15.       value: "%RESPONSE_CODE%"
  16.     - key: "response_flags"
  17.       value: "%RESPONSE_FLAGS%"
  18.     - key: "bytes_received"
  19.       value: "%BYTES_RECEIVED%"
  20.     - key: "bytes_sent"
  21.       value: "%BYTES_SENT%"
  22.     - key: "duration_ms"
  23.       value: "%DURATION%"
  24.     - key: "upstream_service_time"
  25.       value: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
  26.     - key: 'x_forwarded_for'
  27.       value: '"%REQ(X-FORWARDED-FOR)%"'
  28.     - key: 'user_agent'
  29.       value: '"%REQ(USER-AGENT)%"'
  30.     - key: 'request_id'
  31.       value: '"%REQ(X-REQUEST-ID)%"'
  32.     - key: 'authority'
  33.       value: '"%REQ(:AUTHORITY)%"'
  34.     - key: "kuma_source_service"
  35.       value: "%KUMA_SOURCE_SERVICE%"
  36.     - key: "kuma_destination_service"
  37.       value: "%KUMA_DESTINATION_SERVICE%"
  38.     - key: "kuma_source_address_without_port"
  39.       value: "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%"
  40.     - key: "upstream_host"
  41.       value: "%UPSTREAM_HOST%"

Backends

A backend determines where the logs end up.

TCP

A TCP backend streams logs to a server via TCP protocol. You can configure a TCP backend with an address:

  1. backends:
  2.   - type: Tcp
  3.     tcp:
  4.       address: 127.0.0.1:5000

File

A file backend streams logs to a text file. You can configure a file backend with a path:

  1. backends:
  2.   - type: File
  3.     file:
  4.       path: /dev/stdout

OpenTelemetry

An OpenTelemetry (OTel) backend sends data to an OpenTelemetry server. You can configure an OpenTelemetry backend with an endpoint, attributes (which contain additional information about the log) and body (can be a string message, including multi-line, or it can be a structured data). Attributes and endpoints can use placeholders described in the format section.

  1. backends:
  2.   - type: OpenTelemetry
  3.     openTelemetry:
  4.       endpoint: otel-collector:4317
  5.       body:
  6.         kvlistValue:
  7.           values:
  8.             - key: "mesh"
  9.               value:
  10.                 stringValue: "%KUMA_MESH%"
  11.       attributes:
  12.         - key: "start_time"
  13.           value: "%START_TIME%"

Body

Body is of type any (defined here) and can be one of the following forms:

  1. body:
  2.   stringValue: "%KUMA_MESH%"
  1. body:
  2.   boolValue: true
  1. body:
  2.   intValue: 123
  1. body:
  2.   doubleValue: 1.2
  1. body:
  2.   bytesValue: aGVsbG8=
  1. body:
  2.   arrayValue:
  3.     values:
  4.       - stringValue: "%KUMA_MESH%"
  1. body:
  2.   kvlistValue:
  3.     values:
  4.       - key: "mesh"
  5.         value:
  6.           stringValue: "%KUMA_MESH%"

Examples

Log outgoing traffic from specific frontend version to a backend service

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshAccessLog
  3. metadata:
  4.   name: default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if it isn't configured
  8. spec:
  9.   targetRef:
  10.     kind: MeshServiceSubset
  11.     name: frontend
  12.     tags:
  13.       version: canary
  14.   to:
  15.     - targetRef:
  16.         kind: MeshService
  17.         name: backend
  18.       default:
  19.         backends:
  20.           - type: File
  21.             file:
  22.               path: /dev/stdout

Apply the configuration with kubectl apply -f [..].

  1. type: MeshAccessLog
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshServiceSubset
  7.     name: frontend
  8.     tags:
  9.       version: canary
  10.   to:
  11.     - targetRef:
  12.         kind: MeshService
  13.         name: backend
  14.       default:
  15.         backends:
  16.           - type: File
  17.             file:
  18.               path: /dev/stdout

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Logging to multiple backends

This configuration logs to three backends: TCP, file and OpenTelemetry.

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshAccessLog
  3. metadata:
  4.   name: default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if it isn't configured
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   from:
  12.     - targetRef:
  13.         kind: Mesh
  14.       default:
  15.         backends:
  16.           - type: Tcp
  17.             tcp:
  18.               address: 127.0.0.1:5000
  19.               format:
  20.                 type: Json
  21.                 json:
  22.                   - key: "start_time"
  23.                     value: "%START_TIME%"
  24.           - type: File
  25.             file:
  26.               path: /dev/stdout
  27.               format:
  28.                 type: Plain
  29.                 plain: '[%START_TIME%]'
  30.           - type: OpenTelemetry
  31.             openTelemetry:
  32.               endpoint: otel-collector:4317
  33.               body:
  34.                 kvlistValue:
  35.                 values:
  36.                   - key: "mesh"
  37.                     value:
  38.                       stringValue: "%KUMA_MESH%"
  39.               attributes:
  40.                 - key: "start_time"
  41.                   value: "%START_TIME%"

Apply the configuration with kubectl apply -f [..].

  1. type: MeshAccessLog
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   from:
  8.     - targetRef:
  9.         kind: Mesh
  10.       default:
  11.         backends:
  12.           - type: Tcp
  13.             tcp:
  14.               address: 127.0.0.1:5000
  15.               format:
  16.                 type: Json
  17.                 json:
  18.                   - key: "start_time"
  19.                     value: "%START_TIME%"
  20.           - type: File
  21.             file:
  22.               path: /dev/stdout
  23.               format:
  24.                 type: Plain
  25.                 plain: '[%START_TIME%]'
  26.           - type: OpenTelemetry
  27.             openTelemetry:
  28.               endpoint: otel-collector:4317
  29.               body:
  30.                 kvlistValue:
  31.                 values:
  32.                   - key: "mesh"
  33.                     value:
  34.                       stringValue: "%KUMA_MESH%"
  35.               attributes:
  36.                 - key: "start_time"
  37.                   value: "%START_TIME%"

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Log all incoming and outgoing traffic

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshAccessLog
  3. metadata:
  4.   name: default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if it isn't configured
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   from: # delete this section if you don't want to log incoming traffic
  12.     - targetRef:
  13.         kind: Mesh
  14.       default:
  15.         backends:
  16.           - type: File
  17.             file:
  18.               path: /dev/stdout
  19.   to: # delete this section if you don't want to log outgoing traffic
  20.     - targetRef:
  21.         kind: Mesh
  22.       default:
  23.         backends:
  24.           - type: File
  25.             file:
  26.               path: /dev/stdout

Apply the configuration with kubectl apply -f [..].

  1. type: MeshAccessLog
  2. name: default
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   from: # delete this section if you don't want to log incoming traffic
  8.     - targetRef:
  9.         kind: Mesh
  10.       default:
  11.         backends:
  12.           - type: File
  13.             file:
  14.               path: /dev/stdout
  15.   to: # delete this section if you don't want to log outgoing traffic
  16.     - targetRef:
  17.         kind: Mesh
  18.       default:
  19.         backends:
  20.           - type: File
  21.             file:
  22.               path: /dev/stdout

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Logging traffic going outside the Mesh

To target ExternalServices, use MeshService as the targetRef kind with name set to
the kuma.io/service value.

To target other non-mesh traffic, i.e. passthrough traffic, use Mesh as the targetRef kind. In this case, %KUMA_DESTINATION_SERVICE% is set to external.

Select a built-in gateway

You can select a built-in gateway using the kuma.io/service value. A current limitation is that traffic routed from a gateway to a service is logged by that gateway as having destination "*".

All policy options