Traffic Log

With TrafficLog policy you can easily set up access logs on every data-plane in a Mesh.

The logs can be then forwarded to a collector that can further transmit them into systems like Splunk, ELK and Datadog.

Configuring access logs in Kuma is a 2-step process:

  1. First, you need to configure logging backends that will be available for use in a given Mesh.

    A logging backend is essentially a sink for access logs.

    In the current release of Kuma, a logging backend can be either a file or a TCP log collector, such as Logstash.

  2. Second, you need to create a TrafficLog policy to select a subset of traffic and forward its access logs into one of the logging backends configured for that Mesh.

On Universal

  1. type: Mesh
  2. name: default
  3. logging:
  4.   # TrafficLog policies may leave the `backend` field undefined.
  5.   # In that case the logs will be forwarded into the `defaultBackend` of that Mesh.
  6.   defaultBackend: file
  7.   # List of logging backends that can be referred to by name
  8.   # from TrafficLog policies of that Mesh.
  9.   backends:
  10.     - name: logstash
  11.       # Use `format` field to adjust the access log format to your use case.
  12.       format: '{"start_time": "%START_TIME%", "source": "%KUMA_SOURCE_SERVICE%", "destination": "%KUMA_DESTINATION_SERVICE%", "source_address": "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%", "destination_address": "%UPSTREAM_HOST%", "duration_millis": %DURATION%, "bytes_received": %BYTES_RECEIVED%, "bytes_sent": %BYTES_SENT%}'
  13.       # Use `tcp` field to co configure a TCP logging backend.
  14.       tcp:
  15.         # Address of a log collector.
  16.         address: 127.0.0.1:5000
  17.     - name: file
  18.       # Use `file` field to configure a file-based logging backend.
  19.       file:
  20.         path: /tmp/access.log
  21.       # When `format` field is omitted, the default access log format will be used.
  1. type: TrafficLog
  2. name: all-traffic
  3. mesh: default
  4. # This TrafficLog policy applies to all traffic in the Mesh.
  5. sources:
  6.   - match:
  7.       service: '*'
  8. destinations:
  9.   - match:
  10.       service: '*'
  11. # When `backend ` field is omitted, the logs will be forwarded into the `defaultBackend` of that Mesh.
  1. type: TrafficLog
  2. name: backend-to-database-traffic
  3. mesh: default
  4. # this TrafficLog policy applies only to traffic from service `backend` to service `database`.
  5. sources:
  6.   - match:
  7.       service: backend
  8. destinations:
  9.   - match:
  10.       service: database
  11. conf:
  12.   # Forward the logs into the logging backend named `logstash`.
  13.   backend: logstash

On Kubernetes

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   logging:
  7.     # TrafficLog policies may leave the `backend` field undefined.
  8.     # In that case the logs will be forwarded into the `defaultBackend` of that Mesh.
  9.     defaultBackend: file
  10.     # List of logging backends that can be referred to by name
  11.     # from TrafficLog policies of that Mesh.
  12.     backends:
  13.       - name: logstash
  14.         # Use `format` field to adjust the access log format to your use case.
  15.         format: '{"start_time": "%START_TIME%", "source": "%KUMA_SOURCE_SERVICE%", "destination": "%KUMA_DESTINATION_SERVICE%", "source_address": "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%", "destination_address": "%UPSTREAM_HOST%", "duration_millis": %DURATION%, "bytes_received": %BYTES_RECEIVED%, "bytes_sent": %BYTES_SENT%}'
  16.         # Use `tcp` field to co configure a TCP logging backend.
  17.         tcp:
  18.           # Address of a log collector.
  19.           address: 127.0.0.1:5000
  20.       - name: file
  21.         # Use `file` field to configure a file-based logging backend.
  22.         file:
  23.           path: /tmp/access.log
  24.         # When `format` field is omitted, the default access log format will be used.
  1. apiVersion: kuma.io/v1alpha1
  2. kind: TrafficLog
  3. metadata:
  4.   namespace: kuma-example
  5.   name: all-traffic
  6. mesh: default
  7. spec:
  8.   # This TrafficLog policy applies all traffic in that Mesh.
  9.   sources:
  10.     - match:
  11.         service: '*'
  12.   destinations:
  13.     - match:
  14.         service: '*'
  15.   # When `backend ` field is omitted, the logs will be forwarded into the `defaultBackend` of that Mesh.
  1. apiVersion: kuma.io/v1alpha1
  2. kind: TrafficLog
  3. metadata:
  4.   namespace: kuma-example
  5.   name: backend-to-database-traffic
  6. spec:
  7.   # This TrafficLog policy applies only to traffic from service `backend` to service `database`.
  8.   sources:
  9.     - match:
  10.         service: backend.kuma-example.svc:8080
  11.   destinations:
  12.     - match:
  13.         service: database.kuma-example.svc:5432
  14.   conf:
  15.     # Forward the logs into the logging backend named `logstash`.
  16.     backend: logstash

When backend field of a TrafficLog policy is omitted, the logs will be forwarded into the defaultBackend of that Mesh.

Access Log Format

Kuma gives you full control over the format of access logs.

The shape of a single log record is defined by a template string that uses command operatorsTraffic Log - 图1 to extract and format data about a TCP connection or an HTTP request.

E.g.,

  1. %START_TIME% %KUMA_SOURCE_SERVICE% => %KUMA_DESTINATION_SERVICE% %DURATION%

where %START_TIME% and %KUMA_SOURCE_SERVICE% are examples of available command operators.

A complete set of supported command operators consists of:

  1. All command operators supported by EnvoyTraffic Log - 图2
  2. Command operators unique to Kuma

The latter include:

Command OperatorDescription
%KUMA_MESH%name of the mesh in which traffic is flowing
%KUMA_SOURCE_SERVICE%name of a service that is the source of traffic
%KUMA_DESTINATION_SERVICE%name of a service that is the destination of traffic
%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%address of a Dataplane that is the source of traffic

Access Logs for TCP and HTTP traffic

All access log command operators are valid to use with both TCP and HTTP traffic.

If a command operator is specific to HTTP traffic, such as %REQ(X?Y):Z% or %RESP(X?Y):Z%, it will be replaced by a symbol “-“ in case of TCP traffic.

Internally, Kuma determines traffic protocol based on the value of protocol tag on the inbound interface of a destination Dataplane.

The default format string for TCP traffic is:

  1. [%START_TIME%] %RESPONSE_FLAGS% %KUMA_MESH% %KUMA_SOURCE_ADDRESS_WITHOUT_PORT%(%KUMA_SOURCE_SERVICE%)->%UPSTREAM_HOST%(%KUMA_DESTINATION_SERVICE%) took %DURATION%ms, sent %BYTES_SENT% bytes, received: %BYTES_RECEIVED% bytes

The default format string for HTTP traffic is:

  1. [%START_TIME%] %KUMA_MESH% "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%KUMA_SOURCE_SERVICE%" "%KUMA_DESTINATION_SERVICE%" "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%" "%UPSTREAM_HOST%"

To provide different format for TCP and HTTP logging you can define two separate logging backends with the same address and different format. Then define two TrafficLog entity, one for TCP and one for HTTP with protocol: http selector.

Access Logs in JSON format

If you need an access log with entries in JSON format, you have to provide a template string that is a valid JSON object, e.g.

  1. {
  2.   "start_time":          "%START_TIME%",
  3.   "source":              "%KUMA_SOURCE_SERVICE%",
  4.   "destination":         "%KUMA_DESTINATION_SERVICE%",
  5.   "source_address":      "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%",
  6.   "destination_address": "%UPSTREAM_HOST%",
  7.   "duration_millis":     %DURATION%,
  8.   "bytes_received":      %BYTES_RECEIVED%,
  9.   "bytes_sent":          %BYTES_SENT%
  10. }

To use it with Logstash, use json_lines codec and make sure your JSON is formatted into one line.