Configure access logs with Telemetry API

Telemetry API has been in Istio as a first-class API for quite sometime now. Previously users had to configure telemetry in the MeshConfig section of Istio configuration.

Before you begin

  • Setup Istio by following the instructions in the Installation guide.

    The egress gateway and access logging will be enabled if you install the demo configuration profile.

  • Deploy the sleep sample app to use as a test source for sending requests. If you have automatic sidecar injection enabled, run the following command to deploy the sample app:

    Zip

    1. $ kubectl apply -f @samples/sleep/sleep.yaml@

    Otherwise, manually inject the sidecar before deploying the sleep application with the following command:

    Zip

    1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)

    You can use any pod with curl installed as a test source.

  • Set the SOURCE_POD environment variable to the name of your source pod:

    1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})

  • Start the httpbin sample.

    If you have enabled automatic sidecar injection, deploy the httpbin service:

    Zip

    1. $ kubectl apply -f @samples/httpbin/httpbin.yaml@

    Otherwise, you have to manually inject the sidecar before deploying the httpbin application:

    Zip

    1. $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@)

Installation

In this example, we will send logs to Grafana Loki so make sure it is installed:

ZipZipZip

  1. $ istioctl install -f @samples/open-telemetry/loki/iop.yaml@ --skip-confirmation
  2. $ kubectl apply -f @samples/addons/loki.yaml@ -n istio-system
  3. $ kubectl apply -f @samples/open-telemetry/loki/otel.yaml@ -n istio-system

Get started with Telemetry API

  1. Enable access logging

    1. $ cat <<EOF | kubectl apply -n istio-system -f -
    2. apiVersion: telemetry.istio.io/v1
    3. kind: Telemetry
    4. metadata:
    5.   name: mesh-logging-default
    6. spec:
    7.   accessLogging:
    8.   - providers:
    9.     - name: otel
    10. EOF

    The above example uses the built-in envoy access log provider, and we do not configure anything other than default settings.

  2. Disable access log for specific workload

    You can disable access log for sleep service with the following configuration:

    1. $ cat <<EOF | kubectl apply -n default -f -
    2. apiVersion: telemetry.istio.io/v1
    3. kind: Telemetry
    4. metadata:
    5.   name: disable-sleep-logging
    6.   namespace: default
    7. spec:
    8.   selector:
    9.     matchLabels:
    10.       app: sleep
    11.   accessLogging:
    12.   - providers:
    13.     - name: otel
    14.     disabled: true
    15. EOF

  3. Filter access log with workload mode

    You can disable inbound access log for httpbin service with the following configuration:

    1. $ cat <<EOF | kubectl apply -n default -f -
    2. apiVersion: telemetry.istio.io/v1
    3. kind: Telemetry
    4. metadata:
    5.   name: disable-httpbin-logging
    6. spec:
    7.   selector:
    8.     matchLabels:
    9.       app: httpbin
    10.   accessLogging:
    11.   - providers:
    12.     - name: otel
    13.     match:
    14.       mode: SERVER
    15.     disabled: true
    16. EOF

  4. Filter access log with CEL expression

    The following configuration displays access log only when response code is greater or equal to 500:

    1. $ cat <<EOF | kubectl apply -n default -f -
    2. apiVersion: telemetry.istio.io/v1alpha1
    3. kind: Telemetry
    4. metadata:
    5.   name: filter-sleep-logging
    6. spec:
    7.   selector:
    8.     matchLabels:
    9.       app: sleep
    10.   accessLogging:
    11.   - providers:
    12.     - name: otel
    13.     filter:
    14.       expression: response.code >= 500
    15. EOF

    There’s no response.code attribute when connections fail. In that case, you should use the CEL expression !has(response.code) || response.code >= 500.

  5. Set default filter access log with CEL expression

    The following configuration displays access logs only when the response code is greater or equal to 400 or the request went to the BlackHoleCluster or the PassthroughCluster: Note: The xds.cluster_name is only available with Istio release 1.16.2 and higher

    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: telemetry.istio.io/v1alpha1
    3. kind: Telemetry
    4. metadata:
    5.   name: default-exception-logging
    6.   namespace: istio-system
    7. spec:
    8.   accessLogging:
    9.   - providers:
    10.     - name: otel
    11.     filter:
    12.       expression: "response.code >= 400 || xds.cluster_name == 'BlackHoleCluster' ||  xds.cluster_name == 'PassthroughCluster' "
    14. EOF

  6. Filter health check access logs with CEL expression

    The following configuration displays access logs only when the logs are not generated by the Amazon Route 53 Health Check Service. Note: The request.useragent is specific to HTTP traffic, therefore to avoid breaking TCP traffic, we need to check for the existence of the field. For more information, see CEL Type Checking

    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: telemetry.istio.io/v1alpha1
    3. kind: Telemetry
    4. metadata:
    5.   name: filter-health-check-logging
    6. spec:
    7.   accessLogging:
    8.   - providers:
    9.     - name: otel
    10.     filter:
    11.       expression: "!has(request.useragent) || !(request.useragent.startsWith("Amazon-Route53-Health-Check-Service"))"
    12. EOF

    For more information, see Use expressions for values

Work with OpenTelemetry provider

Istio supports sending access logs with OpenTelemetry protocol, as explained here.

Cleanup

  1. Remove all Telemetry API:

    1. $ kubectl delete telemetry --all -A

  2. Remove loki:

    ZipZip

    1. $ kubectl delete -f @samples/addons/loki.yaml@ -n istio-system
    2. $ kubectl delete -f @samples/open-telemetry/loki/otel.yaml@ -n istio-system

  3. Uninstall Istio from the cluster:

    1. $ istioctl uninstall --purge --skip-confirmation