Configure access logs with Telemetry API
Telemetry API has been in Istio as a first-class API for quite sometime now. Previously users had to configure telemetry in the MeshConfig
section of Istio configuration.
Before you begin
Setup Istio by following the instructions in the Installation guide.
The egress gateway and access logging will be enabled if you install the
demo
configuration profile.Deploy the sleep sample app to use as a test source for sending requests. If you have automatic sidecar injection enabled, run the following command to deploy the sample app:
$ kubectl apply -f @samples/sleep/sleep.yaml@
Otherwise, manually inject the sidecar before deploying the
sleep
application with the following command:$ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)
You can use any pod with
curl
installed as a test source.Set the
SOURCE_POD
environment variable to the name of your source pod:$ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
Start the httpbin sample.
If you have enabled automatic sidecar injection, deploy the
httpbin
service:$ kubectl apply -f @samples/httpbin/httpbin.yaml@
Otherwise, you have to manually inject the sidecar before deploying the
httpbin
application:$ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@)
Installation
In this example, we will send logs to Grafana Loki so make sure it is installed:
$ istioctl install -f @samples/open-telemetry/loki/iop.yaml@ --skip-confirmation
$ kubectl apply -f @samples/addons/loki.yaml@ -n istio-system
$ kubectl apply -f @samples/open-telemetry/loki/otel.yaml@ -n istio-system
Get started with Telemetry API
Enable access logging
$ cat <<EOF | kubectl apply -n istio-system -f -
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-logging-default
spec:
accessLogging:
- providers:
- name: otel
EOF
The above example uses the built-in
envoy
access log provider, and we do not configure anything other than default settings.Disable access log for specific workload
You can disable access log for
sleep
service with the following configuration:$ cat <<EOF | kubectl apply -n default -f -
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: disable-sleep-logging
namespace: default
spec:
selector:
matchLabels:
app: sleep
accessLogging:
- providers:
- name: otel
disabled: true
EOF
Filter access log with workload mode
You can disable inbound access log for
httpbin
service with the following configuration:$ cat <<EOF | kubectl apply -n default -f -
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: disable-httpbin-logging
spec:
selector:
matchLabels:
app: httpbin
accessLogging:
- providers:
- name: otel
match:
mode: SERVER
disabled: true
EOF
Filter access log with CEL expression
The following configuration displays access log only when response code is greater or equal to 500:
$ cat <<EOF | kubectl apply -n default -f -
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: filter-sleep-logging
spec:
selector:
matchLabels:
app: sleep
accessLogging:
- providers:
- name: otel
filter:
expression: response.code >= 500
EOF
Set default filter access log with CEL expression
The following configuration displays access logs only when the response code is greater or equal to 400 or the request went to the BlackHoleCluster or the PassthroughCluster: Note: The
xds.cluster_name
is only available with Istio release 1.16.2 and higher$ cat <<EOF | kubectl apply -f -
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: default-exception-logging
namespace: istio-system
spec:
accessLogging:
- providers:
- name: otel
filter:
expression: "response.code >= 400 || xds.cluster_name == 'BlackHoleCluster' || xds.cluster_name == 'PassthroughCluster' "
EOF
For more information, see Use expressions for values
Work with OpenTelemetry provider
Istio supports sending access logs with OpenTelemetry protocol, as explained here.
Cleanup
Remove all Telemetry API:
$ kubectl delete telemetry --all -A
Remove
loki
:$ kubectl delete -f @samples/addons/loki.yaml@ -n istio-system
$ kubectl delete -f @samples/open-telemetry/loki/otel.yaml@ -n istio-system
Uninstall Istio from the cluster:
$ istioctl uninstall --purge --skip-confirmation