Health Checking of Istio Services

Kubernetes liveness and readiness probes describes several ways to configure liveness and readiness probes including:

  1. Command
  2. HTTP request

The command approach works with Istio regardless of whether or not mutual TLS is enabled.

The HTTP request approach, on the other hand, requires special Istio configuration when mutual TLS is enabled. This is because the health check requests to the liveness-http service are sent by Kubelet, which does not have an Istio issued certificate. Therefore when mutual TLS is enabled, the health check requests will fail.

Istio solves this problem by rewriting the application PodSpec readiness/liveness probe, so that the probe request is sent to Pilot agent. Pilot agent then redirects the request to the application, strips the response body, only returning the response code.

This feature is enabled by default in all built-in Istio configuration profiles but can be disabled as described below.

Liveness and readiness probes using the command approach

Istio provides a liveness sample that implements this approach. To demonstrate it working with mutual TLS enabled, first create a namespace for the example:

  1. $ kubectl create ns istio-io-health

To configure strict mutual TLS, run:

  1. $ kubectl apply -f - <<EOF
  2. apiVersion: "security.istio.io/v1beta1"
  3. kind: "PeerAuthentication"
  4. metadata:
  5.   name: "default"
  6.   namespace: "istio-io-health"
  7. spec:
  8.   mtls:
  9.     mode: STRICT
  10. EOF

Next, run the following command to deploy the sample service:

Zip

  1. $ kubectl -n istio-io-health apply -f <(istioctl kube-inject -f @samples/health-check/liveness-command.yaml@)

To confirm that the liveness probes are working, check the status of the sample pod to verify that it is running.

  1. $ kubectl -n istio-io-health get pod
  2. NAME                             READY     STATUS    RESTARTS   AGE
  3. liveness-6857c8775f-zdv9r        2/2       Running   0           4m

Liveness and readiness probes using the HTTP request approach

As stated previously, Istio uses probe rewrite to implement HTTP probes by default. You can disable this feature either for specific pods, or globally.

Disable the HTTP probe rewrite for a pod

You can annotate the pod with sidecar.istio.io/rewriteAppHTTPProbers: "false" to disable the probe rewrite option. Make sure you add the annotation to the pod resource because it will be ignored anywhere else (for example, on an enclosing deployment resource).

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: liveness-http
  5. spec:
  6.   selector:
  7.     matchLabels:
  8.       app: liveness-http
  9.       version: v1
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: liveness-http
  14.         version: v1
  15.       annotations:
  16.         sidecar.istio.io/rewriteAppHTTPProbers: "false"
  17.     spec:
  18.       containers:
  19.       - name: liveness-http
  20.         image: docker.io/istio/health:example
  21.         ports:
  22.         - containerPort: 8001
  23.         livenessProbe:
  24.           httpGet:
  25.             path: /foo
  26.             port: 8001
  27.           initialDelaySeconds: 5
  28.           periodSeconds: 5

This approach allows you to disable the health check probe rewrite gradually on individual deployments, without reinstalling Istio.

Disable the probe rewrite globally

Install Istio using --set values.sidecarInjectorWebhook.rewriteAppHTTPProbe=false to disable the probe rewrite globally. Alternatively, update the configuration map for the Istio sidecar injector:

  1. $ kubectl get cm istio-sidecar-injector -n istio-system -o yaml | sed -e 's/"rewriteAppHTTPProbe": true/"rewriteAppHTTPProbe": false/' | kubectl apply -f -

Cleanup

Remove the namespace used for the examples:

  1. $ kubectl delete ns istio-io-health

See also

Istio in 2020 - Following the Trade Winds

A vision statement and roadmap for Istio in 2020.

Remove cross-pod unix domain sockets

A more secure way to manage secrets.

DNS Certificate Management

Provision and manage DNS certificates in Istio.

Introducing the Istio v1beta1 Authorization Policy

Introduction, motivation and design principles for the Istio v1beta1 Authorization Policy.

Secure Webhook Management

A more secure way to manage Istio webhooks.

Multi-Mesh Deployments for Isolation and Boundary Protection

Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.