Health Checking of Istio Services

Kubernetes liveness and readiness probesoffer three different options:

  • Command
  • TCP request
  • HTTP requestThis guide shows how to use these approaches in Istio with mutual TLS enabled.

Command and TCP type probes work with Istio regardless of whether or not mutual TLS is enabled. The HTTP request approach requires different Istio configuration withmutual TLS enabled.

Before you begin

Liveness and readiness probes with command option

First, you need to configure health checking with mutual TLS enabled.

To enable mutual TLS for services in the default namespace, you must configure an authentication policy and a destination rule.Follow these steps to complete the configuration:

  • To configure the authentication policy, run:
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: "authentication.istio.io/v1alpha1"
  3. kind: "Policy"
  4. metadata:
  5.   name: "default"
  6.   namespace: "default"
  7. spec:
  8.   peers:
  9.   - mtls: {}
  10. EOF
  • To configure the destination rule, run:
  1. $ kubectl apply -f - <<EOF
  2. apiVersion: "networking.istio.io/v1alpha3"
  3. kind: "DestinationRule"
  4. metadata:
  5.   name: "default"
  6.   namespace: "default"
  7. spec:
  8.   host: "*.default.svc.cluster.local"
  9.   trafficPolicy:
  10.     tls:
  11.       mode: ISTIO_MUTUAL
  12. EOF

Run the following command to deploy the service:

Zip

  1. $ kubectl apply -f <(istioctl kube-inject -f @samples/health-check/liveness-command.yaml@)

Repeat the check status command to verify that the liveness probes work:

  1. $ kubectl get pod
  2. NAME                             READY     STATUS    RESTARTS   AGE
  3. liveness-6857c8775f-zdv9r        2/2       Running   0           4m

Liveness and readiness probes with HTTP request option

This section shows how to configure health checking with the HTTP request option when mutual TLS is enabled.

Kubernetes HTTP health check request is sent from Kubelet, which does not have Istio issued certificate to the liveness-http service. So when mutual TLS is enabled, the health check request will fail.

We have two options to solve the problem: probe rewrites and separate ports.

Probe rewrite

This approach rewrites the application PodSpec readiness/liveness probe, such that the probe request will be sent toPilot agent. Pilot agent then redirects therequest to application, and strips the response body only returning the response code.

You have two ways to enable Istio to rewrite the liveness HTTP probes.

Enable globally via install option

Install Istio with —set values.sidecarInjectorWebhook.rewriteAppHTTPProbe=true.

Alternatively, update the configuration map of Istio sidecar injection:

  1. $ kubectl get cm istio-sidecar-injector -n istio-system -o yaml | sed -e 's/"rewriteAppHTTPProbe":false/"rewriteAppHTTPProbe":true/' | kubectl apply -f -

The above installation option and configuration map, each instruct the sidecar injection process to automaticallyrewrite the Kubernetes pod’s spec, so health checks are able to work under mutual TLS. No need to update your app or podspec by yourself.

The configuration changes above (by install or by the configuration map) effect all Istio app deployments.

Use annotations on pod

Rather than install Istio with different options, you can annotate the pod with sidecar.istio.io/rewriteAppHTTPProbers: "true". Make sure you add the annotation to the pod resource because it will be ignored anywhere else (for example, on an enclosing deployment resource).

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: liveness-http
  5. spec:
  6.   template:
  7.     metadata:
  8.       labels:
  9.         app: liveness-http
  10.         version: v1
  11.       annotations:
  12.         sidecar.istio.io/rewriteAppHTTPProbers: "true"
  13.     spec:
  14.       containers:
  15.       - name: liveness-http
  16.         image: docker.io/istio/health:example
  17.         ports:
  18.         - containerPort: 8001
  19.         livenessProbe:
  20.           httpGet:
  21.             path: /foo
  22.             port: 8001
  23.           initialDelaySeconds: 5
  24.           periodSeconds: 5

This approach allows you to enable the health check prober rewrite gradually on each deployment without reinstalling Istio.

Re-deploy the liveness health check app

Instructions below assume you turn on the feature globally via install option.Annotations works the same.

ZipZip

  1. $ kubectl delete -f <(istioctl kube-inject -f @samples/health-check/liveness-http-same-port.yaml@)
  2. $ kubectl apply -f <(istioctl kube-inject -f @samples/health-check/liveness-http-same-port.yaml@)
  1. $ kubectl get pod
  2. NAME                             READY     STATUS    RESTARTS   AGE
  3. liveness-http-975595bb6-5b2z7c   2/2       Running   0           1m

This feature is not currently turned on by default. We’d like to hear your feedbackon whether we should change this to default behavior for Istio installation.

Separate port

Another alternative is to use separate port for health checking and regular traffic.

Run these commands to re-deploy the service:

ZipZip

  1. $ kubectl delete -f <(istioctl kube-inject -f @samples/health-check/liveness-http.yaml@)
  2. $ kubectl apply -f <(istioctl kube-inject -f @samples/health-check/liveness-http.yaml@)

Wait for a minute and check the pod status to make sure the liveness probes work with ‘0’ in the ‘RESTARTS’ column.

  1. $ kubectl get pod
  2. NAME                             READY     STATUS    RESTARTS   AGE
  3. liveness-http-67d5db65f5-765bb   2/2       Running   0          1m

Note that the image in liveness-http exposes two ports: 8001 and 8002 (source code). In this deployment, port 8001 serves the regular traffic while port 8002 is used for liveness probes.

Cleanup

Remove the mutual TLS policy and corresponding destination rule added in the steps above:

  1. $ kubectl delete policies default
  2. $ kubectl delete destinationrules default

See also

Citadel Health Checking

Shows how to enable Citadel health checking with Kubernetes.

DNS Certificate Management

Provision and manage DNS certificates in Istio.

Introducing the Istio v1beta1 Authorization Policy

Introduction, motivation and design principles for the Istio v1beta1 Authorization Policy.

Secure Webhook Management

A more secure way to manage Istio webhooks.

Multi-Mesh Deployments for Isolation and Boundary Protection

Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.

App Identity and Access Adapter

Using Istio to secure multi-cloud Kubernetes applications with zero code changes.