Configure Health Checks for Consul on Kubernetes

This topic requires familiarity with Kubernetes Health Checks.

This page describes how Consul on Kubernetes will sync the status of Kubernetes health probes of a pod to Consul for service mesh use cases. Health check synchronization with Consul is done automatically whenever connectInject.enabled is true.

For each Kubernetes pod that is connect-injected the following will be configured:

1. A Consul health check is registered within Consul catalog. The Consul health check’s state reflects the pod’s readiness status.

2. If the pod is using transparent proxy mode, the mutating webhook redirects all http based startup, liveness, and readiness probes in the pod through the Envoy proxy. This webhook is defined in the ExposePaths configuration for each probe so that kubelet can access the endpoint through the Envoy proxy.

The mutation behavior can be disabled, by setting either the consul.hashicorp.com/transparent-proxy-overwrite-probes pod annotation to false or the connectInject.defaultOverwriteProbes Helm value to false.

When readiness probes are set for a pod, the status of the pod will be reflected within Consul and will cause Consul to redirect service mesh traffic to the pod based on the pod’s health. If the pod has failing health checks, Consul will no longer use the service instance associated with the pod for service mesh traffic. When the pod passes its health checks, Consul will then use the respective service instance for service mesh traffic.

In the case where no user defined health checks are assigned to a pod, the default behavior is that the Consul health check will be marked passing until the pod becomes unready.

It is highly recommended to enable TLS for all production configurations to mitigate any security concerns should the pod network ever be compromised. The controller makes calls across the network to Consul agents on all nodes so an attacker could potentially sniff ACL tokens if those calls are not encrypted via TLS.