Kubernetes Health Checks in Consul on Kubernetes

This topic requires familiarity with Kubernetes Health Checks.

This page describes how Consul on Kubernetes will sync the status of Kubernetes health probes of a pod to Consul for service mesh use cases. Health check synchronization with Consul is done automatically whenever connectInject.enabled is true.

For each Kubernetes pod that is connect-injected the following will be configured:

1. A TTL health check is registered within Consul. The Consul health check’s state will reflect the pod’s readiness status, which is the combination of all Kubernetes probes registered with the pod.

2. If the pod is utilizing Transparent Proxy mode, the mutating webhook will mutate all http based Startup, Liveness, and Readiness probes in the pod to redirect through the Envoy proxy. This is done with ExposePaths configuration for each probe so that kubelet can access the endpoint through the Envoy proxy.

The mutation behavior can be disabled by either setting the consul.hashicorp.com/transparent-proxy-overwrite-probes pod annotation to false or the connectInject.defaultOverwriteProbes Helm value to false.

When readiness probes are set for a pod, the status of the pod will be reflected within Consul and will cause Consul to redirect service mesh traffic to the pod based on the pod’s health. If the pod has failing health checks, Consul will no longer use the service instance associated with the pod for service mesh traffic. When the pod passes its health checks, Consul will then use the respective service instance for service mesh traffic.

In the case where no user defined health checks are assigned to a pod, the default behavior is that the Consul health check will be marked passing until the pod becomes unready.

It is highly recommended to enable TLS for all production configurations to mitigate any security concerns should the pod network ever be compromised. The controller makes calls across the network to Consul agents on all nodes so an attacker could potentially sniff ACL tokens if those calls are not encrypted via TLS.