Health Check

Circuit Breaker is an outbound policy. Dataplanes whose configuration is modified are in the sources matcher.

This policy enables Kuma to keep track of the health of every data plane proxy, with the goal of minimizing the number of failed requests in case a data plane proxy is temporarily unhealthy.

By creating an HealthCheck resource we can instruct a data plane proxy to keep track of the health status for any other data plane proxy. When health-checks are properly configured, a data plane proxy will never send a request to another data plane proxy that is considered unhealthy. When an unhealthy proxy returns to a healthy state, Kuma will resume sending requests to it again.

This policy provides active checks. If you want to configure passive checks, please utilize the Circuit Breaker policy. Data plane proxies with active checks will explicitly send requests to other data plane proxies to determine if target proxies are healthy or not. This mode generates extra traffic to other proxies and services as described in the policy configuration.

Usage

As usual, we can apply sources and destinations selectors to determine how health-checks will be performed across our data plane proxies.

The HealthCheck policy supports both L4/TCP (default) and L7/HTTP checks.

Examples

  1. apiVersion: kuma.io/v1alpha1
  2. kind: HealthCheck
  3. mesh: default
  4. metadata:
  5.   name: web-to-backend-check
  6. spec:
  7.   sources:
  8.   - match:
  9.       kuma.io/service: web_default_svc_80
  10.   destinations:
  11.   - match:
  12.       kuma.io/service: backend_default_svc_80
  13.   conf:
  14.     interval: 10s
  15.     timeout: 2s
  16.     unhealthyThreshold: 3
  17.     healthyThreshold: 1
  18.     initialJitter: 5s # optional
  19.     intervalJitter: 6s # optional
  20.     intervalJitterPercent: 10 # optional
  21.     healthyPanicThreshold: 60 # optional, by default 50
  22.     failTrafficOnPanic: true # optional, by default false
  23.     noTrafficInterval: 10s # optional, by default 60s
  24.     eventLogPath: "/tmp/health-check.log" # optional
  25.     alwaysLogHealthCheckFailures: true # optional, by default false
  26.     reuseConnection: false # optional, by default true
  27.     tcp: # only one of tcp or http can be defined
  28.       send: Zm9v
  29.       receive:
  30.       - YmFy
  31.       - YmF6
  32.     http:
  33.       path: /health
  34.       requestHeadersToAdd:
  35.       - append: false
  36.         header:
  37.           key: Content-Type
  38.           value: application/json
  39.       - header:
  40.           key: Accept
  41.           value: application/json
  42.       expectedStatuses: [200, 201]

We will apply the configuration with kubectl apply -f [..].

  1. type: HealthCheck
  2. name: web-to-backend-check
  3. mesh: default
  4. sources:
  5. - match:
  6.     kuma.io/service: web
  7. destinations:
  8. - match:
  9.     kuma.io/service: backend
  10. conf:
  11.   interval: 10s
  12.   timeout: 2s
  13.   unhealthyThreshold: 3
  14.   healthyThreshold: 1
  15.   initialJitter: 5s # optional
  16.   intervalJitter: 6s # optional
  17.   intervalJitterPercent: 10 # optional
  18.   healthyPanicThreshold: 60 # optional, by default 50
  19.   failTrafficOnPanic: true # optional, by default false
  20.   noTrafficInterval: 10s # optional, by default 60s
  21.   eventLogPath: "/tmp/health-check.log" # optional
  22.   alwaysLogHealthCheckFailures: true # optional, by default false
  23.   reuseConnection: false # optional, by default true
  24.   tcp: # only one of tcp or http can be defined
  25.     send: Zm9v
  26.     receive:
  27.     - YmFy
  28.     - YmF6
  29.   http:
  30.     path: /health
  31.     requestHeadersToAdd:
  32.     - append: false
  33.       header:
  34.         key: Content-Type
  35.         value: application/json
  36.     - header:
  37.         key: Accept
  38.         value: application/json
  39.     expectedStatuses: [200, 201]

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

HTTP

HTTP health checks are executed using HTTP 2

  • path - HTTP path which will be requested during the health checks
  • expectedStatuses (optional) - list of status codes which should be considered as a healthy during the checks
    • only statuses in the range [100, 600) are allowed
    • by default, when this property is not provided only responses with status code 200 are being considered healthy
  • requestHeadersToAdd (optional) - list of headers which should be added to every health check request:
    • append (default, optional) - should the value of the provided header be appended to already existing headers (if present)
    • header:
      • key - the name of the header
      • value (optional) - the value of the header

TCP

  • send - Base64 encoded content of the message which should be sent during the health checks
  • receive list of Base64 encoded blocks of strings which should be found in the returning message which should be considered as healthy
    • when checking the response, “fuzzy” matching is performed such that each block must be found, and in the order specified, but not necessarily contiguous;
    • if receive section won’t be provided or will be empty, checks will be performed as “connect only” and will be marked as successful when TCP connection will be successfully established.

Matching

HealthCheck is an Outbound Connection Policy. The only supported value for destinations.match is kuma.io/service.