MeshCircuitBreaker

This policy uses new policy matching algorithm. Do not combine with CircuitBreaker.

This policy will look for errors in the live traffic being exchanged between our data plane proxies. It will mark a data proxy as unhealthy if certain conditions are met. The policy will ensure that no additional traffic can reach an unhealthy data plane proxy until it is healthy again.

Circuit breakers - unlike active MeshHealthChecks - do not send additional traffic to our data plane proxies but they rather inspect the existing service traffic. They are also commonly used to prevent cascading failures.

Like a real-world circuit breaker when the circuit is closed then traffic between a source and destination data plane proxy is allowed to freely flow through it. When it is open then the traffic is interrupted.

The conditions that determine when a circuit breaker is closed or open are being configured on connection limits or outlier detection basis. For outlier detection to open circuit breaker you can configure what we call detectors. This policy provides 5 different types of detectors, and they are triggered on some deviations in the upstream service behavior. All detectors could coexist on the same outbound interface.

Once one of the detectors has been triggered the corresponding data plane proxy is ejected from the set of the load balancer for a period equal to baseEjectionTime. Every further ejection of the same data plane proxy will further extend the baseEjectionTime multiplied by the number of ejections: for example the fourth ejection will be lasting for a period of time of 4 * baseEjectionTime.

This policy provides passive checks. If you want to configure active checks, please utilize the MeshHealthCheck policy. Data plane proxies with passive checks won’t explicitly send requests to other data plane proxies to determine if target proxies are healthy or not.

TargetRef support matrix

targetRefAllowed kinds
targetRef.kindMesh, MeshSubset
to[].targetRef.kindMesh, MeshService
from[].targetRef.kindMesh
targetRefAllowed kinds
targetRef.kindMesh, MeshGateway, MeshGateway with listener tags
to[].targetRef.kindMesh, MeshService
targetRefAllowed kinds
targetRef.kindMesh, MeshSubset
to[].targetRef.kindMesh, MeshService

To learn more about the information in this table, see the matching docs.

Configuration

Connection limits

  • maxConnections - (optional) The maximum number of connections allowed to be made to the upstream Envoy Cluster. If not specified then equal to 1024.
  • maxConnectionPools - (optional) The maximum number of connection pools per Envoy Cluster that are concurrently supported at once. Set this for Envoy Clusters which create a large number of connection pools. If not specified, the default is unlimited.
  • maxPendingRequests - (optional) The maximum number of pending requests that are allowed to the upstream Envoy Cluster. This limit is applied as a connection limit for non-HTTP traffic. If not specified then equal to 1024.
  • maxRetries - (optional) The maximum number of parallel retries that will be allowed to the upstream Envoy Cluster. If not specified then equal to 3.
  • maxRequests - (optional) The maximum number of parallel requests that are allowed to be made to the upstream Envoy Cluster. This limit does not apply to non-HTTP traffic. If not specified then equal to 1024.

Outlier detection

Outlier detection can be configured for HTTP, TCP or gRPC traffic.

For gRPC requests, the outlier detection will use the HTTP status mapped from the grpc-status response header.

  • disabled - (optional) When set to true, outlierDetection configuration won’t take any effect.
  • interval - (optional) The time interval between ejection analysis sweeps. This can result in both new ejections and hosts being returned to service.
  • baseEjectionTime - (optional) The base time that a host is ejected for. The real time is equal to the base time multiplied by the number of times the host has been ejected.
  • maxEjectionPercent - (optional) The maximum % of an upstream Envoy Clusters that can be ejected due to outlier detection. Defaults to 10% but will eject at least one host regardless of the value.
  • splitExternalAndLocalErrors - (optional) Determines whether to distinguish local origin failures from external errors. If set to true the following configuration parameters are taken into account: detectors.localOriginFailures.consecutive.
  • detectors - Contains configuration for supported outlier detectors. At least one detector needs to be configured when policy is configured for outlier detection.

Detectors configuration

Configuration for supported outlier detectors. At least one detector needs to be configured when policy is configured for outlier detection.

Depending on mode the outlier detection can take into account all or externally originated (transaction) errors only.

Default mode is when splitExternalAndLocalErrors is not set or equal false

This detection type takes into account all generated errors: locally originated and externally originated (transaction) errors.

Configuration

  • totalFailures.consecutive - The number of consecutive server-side error responses (for HTTP traffic, 5xx responses; for TCP traffic, connection failures; etc.) before a consecutive total failure ejection occurs.

Example

  1. type: MeshCircuitBreaker
  2. mesh: default
  3. name: circuit-breaker
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   to:
  8.   - targetRef:
  9.       kind: Mesh
  10.     default:
  11.       outlierDetection:
  12.         detectors:
  13.           totalFailures:
  14.             consecutive: 10

Split Mode is when splitExternalAndLocalErrors is equal true

This detection type takes into account only externally originated (transaction) errors, ignoring locally originated ones.

HTTP

If an upstream host is an HTTP-server, only 5xx types of error are taken into account (see Consecutive Gateway Failure for exceptions).

Properly formatted responses, even when they carry an operational error (like index not found, access denied) are not taken into account.

Configuration

  • totalFailures.consecutive - The number of consecutive server-side error responses (for HTTP traffic, 5xx responses) before a consecutive total failure ejection occurs.

Example

  1. type: MeshCircuitBreaker
  2. mesh: default
  3. name: circuit-breaker
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   to:
  8.   - targetRef:
  9.       kind: Mesh
  10.     default:
  11.       outlierDetection:
  12.         splitExternalAndLocalErrors: true
  13.         detectors:
  14.           totalFailures:
  15.             consecutive: 10

Depending on mode the outlier detection can take into account gateway failures with locally originated failures (default mode) or gateway failures only (split mode).

Default mode is when splitExternalAndLocalErrors is not set or equal false

This detection type takes into account a subset of 5xx errors, called “gateway errors” (502, 503 or 504 status code) and local origin failures, such as timeout, TCP reset etc.

Configuration

  • gatewayFailures.consecutive - The number of consecutive gateway failures (502, 503, 504 status codes) before a consecutive gateway failure ejection occurs.

Example

  1. type: MeshCircuitBreaker
  2. mesh: default
  3. name: circuit-breaker
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   to:
  8.   - targetRef:
  9.       kind: Mesh
  10.     default:
  11.       outlierDetection:
  12.         detectors:
  13.           gatewayFailures:
  14.             consecutive: 10

Split Mode is when splitExternalAndLocalErrors is equal true

This detection type takes into account a subset of 5xx errors, called “gateway errors” (502, 503 or 504 status code).

This detector is supported only for HTTP traffic.

Configuration

  • gatewayFailures.consecutive - The number of consecutive gateway failures (502, 503, 504 status codes) before a consecutive gateway failure ejection occurs.

Example

  1. type: MeshCircuitBreaker
  2. mesh: default
  3. name: circuit-breaker
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   to:
  8.   - targetRef:
  9.       kind: Mesh
  10.     default:
  11.       outlierDetection:
  12.         splitExternalAndLocalErrors: true
  13.         detectors:
  14.           gatewayFailures:
  15.             consecutive: 10

This detection is supported only in Split Mode

This detection takes into account only locally originated errors (timeout, reset, etc).

If Envoy repeatedly cannot connect to an upstream host or communication with the upstream host is repeatedly interrupted, it will be ejected. Various locally originated problems are detected: timeout, TCP reset, ICMP errors, etc.

Split Mode is when splitExternalAndLocalErrors is equal true

Configuration

  • localOriginFailures.consecutive - The number of consecutive locally originated failures before ejection occurs.

Example

  1. type: MeshCircuitBreaker
  2. mesh: default
  3. name: circuit-breaker
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   to:
  8.   - targetRef:
  9.       kind: Mesh
  10.     default:
  11.       outlierDetection:
  12.         splitExternalAndLocalErrors: true
  13.         detectors:
  14.           localOriginFailures:
  15.             consecutive: 10

This detection is not supported in the Default Mode

Success Rate based outlier detection aggregates success rate data from every host in an Envoy Cluster. Then at given intervals ejects hosts based on statistical outlier detection.

Success Rate outlier detection will not be calculated for a host if its request volume over the aggregation interval is less than the value of successRate.requestVolume value.

Moreover, detection will not be performed for a cluster if the number of hosts with the minimum required request volume in an interval is less than the successRate.minimumHosts value.

Default mode is when splitExternalAndLocalErrors is not set or equal false

This detection type takes into account all types of errors: locally and externally originated.

Split Mode is when splitExternalAndLocalErrors is equal true

Locally originated errors and externally originated (transaction) errors are counted and treated separately.

Configuration

  • successRate.minimumHosts - The number of hosts in an Envoy Cluster that must have enough request volume to detect success rate outliers. If the number of hosts is less than this setting, outlier detection via success rate statistics is not performed for any host in the Cluster.
  • successRate.requestVolume - The minimum number of total requests that must be collected in one interval (as defined by the interval duration configured in outlierDetection section) to include this host in success rate based outlier detection. If the volume is lower than this setting, outlier detection via success rate statistics is not performed for that host.
  • successRate.standardDeviationFactor - This factor is used to determine the ejection threshold for success rate outlier ejection. The ejection threshold is the difference between the mean success rate, and the product of this factor and the standard deviation of the mean success rate: mean - (standard_deviation *success_rate_standard_deviation_factor). Either int or decimal represented as string.

Example

  1. type: MeshCircuitBreaker
  2. mesh: default
  3. name: circuit-breaker
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   to:
  8.   - targetRef:
  9.       kind: Mesh
  10.     default:
  11.       outlierDetection:
  12.         splitExternalAndLocalErrors: true
  13.         detectors:
  14.           successRate:
  15.             minimumHosts: 5
  16.             requestVolume: 10
  17.             standardDeviationFactor: "1.9"

Failure Percentage based outlier detection functions similarly to success rate detection, in that it relies on success rate data from each host in an Envoy Cluster. However, rather than compare those values to the mean success rate of the Cluster as a whole, they are compared to a flat user-configured threshold. This threshold is configured via the failurePercentageThreshold field.

The other configuration fields for failure percentage based detection are similar to the fields for success rate detection. As with success rate detection, detection will not be performed for a host if its request volume over the aggregation interval is less than the failurePercentage.requestVolume value.

Detection also will not be performed for an Envoy Cluster if the number of hosts with the minimum required request volume in an interval is less than the failurePercentage.minimumHosts value.

Default mode is when splitExternalAndLocalErrors is not set or equal false

This detection type takes into account all types of errors: locally and externally originated.

Split Mode is when splitExternalAndLocalErrors is equal true

Locally originated errors and externally originated (transaction) errors are counted and treated separately.

Configuration

  • failurePercentage.requestVolume - The minimum number of hosts in an Envoy Cluster in order to perform failure percentage-based ejection. If the total number of hosts in the Cluster is less than this value, failure percentage-based ejection will not be performed.
  • failurePercentage.minimumHosts - The minimum number of total requests that must be collected in one interval (as defined by the interval duration above) to perform failure percentage-based ejection for this host. If the volume is lower than this setting, failure percentage-based ejection will not be performed for this host.
  • failurePercentage.threshold - The failure percentage to use when determining failure percentage-based outlier detection. If the failure percentage of a given host is greater than or equal to this value, it will be ejected.

Example

  1. type: MeshCircuitBreaker
  2. mesh: default
  3. name: circuit-breaker
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   to:
  8.   - targetRef:
  9.       kind: Mesh
  10.     default:
  11.       outlierDetection:
  12.         splitExternalAndLocalErrors: true
  13.         detectors:
  14.           failurePercentage:
  15.             requestVolume: 10
  16.             minimumHosts: 5
  17.             threshold: 85

Examples

Basic circuit breaker for outbound traffic from web, to backend service

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshCircuitBreaker
  3. metadata:
  4.   name: web-to-backend-circuit-breaker
  5.   namespace: kuma-demo
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: MeshSubset
  11.     tags:
  12.       app: web
  13.   to:
  14.   - targetRef:
  15.       kind: MeshService
  16.       name: backend_kuma-demo_svc_8080
  17.     default:
  18.       connectionLimits:
  19.         maxConnections: 2
  20.         maxPendingRequests: 8
  21.         maxRetries: 2
  22.         maxRequests: 2
  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshCircuitBreaker
  3. metadata:
  4.   name: web-to-backend-circuit-breaker
  5.   namespace: kuma-demo
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: MeshSubset
  11.     tags:
  12.       app: web
  13.   to:
  14.   - targetRef:
  15.       kind: MeshService
  16.       name: backend
  17.       namespace: kuma-demo
  18.       sectionName: http
  19.     default:
  20.       connectionLimits:
  21.         maxConnections: 2
  22.         maxPendingRequests: 8
  23.         maxRetries: 2
  24.         maxRequests: 2

I am using MeshService

  1. type: MeshCircuitBreaker
  2. name: web-to-backend-circuit-breaker
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     tags:
  8.       app: web
  9.   to:
  10.   - targetRef:
  11.       kind: MeshService
  12.       name: backend
  13.     default:
  14.       connectionLimits:
  15.         maxConnections: 2
  16.         maxPendingRequests: 8
  17.         maxRetries: 2
  18.         maxRequests: 2
  1. type: MeshCircuitBreaker
  2. name: web-to-backend-circuit-breaker
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     tags:
  8.       app: web
  9.   to:
  10.   - targetRef:
  11.       kind: MeshService
  12.       name: backend
  13.       sectionName: http
  14.     default:
  15.       connectionLimits:
  16.         maxConnections: 2
  17.         maxPendingRequests: 8
  18.         maxRetries: 2
  19.         maxRequests: 2

Outlier detection for inbound traffic to backend service

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshCircuitBreaker
  3. metadata:
  4.   name: backend-inbound-outlier-detection
  5.   namespace: kuma-demo
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: MeshSubset
  11.     tags:
  12.       app: web
  13.   from:
  14.   - targetRef:
  15.       kind: Mesh
  16.     default:
  17.       outlierDetection:
  18.         interval: 5s
  19.         baseEjectionTime: 30s
  20.         maxEjectionPercent: 20
  21.         splitExternalAndLocalErrors: true
  22.         detectors:
  23.           totalFailures:
  24.             consecutive: 10
  25.           gatewayFailures:
  26.             consecutive: 10
  27.           localOriginFailures:
  28.             consecutive: 10
  29.           successRate:
  30.             minimumHosts: 5
  31.             requestVolume: 10
  32.             standardDeviationFactor: 1.9
  33.           failurePercentage:
  34.             requestVolume: 10
  35.             minimumHosts: 5
  36.             threshold: 85
  1. type: MeshCircuitBreaker
  2. name: backend-inbound-outlier-detection
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     tags:
  8.       app: web
  9.   from:
  10.   - targetRef:
  11.       kind: Mesh
  12.     default:
  13.       outlierDetection:
  14.         interval: 5s
  15.         baseEjectionTime: 30s
  16.         maxEjectionPercent: 20
  17.         splitExternalAndLocalErrors: true
  18.         detectors:
  19.           totalFailures:
  20.             consecutive: 10
  21.           gatewayFailures:
  22.             consecutive: 10
  23.           localOriginFailures:
  24.             consecutive: 10
  25.           successRate:
  26.             minimumHosts: 5
  27.             requestVolume: 10
  28.             standardDeviationFactor: 1.9
  29.           failurePercentage:
  30.             requestVolume: 10
  31.             minimumHosts: 5
  32.             threshold: 85

All policy options