MeshRetry (beta)

This policy uses new policy matching algorithm and is in beta state, it should not be mixed with Retry.

This policy enables Kuma to know how to behave if there is a failed scenario (i.e. HTTP request) which could be retried.

TargetRef support matrix

targetRef.kindtop leveltofrom
Mesh✅✅❌
MeshSubset✅❌❌
MeshService✅✅❌
MeshServiceSubset✅❌❌

To learn more about the information in this table, see the matching docs.

Configuration

The policy let you configure retry behaviour for HTTP, GRPC and TCP protocols. The protocol is selected by picking the most specific protocol.

Each protocol has a separate section under default in the policy YAML. Some sections are common between protocols or have similar meaning.

Retry on

The field retryOn is a list of conditions which will cause a retry.

For HTTP these are related to the response status code or method (“5xx”, “429”, “HttpMethodGet”). For gRPC these are status codes in response headers (“canceled”, “deadline-exceeded”, etc.). There is no equivalent for TCP.

One or more conditions can be specified, for example:

  1. retryOn:
  2.   - "429"
  3.   - "503"

means that it the policy will retry on a status code 429 or 503.

Full list of available HTTP conditions:

  1. retryOn:
  2.   - 5XX
  3.   - GatewayError
  4.   - Reset
  5.   - Retriable4xx
  6.   - ConnectFailure
  7.   - EnvoyRatelimited
  8.   - RefusedStream
  9.   - Http3PostConnectFailure
  10.   - HttpMethodConnect
  11.   - HttpMethodDelete
  12.   - HttpMethodGet
  13.   - HttpMethodHead
  14.   - HttpMethodOptions
  15.   - HttpMethodPatch
  16.   - HttpMethodPost
  17.   - HttpMethodPut
  18.   - HttpMethodTrace
  19.   - "429" # any HTTP status code
  20.   - "503"

Full list of available gRPC conditions:

  1. retryOn:
  2.   - Canceled
  3.   - DeadlineExceeded
  4.   - Internal
  5.   - ResourceExhausted
  6.   - Unavailable

Backoff

This parameter is applicable to both HTTP and GRPC.

It consists of BaseInterval (the amount of time between retries) and MaxInterval (the maximal amount of time taken between retries).

We use a fully jittered exponential back-off algorithm for retries. Given a base interval B and retry number N, the back-off for the retry is in the range [0, (2N - 1) × B).

For example, given a 25ms interval, the first retry will be delayed randomly by 0-24ms, the 2nd by 0-74ms, the 3rd by 0-174ms, and so on.

The interval is capped at a MaxInterval, which defaults to 10 times the BaseInterval.

Rate limited backoff

This parameter is applicable to both HTTP and GRPC.

MeshRetry can be configured in such a way that when the upstream server rate limits the request and responds with a header like retry-after or x-ratelimit-reset it uses the value from the header to determine when to send the retry request instead of the backoff algorithm.

Example

Given this configuration:

  1. retryOn:
  2.   - "503"
  3. rateLimitedBackOff:
  4.   resetHeaders:
  5.     - name: retry-after
  6.       format: Seconds
  7.     - name: x-ratelimit-reset
  8.       format: UnixTimestamp

and an HTTP response:

  1. HTTP/1.1 503 Service Unavailable
  2. retry-after: 15

The retry request will be issued after 15 seconds.

If the response is as follows:

  1. HTTP/1.1 503 Service Unavailable
  2. x-ratelimit-reset: 1706096119

The request will be retried at Wed Jan 24 2024 11:35:19 GMT+0000.

If the response does not contain retry-after or x-ratelimit-reset header (with valid integer value) then the amount of time to wait before issuing a request is determined by backoff algorithm.

Examples

HTTP web to backend on 5xx

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshRetry
  3. metadata:
  4.   name: web-to-backend-retry-http
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if unset
  8. spec:
  9.   targetRef:
  10.     kind: MeshService
  11.     name: web
  12.   to:
  13.     - targetRef:
  14.         kind: MeshService
  15.         name: backend
  16.       default:
  17.         http:
  18.           numRetries: 10
  19.           backOff:
  20.             baseInterval: 15s
  21.             maxInterval: 20m
  22.           retryOn:
  23.             - "5xx"

Apply the configuration with kubectl apply -f [..].

  1. type: MeshRetry
  2. name: web-to-backend-retry-http
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshService
  7.     name: web
  8.   to:
  9.     - targetRef:
  10.         kind: MeshService
  11.         name: backend
  12.       default:
  13.         http:
  14.           numRetries: 10
  15.           backOff:
  16.             baseInterval: 15s
  17.             maxInterval: 20m
  18.           retryOn:
  19.             - "5xx"

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

gRPC web to backend on DeadlineExceeded

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshRetry
  3. metadata:
  4.   name: web-to-backend-retry-grpc
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if unset
  8. spec:
  9.   targetRef:
  10.     kind: MeshService
  11.     name: web
  12.   to:
  13.     - targetRef:
  14.         kind: MeshService
  15.         name: backend
  16.       default:
  17.         grpc:
  18.           numRetries: 5
  19.           backOff:
  20.             baseInterval: 5s
  21.             maxInterval: 1m
  22.           retryOn:
  23.             - "DeadlineExceeded"

Apply the configuration with kubectl apply -f [..].

  1. type: MeshRetry
  2. name: web-to-backend-retry-grpc
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshService
  7.     name: web
  8.   to:
  9.     - targetRef:
  10.         kind: MeshService
  11.         name: backend
  12.       default:
  13.         grpc:
  14.           numRetries: 5
  15.           backOff:
  16.             baseInterval: 5s
  17.             maxInterval: 1m
  18.           retryOn:
  19.             - "DeadlineExceeded"

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

TCP web to backend

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshRetry
  3. metadata:
  4.   name: web-to-backend-retry-tcp
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # optional, defaults to `default` if unset
  8. spec:
  9.   targetRef:
  10.     kind: MeshService
  11.     name: web
  12.   to:
  13.     - targetRef:
  14.         kind: MeshService
  15.         name: backend
  16.       default:
  17.         tcp:
  18.           maxConnectAttempt: 5

Apply the configuration with kubectl apply -f [..].

  1. type: MeshRetry
  2. name: web-to-backend-retry-tcp
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshService
  7.     name: web
  8.   to:
  9.     - targetRef:
  10.         kind: MeshService
  11.         name: backend
  12.       default:
  13.         tcp:
  14.           maxConnectAttempt: 5

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

All policy options