Migration to the new policies

Kuma provides two set of policies to configure proxies. The original source/destination policies, while provided a lot of features, haven’t met users expectations in terms of flexibility and transparency. The new targetRef policies were designed to preserve what already worked well, and enhance the matching functionality and overall UX.

In this guide, we’re going to setup a demo with old policies and then perform a migration to the new policies.

Prerequisites

  • Helm - a package manager for Kubernetes
  • Kind - a tool for running local Kubernetes clusters
  • jq - a command-line JSON processor
  • jd - a command-line util to visualise JSONPatch

Start Kubernetes cluster

Start a new Kubernetes cluster on your local machine by executing:

  1. kind create cluster --name=mesh-zone

You can skip this step if you already have a Kubernetes cluster running. It can be a cluster running locally or in a public cloud like AWS EKS, GCP GKE, etc.

Install Kuma

Install Kuma control plane with skipMeshCreation set to true by executing:

  1. helm repo add kuma https://kumahq.github.io/charts
  2. helm repo update
  3. helm install --create-namespace --namespace kuma-system kuma kuma/kuma --set "controlPlane.defaults.skipMeshCreation=true"

Make sure the list of meshes is empty:

  1. kubectl get meshes

Expected output:

  1. No resources found

Setup demo with old policies

In the first half of this guide we’re going to deploy a demo app in the default mesh and configure it using old policies.

Create default mesh

  1. echo 'apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   # for the purpose of this guide we want to setup mesh with old policies first,
  7.   # that is why we are skipping the default policies creation
  8.   skipCreatingInitialPolicies: ["*"] ' | kubectl apply -f-

Deploy demo application

  1. Deploy the application

    1. kubectl apply -f https://raw.githubusercontent.com/kumahq/kuma-counter-demo/master/demo.yaml
    2. kubectl wait -n kuma-demo --for=condition=ready pod --selector=app=demo-app --timeout=90s

  2. Port-forward the service to the namespace on port 5000:

    1. kubectl port-forward svc/demo-app -n kuma-demo 5000:5000

  3. In a browser, go to 127.0.0.1:5000 and increment the counter.

Enable Mutual TLS and Traffic Permissions

  1. echo 'apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   skipCreatingInitialPolicies: ["*"]
  7.   mtls:
  8.     enabledBackend: ca-1
  9.     backends:
  10.       - name: ca-1
  11.         type: builtin' | kubectl apply -f-
  1. echo 'apiVersion: kuma.io/v1alpha1
  2. kind: TrafficPermission
  3. mesh: default
  4. metadata:
  5.   name: app-to-redis
  6. spec:
  7.   sources:
  8.     - match:
  9.         kuma.io/service: demo-app_kuma-demo_svc_5000
  10.   destinations:
  11.     - match:
  12.         kuma.io/service: redis_kuma-demo_svc_6379' | kubectl apply -f -

Deploy TrafficRoute

  1. echo 'apiVersion: kuma.io/v1alpha1
  2. kind: TrafficRoute
  3. mesh: default
  4. metadata:
  5.   name: route-all-default
  6. spec:
  7.   sources:
  8.     - match:
  9.         kuma.io/service: "*"
  10.   destinations:
  11.     - match:
  12.         kuma.io/service: "*"
  13.   conf:
  14.     destination:
  15.       kuma.io/service: "*"' | kubectl apply -f-

Deploy Timeouts

  1. echo 'apiVersion: kuma.io/v1alpha1
  2. kind: Timeout
  3. mesh: default
  4. metadata:
  5.   name: timeout-global
  6. spec:
  7.   sources:
  8.     - match:
  9.         kuma.io/service: "*"
  10.   destinations:
  11.     - match:
  12.         kuma.io/service: "*"
  13.   conf:
  14.     connectTimeout: 21s
  15.     tcp:
  16.       idleTimeout: 22s
  17.     http:
  18.       idleTimeout: 22s
  19.       requestTimeout: 23s
  20.       streamIdleTimeout: 25s
  21.       maxStreamDuration: 26s' | kubectl apply -f-

Deploy CircuitBreaker

  1. echo 'apiVersion: kuma.io/v1alpha1
  2. kind: CircuitBreaker
  3. mesh: default
  4. metadata:
  5.   name: cb-global
  6. spec:
  7.   sources:
  8.   - match:
  9.       kuma.io/service: "*"
  10.   destinations:
  11.   - match:
  12.       kuma.io/service: "*"
  13.   conf:
  14.     interval: 21s
  15.     baseEjectionTime: 22s
  16.     maxEjectionPercent: 23
  17.     splitExternalAndLocalErrors: false
  18.     thresholds:
  19.       maxConnections: 24
  20.       maxPendingRequests: 25
  21.       maxRequests: 26
  22.       maxRetries: 27
  23.     detectors:
  24.       totalErrors:
  25.         consecutive: 28
  26.       gatewayErrors:
  27.         consecutive: 29
  28.       localErrors:
  29.         consecutive: 30
  30.       standardDeviation:
  31.         requestVolume: 31
  32.         minimumHosts: 32
  33.         factor: 1.33
  34.       failure:
  35.         requestVolume: 34
  36.         minimumHosts: 35
  37.         threshold: 36' | kubectl apply -f-

Migration steps

It’s time to migrate the demo app to the new policies.

Each type of policy can be migrated separately; for example, once we have completely finished with the Timeouts, we will proceed to the next policy type, CircuitBreakers. It’s possible to migrate all policies at once, but small portions are preferable as they’re easily reversible.

The generalized migration process roughly consists of 4 steps:

  1. Create a new targetRef policy as a replacement for exising source/destination policy (do not forget about default policies that might not be stored in your source control). The corresponding new policy type can be found in the table. Deploy the policy in shadow mode to avoid any traffic disruptions.
  2. Using Inspect API review the list of changes that are going to be created by the new policy.
  3. Remove kuma.io/effect: shadow label so that policy is applied in a normal mode.
  4. Observe metrics, traces and logs. If something goes wrong change policy’s mode back to shadow and return to the step 2. If everything is fine then remove the old policies.

The order of migrating policies generally doesn’t matter, except for the TrafficRoute policy, which should be the last one deleted when removing old policies. This is because many old policies, like Timeout and CircuitBreaker, depend on TrafficRoutes to function correctly.

TrafficPermission -> MeshTrafficPermission

  1. Create a replacement policy for app-to-redis TrafficPermission and apply it with kuma.io/effect: shadow label:

    1. echo 'apiVersion: kuma.io/v1alpha1
    2. kind: MeshTrafficPermission
    3. metadata:
    4.   namespace: kuma-system
    5.   name: app-to-redis
    6.   labels:
    7.     kuma.io/mesh: default
    8.     kuma.io/effect: shadow
    9. spec:
    10.   targetRef:
    11.     kind: MeshService
    12.     name: redis_kuma-demo_svc_6379
    13.   from:
    14.     - targetRef:
    15.         kind: MeshSubset
    16.         tags:
    17.           kuma.io/service: demo-app_kuma-demo_svc_5000
    18.       default:
    19.         action: Allow' | kubectl apply -f -

  2. Check the list of changes for redis_kuma-demo_svc_6379 pod in Envoy configuration using kumactl, jq and jd:

    1. kumactl inspect dataplane redis-8fcbfc795-twlst.kuma-demo --type=config --shadow --include=diff | jq '.diff' | jd -t patch2jd

    Expected output:

    1. @ ["type.googleapis.com/envoy.config.listener.v3.Listener","inbound:10.42.0.13:6379","filterChains","0","filters","0","typedConfig","rules","policies","allow-all-default"]
    2. - {"permissions":[{"any":true}],"principals":[{"authenticated":{"principalName":{"exact":"spiffe://default/demo-app_kuma-demo_svc_5000"}}}]}
    3. @ ["type.googleapis.com/envoy.config.listener.v3.Listener","inbound:10.42.0.13:6379","filterChains","0","filters","0","typedConfig","rules","policies","MeshTrafficPermission"]
    4. + {"permissions":[{"any":true}],"principals":[{"authenticated":{"principalName":{"exact":"spiffe://default/demo-app_kuma-demo_svc_5000"}}}]}

    As we can see, the only difference is the policy name “MeshTrafficPermission” instead of “allow-all-default”. The value of the policy is the same.

  3. Remove the kuma.io/effect: shadow label:

    1. echo 'apiVersion: kuma.io/v1alpha1
    2. kind: MeshTrafficPermission
    3. metadata:
    4.   namespace: kuma-system
    5.   name: app-to-redis
    6.   labels:
    7.     kuma.io/mesh: default
    8. spec:
    9.   targetRef:
    10.     kind: MeshService
    11.     name: redis_kuma-demo_svc_6379
    12.   from:
    13.     - targetRef:
    14.         kind: MeshSubset
    15.         tags:
    16.           kuma.io/service: demo-app_kuma-demo_svc_5000
    17.       default:
    18.         action: Allow' | kubectl apply -f -

    Even though the old TrafficPermission and the new MeshTrafficPermission are both in use, the new policy takes precedence, making the old one ineffective.

  4. Observe the demo app behaves as expected. If everything goes well, we can safely remove TrafficPermission and conclude the migration.

Timeout -> MeshTimeout

  1. Create a replacement policy for timeout-global Timeout and apply it with kuma.io/effect: shadow label:

    1. echo 'apiVersion: kuma.io/v1alpha1
    2. kind: MeshTimeout
    3. metadata:
    4.   namespace: kuma-system
    5.   name: timeout-global
    6.   labels:
    7.     kuma.io/mesh: default
    8.     kuma.io/effect: shadow
    9. spec:
    10.   targetRef:
    11.     kind: Mesh
    12.   to:
    13.   - targetRef:
    14.       kind: Mesh
    15.     default:
    16.       connectionTimeout: 21s
    17.       idleTimeout: 22s
    18.       http:
    19.         requestTimeout: 23s
    20.         streamIdleTimeout: 25s
    21.         maxStreamDuration: 26s
    22.   from:
    23.   - targetRef:
    24.       kind: Mesh
    25.     default:
    26.       connectionTimeout: 10s
    27.       idleTimeout: 2h
    28.       http:
    29.         requestTimeout: 0s
    30.         streamIdleTimeout: 2h' | kubectl apply -f-

  2. Check the list of changes for redis_kuma-demo_svc_6379 pod in Envoy configuration using kumactl, jq and jd:

    1. kumactl inspect dataplane redis-8fcbfc795-twlst.kuma-demo --type=config --shadow --include=diff | jq '.diff' | jd -t patch2jd

    Expected output:

    1. @ ["type.googleapis.com/envoy.config.cluster.v3.Cluster","demo-app_kuma-demo_svc_5000","typedExtensionProtocolOptions","envoy.extensions.upstreams.http.v3.HttpProtocolOptions","commonHttpProtocolOptions","maxConnectionDuration"]
    2. + "0s"
    3. @ ["type.googleapis.com/envoy.config.listener.v3.Listener","outbound:10.43.146.6:5000","filterChains","0","filters","0","typedConfig","commonHttpProtocolOptions","idleTimeout"]
    4. - "22s"
    5. @ ["type.googleapis.com/envoy.config.listener.v3.Listener","outbound:10.43.146.6:5000","filterChains","0","filters","0","typedConfig","commonHttpProtocolOptions","idleTimeout"]
    6. + "0s"
    7. @ ["type.googleapis.com/envoy.config.listener.v3.Listener","outbound:10.43.146.6:5000","filterChains","0","filters","0","typedConfig","routeConfig","virtualHosts","0","routes","0","route","idleTimeout"]
    8. + "25s"
    9. @ ["type.googleapis.com/envoy.config.listener.v3.Listener","outbound:10.43.146.6:5000","filterChains","0","filters","0","typedConfig","requestHeadersTimeout"]
    10. + "0s"

    Review the list and ensure the new MeshTimeout policy won’t change the important settings. The key differences between old and new timeout policies:

    • Previously, there was no way to specify requestHeadersTimeout, maxConnectionDuration and maxStreamDuration (on inbound). These timeouts were unset. With the new MeshTimeout policy we explicitly set them to 0s by default.
    • idleTimeout was configured both on the cluster and listener. MeshTimeout configures it only on the cluster.
    • route/idleTimeout is duplicated value of streamIdleTimeout but per-route. Previously we’ve set it only per-listener.

    These 3 facts perfectly explain the list of changes we’re observing.

  3. Remove the kuma.io/effect: shadow label. Even though the old Timeout and the new MeshTimeout are both in use, the new policy takes precedence, making the old one ineffective.

  4. Observe the demo app behaves as expected. If everything goes well, we can safely remove Timeout and conclude the migration.

CircuitBreaker -> MeshCircuitBreaker

  1. Create a replacement policy for cb-global CircutBreaker and apply it with kuma.io/effect: shadow label:

    1. echo 'apiVersion: kuma.io/v1alpha1
    2. kind: MeshCircuitBreaker
    3. metadata:
    4.   namespace: kuma-system
    5.   name: cb-global
    6.   labels:
    7.     kuma.io/mesh: default
    8.     kuma.io/effect: shadow
    9. spec:
    10.   targetRef:
    11.     kind: Mesh
    12.   to:
    13.   - targetRef:
    14.       kind: Mesh
    15.     default:
    16.       connectionLimits:
    17.         maxConnections: 24
    18.         maxPendingRequests: 25
    19.         maxRequests: 26
    20.         maxRetries: 27
    21.       outlierDetection:
    22.         interval: 21s
    23.         baseEjectionTime: 22s
    24.         maxEjectionPercent: 23
    25.         splitExternalAndLocalErrors: false
    26.         detectors:
    27.           totalFailures:
    28.             consecutive: 28
    29.           gatewayFailures:
    30.             consecutive: 29
    31.           localOriginFailures:
    32.             consecutive: 30
    33.           successRate:
    34.             requestVolume: 31
    35.             minimumHosts: 32
    36.             standardDeviationFactor: "1.33"
    37.           failurePercentage:
    38.             requestVolume: 34
    39.             minimumHosts: 35
    40.             threshold: 36' | kubectl apply -f-

  2. Check the list of changes for redis_kuma-demo_svc_6379 pod in Envoy configuration using kumactl, jq and jd:

    1. kumactl inspect dataplane demo-app-b4f98898-zxrqj.kuma-demo --type=config --shadow --include=diff | jq '.diff' | jd -t patch2jd

    The expected output is empty. CircuitBreaker and MeshCircuitBreaker configures Envoy in the exact similar way.

  3. Remove the kuma.io/effect: shadow label. Even though the old CircuitBreaker and the new MeshCircuitBreaker are both in use, the new policy takes precedence, making the old one ineffective.

  4. Observe the demo app behaves as expected. If everything goes well, we can safely remove CircuitBreaker and conclude the migration.

TrafficRoute -> MeshTCPRoute

It’s safe to simply remove route-all-default TrafficRoute. Traffic will flow through the system even if there are neither TrafficRoutes nor MeshTCPRoutes/MeshHTTPRoutes.

MeshGatewayRoute -> MeshHTTPRoute/MeshTCPRoute

The biggest change is that there are now 2 protocol specific routes, one for TCP and one for HTTP. MeshHTTPRoute always takes precedence over MeshTCPRoute if both exist.

Otherwise the high-level structure of the routes hasn’t changed, though there are a number of details to consider. Some enum values and some field structures were updated, largely to reflect Gateway API.

Please first read the MeshGatewayRoute docs, the MeshHTTPRoute docs and the MeshTCPRoute docs. Always refer to the spec to ensure your new resource is valid.

Note that MeshHTTPRoute has precedence over MeshGatewayRoute.

Targeting

The main consideration is specifying which gateways are affected by the route. The most important change is that instead of solely using tags to select MeshGateway listeners, new routes target MeshGateways by name and optionally with tags for specific listeners.

For example:

  1. spec:
  2.   selectors:
  3.     - match:
  4.         kuma.io/service: edge-gateway
  5.         vhost: foo.example.com

becomes:

  1. spec:
  2.   targetRef:
  3.     kind: MeshGateway
  4.     name: edge-gateway # where this MeshGateway has `kuma.io/service: edge-gateway`
  5.     tags:
  6.       vhost: foo.example.com
  7.   to:

Spec

As with all new policies, the spec is now merged under a default field. MeshTCPRoute is very simple, so the rest of this is focused on MeshHTTPRoute.

Note that for MeshHTTPRoute the hostnames are directly under the to entry:

  1.   conf:
  2.     http:
  3.       hostnames:
  4.         - example.com
  5.       rules:
  6.         - matches:
  7.             - path:
  8.                 match: PREFIX
  9.                 value: /
  10.           # ...
  1.   to:
  2.     - targetRef:
  3.         kind: Mesh
  4.       hostnames:
  5.         - example.com
  6.       rules:
  7.         - matches:
  8.             - path:
  9.                 match: PathPrefix
  10.                 value: /
  11.           default:
  12.             # ...
Matching

Matching works the same as before. Remember that for MeshHTTPRoute that merging is done on a match basis. So it’s possible for one route to define filters and another backendRefs for a given match, and the resulting rule would both apply the filters and route to the backends:

  1.   to:
  2.     - targetRef:
  3.         kind: Mesh
  4.       rules:
  5.         - matches:
  6.             - path:
  7.                 match: PathPrefix
  8.                 value: /
  9.           default:
  10.             filters:
  11.               - type: RequestHeaderModifier
  12.                 requestHeaderModifier:
  13.                   set:
  14.                     - name: x-custom-header
  15.                       value: xyz
  1.   to:
  2.     - targetRef:
  3.         kind: Mesh
  4.       hostnames:
  5.         - example.com
  6.       rules:
  7.         - matches:
  8.             - path:
  9.                 match: PathPrefix
  10.                 value: /
  11.           default:
  12.             backendRefs:
  13.               - kind: MeshServiceSubset
  14.                 name: backend_kuma-demo_svc_3001
  15.                 tags:
  16.                   version: v0

Traffic to / would have the x-custom-header added and be sent to the v0 versions of backend_kuma-demo_svc_3001.

Filters

Every MeshGatewayRoute filter has an equivalent in MeshHTTPRoute. Consult the documentation for both resources to find out how each filter looks in MeshHTTPRoute.

Backends

Backends are similar except that instead of targeting with tags, the targetRef structure with kind: MeshService/kind: MeshServiceSubset is used.

Next steps