Configuring your Mesh and multi-tenancy

This resource describes a very important concept in Kuma, and that is the ability of creating multiple isolated service meshes within the same Kuma cluster which in turn make Kuma a very simple and easy project to operate in environments where more than one mesh is required based on security, segmentation or governance requirements.

Typically, we would want to create a Mesh per line of business, per team, per application or per environment or for any other reason. Typically multiple meshes are being created so that a service mesh can be adopted by an organization with a gradual roll-out that doesn’t require all the teams and their applications to coordinate with each other, or as an extra layer of security and segmentation for our services so that - for example - policies applied to one Mesh do not affect another Mesh.

Mesh is the parent resource of every other resource in Kuma, including:

In order to use Kuma at least one Mesh must exist, and there is no limit to the number of Meshes that can be created. When a data plane proxy connects to the control plane (kuma-cp) it specifies to what Mesh resource it belongs: a data plane proxy can only belong to one Mesh at a time.

When starting a new Kuma cluster from scratch a default Mesh is being created automatically.

Besides the ability of being able to create virtual service mesh, a Mesh resource will also be used for:

  • Mutual TLS, to secure and encrypt our service traffic and assign an identity to the data plane proxies within the Mesh.

  • Zone Egress, to setup if ZoneEgress should be used for cross zone and external service communication.

  • Non-mesh traffic, to setup if passthrough mode should be used for the non-mesh traffic.

To support cross-mesh communication an intermediate API Gateway must be used. Kuma checkout Kuma’s builtin gateway to set this up.

Previously, observability and locality awareness were configured within the Mesh object.

However, for enhanced flexibility and granular control, these configurations have been extracted into separate policies: MeshAccessLog, MeshTrace and MeshMetric for observability, and MeshLoadBalancingStrategy for locality awareness.

This separation allows for more fine-grained adjustments of each aspect, ensuring that observability and locality awareness are tailored to specific requirements.

Usage

The easiest way to create a Mesh is to specify its name. The name of a Mesh must be unique.

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default

We will apply the configuration with kubectl apply -f [..].

  1. type: Mesh
  2. name: default

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

Creating resources in a Mesh

It is possible to determine to what Mesh other resources belong to in the following ways.

Data plane proxies

Every time we start a data plane proxy, we need to specify to what Mesh it belongs, this can be done in the following way:

By using the kuma.io/mesh annotation in a Deployment, like:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: example-app
  5.   namespace: kuma-example
  6. spec:
  7.   ...
  8.   template:
  9.     metadata:
  10.       ...
  11.       annotations:
  12.         # indicate to Kuma what is the Mesh that the data plane proxy belongs to
  13.         kuma.io/mesh: default
  14.     spec:
  15.       containers:
  16.         ...

A Mesh may span multiple Kubernetes namespaces. Any Kuma resource in the cluster which specifies a particular Mesh will be part of that Mesh.

By using the -m or --mesh argument when running kuma-dp, for example:

  1. kuma-dp run \
  2.   --name=backend-1 \
  3.   --mesh=default \
  4.   --cp-address=https://127.0.0.1:5678 \
  5.   --dataplane-token-file=/tmp/kuma-dp-backend-1-token

You can control which data plane proxies are allowed to join the mesh using mesh constraints.

Policies

When creating new Policies we also must specify to what Mesh they belong. This can be done in the following way:

By using the kuma.io/mesh label, like:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshHTTPRoute
  3. metadata:
  4.   name: route-1
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default # indicate to Kuma what is the Mesh that the resource belongs to
  8. spec:
  9.   ...

Kuma consumes all Policies on the cluster and joins each to an individual Mesh, identified by this property.

By using the mesh property, like:

  1. type: MeshHTTPRoute
  2. name: route-1
  3. mesh: default # indicate to Kuma what is the Mesh that the resource belongs to
  4. ...

Skipping default resource creation

By default, to help users get started we create the following default policies:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshTimeout
  3. metadata:
  4.   name: mesh-gateways-timeout-all-default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: MeshSubset
  11.     proxyTypes:
  12.     - Gateway
  13.   to:
  14.   - targetRef:
  15.       kind: Mesh
  16.     default:
  17.       idleTimeout: 1h
  18.       http:
  19.         streamIdleTimeout: 5s
  20.   from:
  21.   - targetRef:
  22.       kind: Mesh
  23.     default:
  24.       idleTimeout: 5m
  25.       http:
  26.         streamIdleTimeout: 5s
  27.         requestHeadersTimeout: 500ms
  28. ---
  29. apiVersion: kuma.io/v1alpha1
  30. kind: MeshTimeout
  31. metadata:
  32.   name: mesh-timeout-all-default
  33.   namespace: kuma-system
  34.   labels:
  35.     kuma.io/mesh: default
  36. spec:
  37.   targetRef:
  38.     kind: Mesh
  39.     proxyTypes:
  40.     - Sidecar
  41.   to:
  42.   - targetRef:
  43.       kind: Mesh
  44.     default:
  45.       connectionTimeout: 5s
  46.       idleTimeout: 1h
  47.       http:
  48.         requestTimeout: 15s
  49.         streamIdleTimeout: 30m
  50.   from:
  51.   - targetRef:
  52.       kind: Mesh
  53.     default:
  54.       connectionTimeout: 10s
  55.       idleTimeout: 2h
  56.       http:
  57.         requestTimeout: 0s
  58.         streamIdleTimeout: 1h
  59.         maxStreamDuration: 0s
  60. ---
  61. apiVersion: kuma.io/v1alpha1
  62. kind: MeshRetry
  63. metadata:
  64.   name: mesh-retry-all-default
  65.   namespace: kuma-system
  66.   labels:
  67.     kuma.io/mesh: default
  68. spec:
  69.   targetRef:
  70.     kind: Mesh
  71.   to:
  72.   - targetRef:
  73.       kind: Mesh
  74.     default:
  75.       tcp:
  76.         maxConnectAttempt: 5
  77.       http:
  78.         numRetries: 5
  79.         perTryTimeout: 16s
  80.         backOff:
  81.           baseInterval: 25ms
  82.           maxInterval: 250ms
  83.       grpc:
  84.         numRetries: 5
  85.         perTryTimeout: 16s
  86.         backOff:
  87.           baseInterval: 25ms
  88.           maxInterval: 250ms
  89. ---
  90. apiVersion: kuma.io/v1alpha1
  91. kind: MeshCircuitBreaker
  92. metadata:
  93.   name: mesh-circuit-breaker-all-default
  94.   namespace: kuma-system
  95.   labels:
  96.     kuma.io/mesh: default
  97. spec:
  98.   targetRef:
  99.     kind: Mesh
  100.   to:
  101.   - targetRef:
  102.       kind: Mesh
  103.     default:
  104.       connectionLimits:
  105.         maxConnections: 1024
  106.         maxPendingRequests: 1024
  107.         maxRetries: 3
  108.         maxRequests: 1024
  1. type: MeshTimeout
  2. mesh: default
  3. name: mesh-gateways-timeout-all-default
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     proxyTypes:
  8.     - Gateway
  9.   to:
  10.   - targetRef:
  11.       kind: Mesh
  12.     default:
  13.       idleTimeout: 1h
  14.       http:
  15.         streamIdleTimeout: 5s
  16.   from:
  17.   - targetRef:
  18.       kind: Mesh
  19.     default:
  20.       idleTimeout: 5m
  21.       http:
  22.         streamIdleTimeout: 5s
  23.         requestHeadersTimeout: 500ms
  24. ---
  25. type: MeshTimeout
  26. mesh: default
  27. name: mesh-timeout-all-default
  28. spec:
  29.   targetRef:
  30.     kind: Mesh
  31.     proxyTypes:
  32.     - Sidecar
  33.   to:
  34.   - targetRef:
  35.       kind: Mesh
  36.     default:
  37.       connectionTimeout: 5s
  38.       idleTimeout: 1h
  39.       http:
  40.         requestTimeout: 15s
  41.         streamIdleTimeout: 30m
  42.   from:
  43.   - targetRef:
  44.       kind: Mesh
  45.     default:
  46.       connectionTimeout: 10s
  47.       idleTimeout: 2h
  48.       http:
  49.         requestTimeout: 0s
  50.         streamIdleTimeout: 1h
  51.         maxStreamDuration: 0s
  52. ---
  53. type: MeshRetry
  54. mesh: default
  55. name: mesh-retry-all-default
  56. spec:
  57.   targetRef:
  58.     kind: Mesh
  59.   to:
  60.   - targetRef:
  61.       kind: Mesh
  62.     default:
  63.       tcp:
  64.         maxConnectAttempt: 5
  65.       http:
  66.         numRetries: 5
  67.         perTryTimeout: 16s
  68.         backOff:
  69.           baseInterval: 25ms
  70.           maxInterval: 250ms
  71.       grpc:
  72.         numRetries: 5
  73.         perTryTimeout: 16s
  74.         backOff:
  75.           baseInterval: 25ms
  76.           maxInterval: 250ms
  77. ---
  78. type: MeshCircuitBreaker
  79. mesh: default
  80. name: mesh-circuit-breaker-all-default
  81. spec:
  82.   targetRef:
  83.     kind: Mesh
  84.   to:
  85.   - targetRef:
  86.       kind: Mesh
  87.     default:
  88.       connectionLimits:
  89.         maxConnections: 1024
  90.         maxPendingRequests: 1024
  91.         maxRetries: 3
  92.         maxRequests: 1024

If you want to not have these policies be added on creation of the mesh set the configuration: skipCreatingInitialPolicies:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   skipCreatingInitialPolicies: ['*']
  1. type: Mesh
  2. name: default
  3. skipCreatingInitialPolicies: ['*']

You can also skip creating the default mesh by setting the control-plane configuration: KUMA_DEFAULTS_SKIP_MESH_CREATION=true.

All options