Secure access across services

This page explains how secure access is provided across a Kuma deployment:

  • Security between our services, via the mTLS policy.
  • Security between the Kuma control plane and its data plane proxies, via the data plane proxy token.
  • Security when accessing the control plane.
  • Security when external system access the control plane.

Kuma stores autogenerated certificates and other files in a working directory. The default value for this directory is $HOME/.kuma. You can change the working directory with KUMA_GENERAL_WORK_DIR.

This section is not to be confused with the mTLS policy that we can apply to a Mesh to secure service-to-service traffic.

Data plane proxy to control plane communication

A data plane proxy connects to the control plane for its configuration, including mTLS certificates described in the following sections.

Encrypted communication

Because the data plane proxy and the control plane exchange sensitive information, the communication needs to be encrypted by TLS. By default, the control plane’s server that is consumed by the data plane proxy is secured by TLS with autogenerated certificates.

It is recommended that the data plane proxy verifies the identity of the control plane. To do so, data plane proxies need to obtain the CA that was used to generate the certificate by which the control plane’s server is secured. Note, this CA is not the same CA for service-to-service communication.

To override autogenerated certificates

  1. Prepare certificates

Generate TLS pair with a PKI of your choice and store it in PEM-encoded format in /tmp/tls.crt, /tmp/tls.key. Store the CA that was used to sign this pair in /tmp/ca.crt

We can also use kumactl to generate self-signed certs.

  1. $ kumactl generate tls-certificate \
  2.   --type=server \
  3.   --cp-hostname=kuma-control-plane.kuma-system \ # adjust the name if you are installing Kuma to other namespace
  4.   --cp-hostname=kuma-control-plane.kuma-system.svc # adjust the name if you are installing Kuma to other namespace
  5.   --cert-file=/tmp/tls.crt \
  6.   --key-file=/tmp/tls.key
  7. $ cp /tmp/tls.crt /tmp/ca.crt # since this is self-signed cert, the cert is also a CA
  1. Create a secret in the namespace where the control plane is installed
  1. $ kubectl create secret generic general-tls-certs -n kuma-system \
  2.   --from-file=tls.crt=/tmp/tls.crt \
  3.   --from-file=tls.key=/tmp/tls.key \
  4.   --from-file=ca.crt=/tmp/ca.crt
  1. Point to this secret when installing Kuma
  1. $ kumactl install control-plane \
  2.   --tls-general-secret=general-tls-certs
  3.   --tls-general-ca-bundle=$(cat /tmp/ca.crt | base64)

The data plane proxy Injector in the control plane automatically provides the CA to the Kuma DP sidecar so Kuma DP can confirm the control plane identity.

  1. Prepare certificates

Generate TLS pair with a PKI of your choice and store it in PEM-encoded format in /tmp/tls.crt, /tmp/tls.key. Store the CA that was used to sign this pair in /tmp/ca.crt

We can also use kumactl to generate self-signed certs.

  1. $ kumactl generate tls-certificate \
  2.   --type=server \
  3.   --cp-hostname=kuma-control-plane.kuma-system \ # adjust the name if you are installing Kuma to other namespace
  4.   --cp-hostname=kuma-control-plane.kuma-system.svc # adjust the name if you are installing Kuma to other namespace
  5.   --cert-file=/tmp/tls.crt \
  6.   --key-file=/tmp/tls.key
  7. $ cp /tmp/tls.crt /tmp/ca.crt # since this is self-signed cert, the cert is also a CA
  1. Create a secret in the namespace where the control plane is installed
  1. $ kubectl create secret generic general-tls-certs -n kuma-system \
  2.   --from-file=tls.crt=/tmp/tls.crt \
  3.   --from-file=tls.key=/tmp/tls.key \
  4.   --from-file=ca.crt=/tmp/ca.crt
  1. Point to this secret when installing Kuma

Set controlPlane.tls.general.secretName to general-tls-certs and controlPlane.tls.general.caBundle to <BASE64 content of ca.crt>

The data plane proxy Injector in the control plane automatically provides the CA to the Kuma DP sidecar so Kuma DP can confirm the control plane identity.

  1. Prepare certificates Generate TLS pair with a PKI of your choice and store it in PEM-encoded format in /tmp/tls.crt, /tmp/tls.key. Store the CA that was used to sign this pair in /tmp/ca.crt

We can also use kumactl to generate self-signed certs.

  1. $ kumactl generate tls-certificate \
  2.   --type=server \
  3.   --cp-hostname=<KUMA_CP_DNS_NAME> \ # set the hostname to a name which will be used by Kuma DP to connect to Kuma CP 
  4.   --cert-file=/tmp/tls.crt \
  5.   --key-file=/tmp/tls.key
  6. $ cp /tmp/tls.crt /tmp/ca.crt # since this is self-signed cert, the cert is also a CA
  1. Configure the control plane with generated certificates
  1. $ KUMA_DP_SERVER_TLS_CERT_FILE=/tmp/tls.crt \
  2.   KUMA_DP_SERVER_TLS_KEY_FILE=/tmp/tls.key
  3.   kuma-cp run
  1. Configure the data plane proxy with CA
  1. $ kuma-dp run \
  2.   --cp-address=https://<KUMA_CP_DNS_NAME>:5678 \ # make sure the address matches the hostname in the certificate
  3.   --ca-cert-file=/tmp/ca.crt # provide a file with CA
  4.   --dataplane-file=dp.yaml
  5.   --dataplane-token-file=/tmp/kuma-dp-redis-1-token

You can also provide the CA via environment variable KUMA_CONTROL_PLANE_CA_CERT.

Data plane proxy authentication

To obtain an mTLS certificate from the server (SDSSecure access across services - 图1 (opens new window) built-in in the control plane), a data plane proxy must authenticate itself.

A data plane proxy proves its identity by leveraging ServiceAccountTokenSecure access across services - 图2 (opens new window) that is mounted in every pod.

A data plane proxy must be explicitly configured with a unique security token (data plane proxy token) that will be used to prove its identity. The data plane proxy token is a signed JWT tokenSecure access across services - 图3 (opens new window) that carries the data plane proxy name and name of the mesh that it belongs to.

It is signed by an RSA key auto-generated by the control plane when Mesh is created. Tokens are never stored in the control plane, the only thing that is stored is a signing key that is used to verify if a token is valid.

You can generate token either by REST API

  1. $ curl -XPOST \ 
  2.   -H "Content-Type: application/json" \
  3.   --data '{"name": "dp-echo-1", "mesh": "default", "tags": {"kuma.io/service": ["backend", "backend-admin"]}}' \
  4.   http://localhost:5681/tokens/dataplane

or by using kumactl

  1. $ kumactl generate dataplane-token \
  2.   --name dp-echo-1 \
  3.   --mesh default \
  4.   --tag kuma.io/service=backend,backend-admin > /tmp/kuma-dp-echo1-token

The token should be stored in a file and then used when starting kuma-dp

  1. $ kuma-dp run \
  2.   --name=dp-echo-1 \
  3.   --mesh=default \
  4.   --cp-address=https://127.0.0.1:5678 \
  5.   --dataplane-token-file=/tmp/kuma-dp-echo-1-token

You can also pass Dataplane Token as KUMA_DATAPLANE_RUNTIME_TOKEN Environment Variable.

Data Plane Proxy Token boundary

As we can see in the example above, we can generate a token by passing a name, mesh, and a list of tags. The control plane will then verify the data plane proxy resources that are connecting to it against the token. This means we can generate a token by specifying:

  • Only mesh. By doing so we can reuse the token for all dataplanes in a given mesh.
  • mesh + tag (ex. kuma.io/service). This way we can use one token across all instances/replicas of the given service. Please keep in mind that we have to specify to include all the services that a data plane proxy is in charge of. For example, if we have a Dataplane with two inbounds, one valued with kuma.io/service: backend and one with kuma.io/service: backend-admin, we need to specify both values (--tag kuma.io/service=backend,backend-admin).
  • mesh + name + tag (ex. kuma.io/service). This way we can use one token for one instance/replica of the given service.
  • type. The type can be either dataplane (default if not specified) or ingress. A zone-ingress proxy in a multi-zone deployment requires an ingress token.

We can turn off the authentication by setting KUMA_DP_SERVER_AUTH_TYPE to none.

If we disable the authentication between the control plane and the data plane proxies, any data plane proxy will be able to impersonate any service, therefore this is not recommended in production.

Zone Ingress proxy authentication

To obtain Envoy resources over xDS from the server, a zone-ingress proxy must authenticate itself.

A zone-ingress proxy proves its identity by leveraging the ServiceAccountTokenSecure access across services - 图4 (opens new window) that is mounted in every pod.

A zone-ingress proxy must be explicitly configured with a unique security token (Zone Ingress proxy token) that will be used to prove its identity. The Zone Ingress proxy token is a signed JWT tokenSecure access across services - 图5 (opens new window) that carries the zone-ingress proxy name and name of the zone that it belongs to.

It is signed by an RSA key that is generated by the control plane on first start. Tokens are never stored in the control plane — only a signing key is stored, to verify whether a token is valid.

Generate the token with the REST API:

  1. $ curl -XPOST \ 
  2.   -H "Content-Type: application/json" \
  3.   --data '{"zone": "us-east"}' \
  4.   http://localhost:5681/tokens/zone-ingress

or with kumactl:

  1. $ kumactl generate zone-ingress-token \
  2.   --zone us-east  > /tmp/kuma-dp-echo1-token

The token should be stored in a file and then passed when you start kuma-dp:

  1. $ kuma-dp run \
  2.   --proxy-type=ingress \
  3.   --name=dp-ingress \
  4.   --cp-address=https://127.0.0.1:5678 \
  5.   --dataplane-token-file=/tmp/kuma-dp-echo-1-token

You can also pass the token as a KUMA_DATAPLANE_RUNTIME_TOKEN environment variable.

You can turn off authentication by setting KUMA_DP_SERVER_AUTH_TYPE to none.

You should not disable authentication between the control plane and the data plane proxies in production. Disabling means that any data plane proxy can impersonate any service.

Multizone

When running in multizone, mode we can generate data plane proxy token both on global and zone control plane. If the deployment pipeline is configured to generate data plane proxy token before running the proxy, it can rely on the Zone CP. This way Global CP is not a single point of failure.

Data plane to data plane communication

Kuma helps us to secure the existing infrastructure with mTLS.

Mutual TLS

Once a data plane proxy has proved its identity to the control plane, it will be allowed to fetch its own identity certificate and a root CA certificate of the mesh. When establishing a connection between two data plane proxies, each side validates the other’s certificate using the root CA of the mesh, which then confirms their identity.

mTLS is not enabled by default. To enable it, we must apply proper settings in Mesh policy. Additionally, when running on Universal we have to ensure that every data plane proxy in the mesh has been configured with a dataplane proxy token.

Certificates

In Kuma, any TLS certificate that is being issued by the control plane must have a CA (Certificate Authority) that can be either auto-generated by Kuma via a builtin backend, or can be initialized with a customer certificate and key via a provided backend.

Third-party extensions, cloud implementations or commercial offerings may be extending the CA backend support.

For both builtin and provided CA backends, on Kubernetes the root CA certificate is stored as a Kubernetes SecretSecure access across services - 图6 (opens new window), while on Universal Kuma leverages the same underlying storage backend that is used for storing policies.

When the mTLS policy is enabled, data plane proxy certificates are ephemeral: thet are re-created on every data plane proxy restart and never persisted on disk.

Data plane proxy certificates generated by Kuma are X.509 certificates that are SPIFFESecure access across services - 图7 (opens new window) compliant. The SAN of the certificates is set to spiffe://<mesh name>/<service name>. The data plane proxy that implements many services (multiple inbound) receives a certificate with many SAN entries, one for each service.

User to control plane communication

Users and automation tools can interact with the control plane via the API Server using tools like curl or kumactl. API Server is exposed by default on :5681 on HTTP and :5682 on HTTPS.

Encrypted communication

API Server HTTPS server is secured by default by autogenerated certificates.

To override autogenerated certificates.

  1. Prepare certificates

Generate TLS pair with a PKI of your choice and store it in PEM-encoded format in /tmp/tls.crt, /tmp/tls.key. Store the CA that was used to sign this pair in /tmp/ca.crt

We can also use kumactl to generate self-signed certs.

  1. $ kumactl generate tls-certificate \
  2.   --type=server \
  3.   --cp-hostname=<KUMA_CP_DNS_NAME> \ # pick a name that will be used by kumactl or other client to connect to the control plane
  4.   --cert-file=/tmp/tls.crt \
  5.   --key-file=/tmp/tls.key
  6. $ cp /tmp/tls.crt /tmp/ca.crt # since this is self-signed cert, the cert is also a CA
  1. Configure the control plane with generated certificates

Create a secret in the namespace in which the control plane is installed

  1. $ kubectl create secret tls api-server-tls -n kuma-system \
  2.   --cert=/tmp/tls.crt \
  3.   --key=/tmp/tls.key

Point to this secret when installing Kuma

  1. $ kumactl install control-plane \
  2.   --tls-api-server-secret=api-server-tls

Create a secret in the namespace in which the control plane is installed

  1. $ kubectl create secret tls api-server-tls -n kuma-system \
  2.   --cert=/tmp/tls.crt \
  3.   --key=/tmp/tls.key

Set controlPlane.tls.apiServer.secretName to api-server-tls

Point to the certificate and the key.

  1. $ KUMA_API_SERVER_HTTPS_TLS_CERT_FILE=/tmp/tls.crt \
  2.   KUMA_API_SERVER_HTTPS_TLS_KEY_FILE=/tmp/tls.key
  3.   kuma-cp run
  1. Configure secure connection using kumactl CLI tool.
  1. kumactl config control-planes add \
  2.   --name=<NAME> \
  3.   --address=https://<KUMA_CP_DNS_NAME>:5682 \
  4.   --ca-cert-file=/tmp/ca.crt \

We can also hide the HTTP version of API Server by binding it to localhost KUMA_API_SERVER_HTTP_INTERFACE: 127.0.0.1 or by disabling it altogether KUMA_API_SERVER_HTTP_ENABLED: false

Authentication

Some endpoints like managing Secrets or generating data plane proxy token require authentication. There are two ways to access those endpoints.

Request originating from localhost

For the simplicity of use, requests that are originating from the same machine as CP are authenticated. We can disable this behavior by setting KUMA_API_SERVER_AUTH_ALLOW_FROM_LOCALHOST to false

On Kubernetes, we can port-forward 5681 port and access the admin endpoints.

Client certificates

When accessing admin endpoints from a different machine we need to use client certificates.

  1. Generate client certificates by using kumactl
  1. $ kumactl generate tls-certificate --type=client \
  2.   --cert-file=/tmp/tls.crt \
  3.   --key-file=/tmp/tls.key
  1. Configure the control plane with client certificates

Create a secret in the namespace in which control plane is installed

  1. $ kubectl create secret generic api-server-client-certs -n kuma-system \
  2.   --from-file=client1.pem=/tmp/tls.crt \

We can provide as many client certificates as we want. Remember to only provide certificates without keys.

Point to this secret when installing Kuma

  1. $ kumactl install control-plane \
  2.   --tls-api-server-client-certs-secret=api-server-client-certs

Create a secret in the namespace in which control plane is installed

  1. $ kubectl create secret generic api-server-client-certs -n kuma-system \
  2.   --from-file=client1.pem=/tmp/tls.crt \

We can provide as many client certificates as we want. Remember to only provide certificates without keys.

Set controlPlane.tls.apiServer.clientCertsSecretName to api-server-client-certs via HELM

Put all the certificates in one directory

  1. $ mkdir /opt/client-certs
  2. $ cp /tmp/tls.crt /opt/client-certs/client1.pem

All client certificates MUST end with .pem extension. Remember to only put provide certificates without keys.

Configure control plane by pointing to this directory

  1. $ KUMA_API_SERVER_AUTH_CLIENT_CERTS_DIR=/opt/client-certs \
  2.   kuma-cp run
  1. Configure kumactl with valid client certificates
  1. $ kumactl config control-planes add \
  2.   --name=<NAME>
  3.   --address=https://<KUMA_CP_DNS_NAME>:5682 \
  4.   --client-cert-file=/tmp/tls.crt \
  5.   --client-key-file=/tmp/tls.key \
  6.   --ca-cert-file=/tmp/ca.crt # CA cert used in "Encrypted communication" section

Control plane to Control plane (Multizone)

A zone control plane connects to a global control plane for policies configuration.

Encrypted communication

Because the global control plane and the zone control plane exchange sensitive information, the communication needs to be encrypted by TLS. By default, the global control plane’s server that is consumed by the zone control plane is secured by TLS with autogenerated certificates.

It is recommended that the zone control plane verifies the identity of the global control plane. To do so, zone control planes need to obtain the CA that was used to generate the certificate by which the control plane’s server is secured.

To override autogenerated certificates

  1. Prepare certificates

Generate TLS pair with a PKI of your choice and store it in PEM-encoded format in /tmp/tls.crt, /tmp/tls.key. Store the CA that was used to sign this pair in /tmp/ca.crt

We can also use kumactl to generate self-signed certs.

  1. $ kumactl generate tls-certificate \
  2.   --type=server \
  3.   --cp-hostname=<CROSS_ZONE_KUMA_CP_DNS_NAME> \
  4.   --cert-file=/tmp/tls.crt \
  5.   --key-file=/tmp/tls.key
  6. $ cp /tmp/tls.crt /tmp/ca.crt # since this is self-signed cert, the cert is also a CA
  1. Configure global control plane

Create a secret in the namespace where the global control plane is installed

  1. $ kubectl create secret tls kds-server-tls -n kuma-system \
  2.   --cert=/tmp/tls.crt \
  3.   --key=/tmp/tls.key

Point to this secret when installing the global control plane

  1. $ kumactl install control-plane \
  2.   --mode=global
  3.   --tls-kds-global-server-secret=general-tls-certs

Create a secret in the namespace where the global control plane is installed

  1. $ kubectl create secret tls kds-server-tls -n kuma-system \
  2.   --cert=/tmp/tls.crt \
  3.   --key=/tmp/tls.key

Set controlPlane.tls.kdsGlobalServer.secretName to kds-server-tls.

Point to the certificate and the key.

  1. $ KUMA_MULTIZONE_GLOBAL_KDS_TLS_CERT_FILE=/tmp/tls.crt \
  2.   KUMA_MULTIZONE_GLOBAL_KDS_TLS_KEY_FILE=/tmp/tls.key \
  3.   KUMA_MODE=global \
  4.   kuma-cp run
  1. Configure the zone control plane

Create a secret in the namespace where the zone control plane is installed

  1. $ kubectl create secret generic kds-ca-certs -n kuma-system \
  2.   --from-file=ca.crt.pem=/tmp/ca.crt

Point to this secret when installing the zone control plane

  1. $ kumactl install control-plane \
  2.   --mode=zone
  3.   --tls-kds-zone-client-secret=kds-ca-certs

Create a secret in the namespace where the zone control plane is installed

  1. $ kubectl create secret generic kds-ca-certs -n kuma-system \
  2.   --from-file=ca.crt.pem=/tmp/ca.crt

Set controlPlane.tls.kdsZoneClient.secretName to kds-ca-certs.

Point to the certificate and the key.

  1. $ KUMA_MULTIZONE_ZONE_KDS_ROOT_CA_FILE=/tmp/ca.crt \
  2.   KUMA_MODE=zone \
  3.   KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS=grpcs://<CROSS_ZONE_KUMA_CP_DNS_NAME>:5685 \
  4.   kuma-cp run

Authentication

Define firewall rules on the global control plane to only accept connections from known IPs of the zone control planes.

Third-party extensions, cloud implementations or commercial offerings may be extending the authentication support.

Control plane to Postgres communication

Since on Universal secrets such as provided CA’s private key are stored in Postgres, a connection between Postgres and Kuma CP should be secured with TLS.

To secure the connection, we first need to pick the security mode using KUMA_STORE_POSTGRES_TLS_MODE. There are several modes:

  • disable - is not secured with TLS (secrets will be transmitted over network in plain text).
  • verifyNone - the connection is secured but neither hostname, nor by which CA the certificate is signed is checked.
  • verifyCa - the connection is secured and the certificate presented by the server is verified using the provided CA.
  • verifyFull - the connection is secured, certificate presented by the server is verified using the provided CA and server hostname must match the one in the certificate.

The CA used to verify the server’s certificate can be set using the KUMA_STORE_POSTGRES_TLS_CA_PATH environment variable.

After configuring the above security settings in Kuma, we also have to configure Postgres’ pg_hba.confSecure access across services - 图8 (opens new window) file to restrict unsecured connections.

Here is an example configuration that will allow only TLS connections and will require username and password:

  1. # TYPE  DATABASE        USER            ADDRESS                 METHOD
  2. hostssl all             all             0.0.0.0/0               password

We can also provide a client key and certificate for mTLS using the KUMA_STORE_POSTGRES_TLS_CERT_PATH and KUMA_STORE_POSTGRES_TLS_KEY_PATH variables. This pair can be used in conjunction with the cert auth-method described hereSecure access across services - 图9 (opens new window).