Configure zone proxy authentication

To obtain a configuration from the control plane, a zone proxy (zone ingress / zone egress) must authenticate itself. There are several authentication methods available.

Service Account Token

On Kubernetes, A zone proxy proves its identity by leveraging the ServiceAccountToken that is mounted in every pod.

Zone Token

On Universal, a zone proxy proxy must be explicitly configured with a unique security token (Zone token) with appropriate scope (egress, ingress), that will be used to prove its identity.

The zone token used to identify zone proxies is a JWT token that contains:

  • Zone in which zone proxy operates
  • Expiration date of the token (required, 10 years if not specified)
  • Scope as a list of items where the token will be valid (required, egress, ingress if not specified)

The zone token is signed by a signing key that is autogenerated during the first start of the control plane. Tokens are never stored in the control plane, the only thing that is stored are signing keys that are used to verify if a token is valid. The signing key is RSA256 encrypted.

You can check for the signing key:

  1. kumactl get global-secrets

which returns something like:

  1. NAME                       AGE
  2. zone-token-signing-key-1   7s

Usage

  1. kumactl generate zone-token \
  2.   --zone us-east \
  3.   --scope egress \
  4.   --valid-for 720h > /tmp/kuma-zone-proxy-token

Generate the token with the REST API:

  1. curl -XPOST \
  2.   -H "Content-Type: application/json" \
  3.   --data '{"zone": "us-east", "validFor": "720h", "scope": ["egress", "ingress"]}' \
  4.   http://localhost:5681/tokens/zone > /tmp/kuma-zone-proxy-token

The token should be stored in a file and then passed when you start kuma-dp:

  1. kuma-dp run \
  2.   --proxy-type=ingress # or egress \
  3.   --dataplane-file=zone-proxy-definition.yaml
  4.   --cp-address=https://127.0.0.1:5678 \
  5.   --dataplane-token-file=/tmp/kuma-zone-proxy-token

You can also pass the token as a KUMA_DATAPLANE_RUNTIME_TOKEN environment variable.

Token Revocation

Kuma does not keep the list of issued tokens. Whenever the single token is compromised, we can add it to revocation list, so it’s no longer valid.

Every token has its own ID which is available in payload under jti key. You can extract ID from token using jwt.io or jwt-cli tool. Here is example of jti

  1. 0e120ec9-6b42-495d-9758-07b59fe86fb9

Specify list of revoked IDs separated by , and store it as GlobalSecret named zone-token-revocations

  1. REVOCATIONS=$(echo '0e120ec9-6b42-495d-9758-07b59fe86fb9' | base64) && echo "apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: zone-token-revocations
  5.   namespace: kuma-system
  6. data:
  7.   value: $REVOCATIONS
  8. type: system.kuma.io/global-secret" | kubectl apply -f -
  1. echo "
  2. type: GlobalSecret
  3. name: zone-token-revocations
  4. data: " | kumactl apply --var revocations=$(echo '0e120ec9-6b42-495d-9758-07b59fe86fb9' | base64) -f -

Signing key rotation

If the signing key is compromised, we must rotate it and all the tokens that were signed by it.

  1. Generate new signing key The signing key is stored as a GlobalSecret with a name that looks like zone-token-signing-key-{serialNumber}.

    Make sure to generate the new signing key with a serial number greater than the serial number of the current signing key.

    Check what is the current highest serial number.

    1. kubectl get secrets -n kuma-system --field-selector='type=system.kuma.io/global-secret'
    2. NAME                       TYPE                           DATA   AGE
    3. zone-token-signing-key-1   system.kuma.io/global-secret   1      25m

    In this case, the highest serial number is 1. Generate a new signing key with a serial number of 2

    1. TOKEN="$(kumactl generate signing-key)" && echo "
    2. apiVersion: v1
    3. data:
    4.   value: $TOKEN
    5. kind: Secret
    6. metadata:
    7.   name: zone-token-signing-key-2
    8.   namespace: kuma-system
    9. type: system.kuma.io/global-secret
    10. " | kubectl apply -f -

    Check what is the current highest serial number.

    1. kumactl get global-secrets
    2. NAME                       AGE
    3. zone-token-signing-key-1   36m

    In this case, the highest serial number is 1. Generate a new signing key with a serial number of 2

    1. echo "
    2. type: GlobalSecret
    3. name: zone-token-signing-key-2
    4. data: " | kumactl apply --var key=$(kumactl generate signing-key) -f -

  2. Regenerate tokens These tokens are automatically created with the signing key that’s assigned the highest serial number, so they’re created with the new signing key. At this point, tokens signed by either new or old signing key are valid.

  3. Remove the old signing key

    1. kubectl delete secret zone-token-signing-key-1 -n kuma-system
    1. kumactl delete global-secret zone-token-signing-key-1

    All new connections to the control plane now require tokens signed with the new signing key.

Offline token issuing

In addition to the regular flow of generating signing keys, storing them in secret, and using them to sign tokens on the control plane, Kuma also offers offline signing of tokens. In this flow, you can generate a pair of public and private keys and configure the control plane only with public keys for token verification. You can generate all the tokens without running the control plane.

The advantages of this mode are:

  • easier, more reproducible deployments of the control plane, and more in line with GitOps.
  • potentially more secure setup, because the control plane does not have access to the private keys.

Here’s how to use offline issuing

  1. Generate a pair of signing keys

    The following commands generate standard RSA key of 2048 bits and outputs it in PEM-encoded format. You can use any external tool to generate a pair of keys.

    1. kumactl generate signing-key --format=pem > /tmp/key-private.pem
    2. kumactl generate public-key --signing-key-path=/tmp/key-private.pem > /tmp/key-public.pem

    The result should be similar to this output

    1. cat /tmp/key-private.pem /tmp/key-public.pem 
    2. -----BEGIN RSA PRIVATE KEY-----
    3. MIIEpAIBAAKCAQEAsS61a79gC4mkr2Ltwi09ajakLyUR8YTkJWzZE805EtTkEn/r
    4. ...
    5. htKtzsYA7yGlt364IuDybrP+PlPMSK9cQAmWRRZIcBNsKOODkAgKFA==
    6. -----END RSA PRIVATE KEY-----
    7. -----BEGIN RSA PUBLIC KEY-----
    8. MIIBCgKCAQEAsS61a79gC4mkr2Ltwi09ajakLyUR8YTkJWzZE805EtTkEn/rL2u/
    9. ...
    10. se7sx2Pt/NPbWFFTMGVFm3A1ueTUoorW+wIDAQAB
    11. -----END RSA PUBLIC KEY-----

  2. Configure the control plane with public key

    Configure a control plane with the following settings

    1. dpServer:
    2.   authn:
    3.     zoneProxy:
    4.       type: zoneToken
    5.       zoneToken:
    6.         enableIssuer: false # disable control plane token issuer that uses secrets
    7.         validator:
    8.           useSecrets: false # do not use signing key stored in secrets to validate the token
    9.           publicKeys:
    10.           - kid: "key-1"
    11.             key: |
    12.               -----BEGIN RSA PUBLIC KEY-----
    13.               MIIBCgKCAQEAsS61a79gC4mkr2Ltwi09ajakLyUR8YTkJWzZE805EtTkEn/rL2u/
    14.               ...
    15.               se7sx2Pt/NPbWFFTMGVFm3A1ueTUoorW+wIDAQAB
    16.               -----END RSA PUBLIC KEY-----

  3. Use the private key to issue tokens offline

    The command is the same as with online signing, but with two additional arguments:

    • --kid - ID of the key that should be used to validate the token. This should match kid specified in the control plane configuration.
    • --signing-key-path - path to a PEM-encoded private key.
    1. kumactl generate zone-token \
    2.   --zone us-east \
    3.   --scope egress \
    4.   --valid-for 720h \
    5.   --signing-key-path /tmp/key-private.pem \
    6.   --kid key-1

    You can also use any external system that can issue JWT tokens using RS256 signing method with the following claims:

    • Zone (string) - the name of the zone
    • Scope ([]string) - the list of scopes (egress, ingress)

Migration

You can use both offline and online issuing by keeping dpServer.authn.zoneProxy.zoneToken.enableIssuer to true. You can use both secrets and public key static config validators by keeping dpServer.authn.zoneProxy.zoneToken.validator.useSecrets to true.

Management

Token revocation works the same when using both online and offline issuing.

Signing key rotation works similarly:

  • generate another pair of signing keys
  • configure a control plane with old and new public keys
  • regenerate tokens for all existing zone proxies with the new private key
  • remove the old public key from the configuration

Multi-zone

When running in multi-zone mode, we can generate zone tokens only on the global control plane. The zone control plane only has a public key of a signing key to verify tokens.

None

You can turn off authentication by setting KUMA_DP_SERVER_AUTH_TYPE to none.

You should not disable authentication between the control plane and the data plane proxies in production. Disabling means that any data plane proxy can impersonate any service.

Legacy Zone Ingress Token

Authenticating Zone Ingress using separate Zone Ingress Token is still possible, but it is deprecated and will be removed in the future.