Setup & configure mTLS certificates

Encrypt communication between applications using self-signed or user supplied x.509 certificates

Dapr supports in-transit encryption of communication between Dapr instances using the Dapr control plane, Sentry service, which is a central Certificate Authority (CA).

Dapr allows operators and developers to bring in their own certificates, or instead let Dapr automatically create and persist self-signed root and issuer certificates.

For detailed information on mTLS, read the security concepts section.

If custom certificates have not been provided, Dapr automatically creates and persist self-signed certs valid for one year. In Kubernetes, the certs are persisted to a secret that resides in the namespace of the Dapr system pods, accessible only to them. In self hosted mode, the certs are persisted to disk.

Control plane Sentry service configuration

The mTLS settings reside in a Dapr control plane configuration file. For example when you deploy the Dapr control plane to Kubernetes this configuration file is automatically created and then you can edit this. The following file shows the available settings for mTLS in a configuration resource, deployed in the daprsystem namespace:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: daprsystem
  5.   namespace: default
  6. spec:
  7.   mtls:
  8.     enabled: true
  9.     workloadCertTTL: "24h"
  10.     allowedClockSkew: "15m"

The file here shows the default daprsystem configuration settings. The examples below show you how to change and apply this configuration to the control plane Sentry service either in Kubernetes and self hosted modes.

Kubernetes

Setting up mTLS with the configuration resource

In Kubernetes, Dapr creates a default control plane configuration resource with mTLS enabled. The Sentry service, the certificate authority system pod, is installed both with Helm and with the Dapr CLI using dapr init --kubernetes.

You can view the control plane configuration resource with the following command:

kubectl get configurations/daprsystem --namespace <DAPR_NAMESPACE> -o yaml.

To make changes to the control plane configuration resource, run the following command to edit it:

  1. kubectl edit configurations/daprsystem --namespace <DAPR_NAMESPACE>

Once the changes are saved, perform a rolling update to the control plane:

  1. kubectl rollout restart deploy/dapr-sentry -n <DAPR_NAMESPACE>
  2. kubectl rollout restart deploy/dapr-operator -n <DAPR_NAMESPACE>
  3. kubectl rollout restart statefulsets/dapr-placement-server -n <DAPR_NAMESPACE>

Note: the control plane Sidecar Injector service does not need to be redeployed

Disabling mTLS with Helm

  1. kubectl create ns dapr-system
  3. helm install \
  4.   --set global.mtls.enabled=false \
  5.   --namespace dapr-system \
  6.   dapr \
  7.   dapr/dapr

Disabling mTLS with the CLI

  1. dapr init --kubernetes --enable-mtls=false

Viewing logs

In order to view the Sentry service logs, run the following command:

  1. kubectl logs --selector=app=dapr-sentry --namespace <DAPR_NAMESPACE>

Bringing your own certificates

Using Helm, you can provide the PEM encoded root cert, issuer cert and private key that will be populated into the Kubernetes secret used by the Sentry service.

Avoiding downtime

To avoid downtime when rotating expiring certificates always sign your certificates with the same private root key.

Note: This example uses the OpenSSL command line tool, this is a widely distributed package, easily installed on Linux via the package manager. On Windows OpenSSL can be installed using chocolatey. On MacOS it can be installed using brew brew install openssl

Create config files for generating the certificates, this is necessary for generating v3 certificates with the SAN (Subject Alt Name) extension fields. First save the following to a file named root.conf:

  1. [req]
  2. distinguished_name = req_distinguished_name
  3. x509_extensions = v3_req
  4. prompt = no
  5. [req_distinguished_name]
  6. C = US
  7. ST = VA
  8. L = Daprville
  9. O = dapr.io/sentry
  10. OU = dapr.io/sentry
  11. CN = cluster.local
  12. [v3_req]
  13. basicConstraints = critical, CA:true
  14. keyUsage = critical, digitalSignature, cRLSign, keyCertSign
  15. extendedKeyUsage = serverAuth, clientAuth
  16. subjectAltName = @alt_names
  17. [alt_names]
  18. DNS.1 = cluster.local

Repeat this for issuer.conf, paste the same contents into the file, but add pathlen:0 to the end of the basicConstraints line, as shown below:

  1. basicConstraints = critical, CA:true, pathlen:0

Run the following to generate the root cert and key

  1. # skip the following line to reuse an existing root key, required for rotating expiring certificates
  2. openssl ecparam -genkey -name prime256v1 | openssl ec -out root.key
  3. openssl req -new -nodes -sha256 -key root.key -out root.csr -config root.conf -extensions v3_req
  4. openssl x509 -req -sha256 -days 365 -in root.csr -signkey root.key -outform PEM -out root.pem -extfile root.conf -extensions v3_req

Next run the following to generate the issuer cert and key:

  1. # skip the following line to reuse an existing issuer key, required for rotating expiring certificates
  2. openssl ecparam -genkey -name prime256v1 | openssl ec -out issuer.key
  3. openssl req -new -sha256 -key issuer.key -out issuer.csr -config issuer.conf -extensions v3_req
  4. openssl x509 -req -in issuer.csr -CA root.pem -CAkey root.key -CAcreateserial -outform PEM -out issuer.pem -days 365 -sha256 -extfile issuer.conf -extensions v3_req

Install Helm and pass the root cert, issuer cert and issuer key to Sentry via configuration:

  1. kubectl create ns dapr-system
  3. helm install \
  4.   --set-file dapr_sentry.tls.issuer.certPEM=issuer.pem \
  5.   --set-file dapr_sentry.tls.issuer.keyPEM=issuer.key \
  6.   --set-file dapr_sentry.tls.root.certPEM=root.pem \
  7.   --namespace dapr-system \
  8.   dapr \
  9.   dapr/dapr

Updating root or issuer certs

If the Root or Issuer certs are about to expire, you can update them and restart the required system services.

Avoiding downtime when rotating certificates

To avoid downtime when rotating expiring certificates your new certificates must be signed with the same private root key as the previous certificates.

First, issue new certificates using the step above in Bringing your own certificates.

Now that you have the new certificates, use Helm to upgrade the certificates:

  1. helm upgrade \
  2.   --set-file dapr_sentry.tls.issuer.certPEM=issuer.pem \
  3.   --set-file dapr_sentry.tls.issuer.keyPEM=issuer.key \
  4.   --set-file dapr_sentry.tls.root.certPEM=root.pem \
  5.   --namespace dapr-system \
  6.   dapr \
  7.   dapr/dapr

Alternatively, you can update the Kubernetes secret that holds them:

  1. kubectl edit secret dapr-trust-bundle -n <DAPR_NAMESPACE>

Replace the ca.crt, issuer.crt and issuer.key keys in the Kubernetes secret with their corresponding values from the new certificates. Note: The values must be base64 encoded

If you signed the new cert root with the same private key the Dapr Sentry service will pick up the new certificates automatically. You can restart your application deployments using kubectl rollout restart with zero downtime. It is not necessary to restart all deployments at once, as long as deployments are restarted before original certificate expiration.

If you signed the new cert root with a different private key, you must restart the Dapr sentry service.

  1. kubectl rollout restart deploy/dapr-sentry -n <DAPR_NAMESPACE>

Next, you must restart all Dapr-enabled pods. The recommended way to do this is to perform a rollout restart of your deployment:

  1. kubectl rollout restart deploy/myapp

You will experience potential downtime due to mismatching certificates until all deployments have successfully been restarted (and hence loaded the new Dapr certificates).

Kubernetes video demo

Watch this video to show how to update mTLS certificates on Kubernetes

Self hosted

Running the control plane Sentry service

In order to run the Sentry service, you can either build from source, or download a release binary from here.

When building from source, please refer to this guide on how to build Dapr.

Second, create a directory for the Sentry service to create the self signed root certs:

  1. mkdir -p $HOME/.dapr/certs

Run the Sentry service locally with the following command:

  1. ./sentry --issuer-credentials $HOME/.dapr/certs --trust-domain cluster.local

If successful, the Sentry service runs and creates the root certs in the given directory. This command uses default configuration values as no custom config file was given. See below on how to start the Sentry service with a custom configuration.

Setting up mTLS with the configuration resource

Dapr instance configuration

When running Dapr in self hosted mode, mTLS is disabled by default. you can enable it by creating the following configuration file:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: daprsystem
  5.   namespace: default
  6. spec:
  7.   mtls:
  8.     enabled: true

In addition to the Dapr configuration, you also need to provide the TLS certificates to each Dapr sidecar instance. You can do so by setting the following environment variables before running the Dapr instance:

  1. export DAPR_TRUST_ANCHORS=`cat $HOME/.dapr/certs/ca.crt`
  2. export DAPR_CERT_CHAIN=`cat $HOME/.dapr/certs/issuer.crt`
  3. export DAPR_CERT_KEY=`cat $HOME/.dapr/certs/issuer.key`
  4. export NAMESPACE=default
  1. $env:DAPR_TRUST_ANCHORS=$(Get-Content -raw $env:USERPROFILE\.dapr\certs\ca.crt)
  2. $env:DAPR_CERT_CHAIN=$(Get-Content -raw $env:USERPROFILE\.dapr\certs\issuer.crt)
  3. $env:DAPR_CERT_KEY=$(Get-Content -raw $env:USERPROFILE\.dapr\certs\issuer.key)
  4. $env:NAMESPACE="default"

If using the Dapr CLI, point Dapr to the config file above to run the Dapr instance with mTLS enabled:

  1. dapr run --app-id myapp --config ./config.yaml node myapp.js

If using daprd directly, use the following flags to enable mTLS:

  1. daprd --app-id myapp --enable-mtls --sentry-address localhost:50001 --config=./config.yaml

Sentry service configuration

Here’s an example of a configuration for Sentry that changes the workload cert TTL to 25 seconds:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: daprsystem
  5.   namespace: default
  6. spec:
  7.   mtls:
  8.     enabled: true
  9.     workloadCertTTL: "25s"

In order to start Sentry service with a custom config, use the following flag:

  1. ./sentry --issuer-credentials $HOME/.dapr/certs --trust-domain cluster.local --config=./config.yaml

Bringing your own certificates

In order to provide your own credentials, create ECDSA PEM encoded root and issuer certificates and place them on the file system. Tell the Sentry service where to load the certificates from using the --issuer-credentials flag.

The next examples creates root and issuer certs and loads them with the Sentry service.

Note: This example uses the step tool to create the certificates. You can install step tool from here. Windows binaries available here

Create the root certificate:

  1. step certificate create cluster.local ca.crt ca.key --profile root-ca --no-password --insecure

Create the issuer certificate:

  1. step certificate create cluster.local issuer.crt issuer.key --ca ca.crt --ca-key ca.key --profile intermediate-ca --not-after 8760h --no-password --insecure

This creates the root and issuer certs and keys. Place ca.crt, issuer.crt and issuer.key in a desired path ($HOME/.dapr/certs in the example below), and launch Sentry:

  1. ./sentry --issuer-credentials $HOME/.dapr/certs --trust-domain cluster.local

Updating root or issuer certificates

If the Root or Issuer certs are about to expire, you can update them and restart the required system services.

First, issue new certificates using the step above in Bringing your own certificates.

Copy ca.crt, issuer.crt and issuer.key to the filesystem path of every configured system service, and restart the process or container. By default, system services will look for the credentials in /var/run/dapr/credentials.

Note:If you signed the cert root with a different private key, restart the Dapr instances.

Last modified February 18, 2022: Update setup-jetstream.md (#2200) (428d8c2)