Setup & configure mutual TLS

Encrypt communication between Dapr instances

Dapr supports in-transit encryption of communication between Dapr instances using Sentry, a central Certificate Authority.

Dapr allows operators and developers to bring in their own certificates, or let Dapr automatically create and persist self signed root and issuer certificates.

For detailed information on mTLS, go to the concepts section here.

If custom certificates have not been provided, Dapr will automatically create and persist self signed certs valid for one year. In Kubernetes, the certs are persisted to a secret that resides in the namespace of the Dapr system pods, accessible only to them. In Self Hosted mode, the certs are persisted to disk. More information on that is shown below.

Sentry configuration

mTLS settings reside in a Dapr configuration file. The following file shows all the available settings for mTLS in a configuration resource:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: daprsystem
  5.   namespace: default
  6. spec:
  7.   mtls:
  8.     enabled: true
  9.     workloadCertTTL: "24h"
  10.     allowedClockSkew: "15m"

The file here shows the default daprsystem configuration settings. The examples below show you how to change and apply this configuration to Sentry in Kubernetes and Self hosted modes.

Kubernetes

Setting up mTLS with the configuration resource

In Kubernetes, Dapr creates a default configuration resource with mTLS enabled. Sentry, the certificate authority system pod, is installed both with Helm and with the Dapr CLI using dapr init --kubernetes.

You can view the configuration resource with the following command:

kubectl get configurations/daprsystem --namespace <DAPR_NAMESPACE> -o yaml.

To make changes to the configuration resource, you can run the following command to edit it:

  1. kubectl edit configurations/daprsystem --namespace <DAPR_NAMESPACE>

Once the changes are saved, perform a rolling update to the control plane:

  1. kubectl rollout restart deploy/dapr-sentry -n <DAPR_NAMESPACE>
  2. kubectl rollout restart deploy/dapr-operator -n <DAPR_NAMESPACE>
  3. kubectl rollout restart statefulsets/dapr-placement-server -n <DAPR_NAMESPACE>

Note: the sidecar injector does not need to be redeployed

Disabling mTLS with Helm

  1. kubectl create ns dapr-system
  3. helm install \
  4.   --set global.mtls.enabled=false \
  5.   --namespace dapr-system \
  6.   dapr \
  7.   dapr/dapr

Disabling mTLS with the CLI

  1. dapr init --kubernetes --enable-mtls=false

Viewing logs

In order to view Sentry logs, run the following command:

  1. kubectl logs --selector=app=dapr-sentry --namespace <DAPR_NAMESPACE>

Bringing your own certificates

Using Helm, you can provide the PEM encoded root cert, issuer cert and private key that will be populated into the Kubernetes secret used by Sentry.

Note: This example uses the OpenSSL command line tool, this is a widely distributed package, easily installed on Linux via the package manager. On Windows OpenSSL can be installed using chocolatey. On MacOS it can be installed using brew brew install openssl

Create config files for generating the certificates, this is necessary for generating v3 certificates with the SAN (Subject Alt Name) extension fields. First save the following to a file named root.conf:

  1. [req]
  2. distinguished_name = req_distinguished_name
  3. x509_extensions = v3_req
  4. prompt = no
  5. [req_distinguished_name]
  6. C = US
  7. ST = VA
  8. L = Daprville
  9. O = dapr.io/sentry
  10. OU = dapr.io/sentry
  11. CN = cluster.local
  12. [v3_req]
  13. basicConstraints = critical, CA:true
  14. keyUsage = critical, digitalSignature, cRLSign, keyCertSign
  15. extendedKeyUsage = serverAuth, clientAuth
  16. subjectAltName = @alt_names
  17. [alt_names]
  18. DNS.1 = cluster.local

Repeat this for issuer.conf, paste the same contents into the file, but add pathlen:0 to the end of the basicConstraints line, as shown below:

  1. basicConstraints = critical, CA:true, pathlen:0

Run the following to generate the root cert and key

  1. openssl ecparam -genkey -name prime256v1 | openssl ec -out root.key
  2. openssl req -new -nodes -sha256 -key root.key -out root.csr -config root.conf -extensions v3_req
  3. openssl x509 -req -sha256 -days 365 -in root.csr -signkey root.key -outform PEM -out root.pem -extfile root.conf -extensions v3_req

Next run the following to generate the issuer cert and key:

  1. openssl ecparam -genkey -name prime256v1 | openssl ec -out issuer.key
  2. openssl req -new -sha256 -key issuer.key -out issuer.csr -config issuer.conf -extensions v3_req
  3. openssl x509 -req -in issuer.csr -CA root.pem -CAkey root.key -CAcreateserial -outform PEM -out issuer.pem -days 365 -sha256 -extfile issuer.conf -extensions v3_req

Install Helm and pass the root cert, issuer cert and issuer key to Sentry via configuration:

  1. kubectl create ns dapr-system
  3. helm install \
  4.   --set-file dapr_sentry.tls.issuer.certPEM=issuer.pem \
  5.   --set-file dapr_sentry.tls.issuer.keyPEM=issuer.key \
  6.   --set-file dapr_sentry.tls.root.certPEM=root.pem \
  7.   --namespace dapr-system \
  8.   dapr \
  9.   dapr/dapr

Updating Root or Issuer Certs

If the Root or Issuer certs are about to expire, you can update them and restart the required system services.

First, issue new certificates using the step above in Bringing your own certificates.

Now that you have the new certificates, you can update the Kubernetes secret that holds them. Edit the Kubernetes secret:

  1. kubectl edit secret dapr-trust-bundle -n <DAPR_NAMESPACE>

Replace the ca.crt, issuer.crt and issuer.key keys in the Kubernetes secret with their corresponding values from the new certificates. Note: The values must be base64 encoded

If you signed the new cert root with a different private key, restart all Dapr-enabled pods. The recommended way to do this is to perform a rollout restart of your deployment:

  1. kubectl rollout restart deploy/myapp

Self-hosted

Running Sentry system service

In order to run Sentry, you can either build from source, or download a release binary from here.

When building from source, please refer to this guide on how to build Dapr.

Second, create a directory for Sentry to create the self signed root certs:

  1. mkdir -p $HOME/.dapr/certs

Run Sentry locally with the following command:

  1. ./sentry --issuer-credentials $HOME/.dapr/certs --trust-domain cluster.local

If successful, sentry will run and create the root certs in the given directory. This command uses default configuration values as no custom config file was given. See below on how to start Sentry with a custom configuration.

Setting up mTLS with the configuration resource

Dapr instance configuration

When running Dapr in self hosted mode, mTLS is disabled by default. you can enable it by creating the following configuration file:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: daprsystem
  5.   namespace: default
  6. spec:
  7.   mtls:
  8.     enabled: true

In addition to the Dapr configuration, you will also need to provide the TLS certificates to each Dapr sidecar instance. You can do so by setting the following environment variables before running the Dapr instance:

  1. export DAPR_TRUST_ANCHORS=`cat $HOME/.dapr/certs/ca.crt`
  2. export DAPR_CERT_CHAIN=`cat $HOME/.dapr/certs/issuer.crt`
  3. export DAPR_CERT_KEY=`cat $HOME/.dapr/certs/issuer.key`
  4. export NAMESPACE=default
  1. $env:DAPR_TRUST_ANCHORS=$(Get-Content -raw $env:USERPROFILE\.dapr\certs\ca.crt)
  2. $env:DAPR_CERT_CHAIN=$(Get-Content -raw $env:USERPROFILE\.dapr\certs\issuer.crt)
  3. $env:DAPR_CERT_KEY=$(Get-Content -raw $env:USERPROFILE\.dapr\certs\issuer.key)
  4. $env:NAMESPACE="default"

If using the Dapr CLI, point Dapr to the config file above to run the Dapr instance with mTLS enabled:

  1. dapr run --app-id myapp --config ./config.yaml node myapp.js

If using daprd directly, use the following flags to enable mTLS:

  1. daprd --app-id myapp --enable-mtls --sentry-address localhost:50001 --config=./config.yaml

Sentry configuration

Here’s an example of a configuration for Sentry that changes the workload cert TTL to 25 seconds:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: daprsystem
  5.   namespace: default
  6. spec:
  7.   mtls:
  8.     enabled: true
  9.     workloadCertTTL: "25s"

In order to start Sentry with a custom config, use the following flag:

  1. ./sentry --issuer-credentials $HOME/.dapr/certs --trust-domain cluster.local --config=./config.yaml

Bringing your own certificates

In order to provide your own credentials, create ECDSA PEM encoded root and issuer certificates and place them on the file system. Tell Sentry where to load the certificates from using the --issuer-credentials flag.

The next examples creates root and issuer certs and loads them with Sentry.

Note: This example uses the step tool to create the certificates. You can install step tool from here. Windows binaries available here

Create the root certificate:

  1. step certificate create cluster.local ca.crt ca.key --profile root-ca --no-password --insecure

Create the issuer certificate:

  1. step certificate create cluster.local issuer.crt issuer.key --ca ca.crt --ca-key ca.key --profile intermediate-ca --not-after 8760h --no-password --insecure

This creates the root and issuer certs and keys. Place ca.crt, issuer.crt and issuer.key in a desired path ($HOME/.dapr/certs in the example below), and launch Sentry:

  1. ./sentry --issuer-credentials $HOME/.dapr/certs --trust-domain cluster.local

Updating Root or Issuer Certs

If the Root or Issuer certs are about to expire, you can update them and restart the required system services.

First, issue new certificates using the step above in Bringing your own certificates.

Copy ca.crt, issuer.crt and issuer.key to the filesystem path of every configured system service, and restart the process or container. By default, system services will look for the credentials in /var/run/dapr/credentials.

Note:If you signed the cert root with a different private key, restart the Dapr instances.

Last modified September 17, 2021 : Merge pull request #1757 from georgestevens99/1440SecretKeyRefExplanation (620a5f8)