Rolling Updates to TLS for Existing Clusters on Kubernetes

As of Consul Helm version 0.16.0, the chart supports TLS for communication within the cluster. If you already have a Consul cluster deployed on Kubernetes, you may want to configure TLS in a way that minimizes downtime to your applications. Consul already supports rolling out TLS on an existing cluster without downtime. However, depending on your Kubernetes use case, your upgrade procedure may be different.

Gradual TLS Rollout without Consul Service Mesh

If you do not use a service mesh, follow this process.

  1. Run a Helm upgrade with the following config:

    1. global:
    2.   tls:
    3.     enabled: true
    4.     # This configuration sets `verify_outgoing`, `verify_server_hostname`,
    5.     # and `verify_incoming` to `false` on servers and clients,
    6.     # which allows TLS-disabled nodes to join the cluster.
    7.     verify: false
    8. server:
    9.   updatePartition: <number_of_server_replicas>

    This upgrade trigger a rolling update of consul-k8s components.

  2. Perform a rolling upgrade of the servers, as described in Upgrade Consul Servers.

  3. Repeat steps 1 and 2, turning on TLS verification by setting global.tls.verify to true.

Gradual TLS Rollout with Consul Service Mesh

Because the sidecar Envoy proxies need to talk to the Consul client agent regularly for service discovery, we can’t enable TLS on the clients without also re-injecting a TLS-enabled proxy into the application pods. To perform TLS rollout with minimal downtime, we recommend instead to add a new Kubernetes node pool and migrate your applications to it.

  1. Add a new identical node pool.

  2. Cordon all nodes in the old pool by running kubectl cordon. This command ensures Kubernetes does not schedule any new workloads on those nodes, and instead schedules onto the new TLS-enabled nodes.

  3. Create the following Helm config file for the upgrade:

    1. global:
    2.   tls:
    3.     enabled: true
    4.     # This configuration sets `verify_outgoing`, `verify_server_hostname`,
    5.     # and `verify_incoming` to `false` on servers and clients,
    6.     # which allows TLS-disabled nodes to join the cluster.
    7.     verify: false
    8. server:
    9.   updatePartition: <number_of_server_replicas>
    10. client:
    11.   updateStrategy: |
    12.     type: OnDelete

    In this configuration, we’re setting server.updatePartition to the number of server replicas as described in Upgrade Consul Servers.

  4. Run helm upgrade with the above config file.

  5. At this point, all components (e.g., Consul service mesh webhook and sync catalog) should be running on the new node pool.

  6. Redeploy all your mesh-enabled applications. One way to trigger a redeploy is to run kubectl drain on the nodes in the old pool. Now that the service mesh webhook is TLS-aware, it adds TLS configuration to the sidecar proxy. Also, Kubernetes should schedule these applications on the new node pool.

  7. Perform a rolling upgrade of the servers described in Upgrade Consul Servers.

  8. If everything is healthy, delete the old node pool.

  9. Finally, set global.tls.verify to true in your Helm config file, remove the client.updateStrategy property, and perform a rolling upgrade of the servers.

Note: It is possible to do this upgrade without fully duplicating the node pool. You could drain a subset of the Kubernetes nodes within your existing node pool and treat it as your “new node pool.” Then follow the above instructions. Repeat this process for the rest of the nodes in the node pool.