k3s secrets-encrypt
K3s supports enabling secrets encryption at rest. For more information, see Secrets Encryption.
Secrets Encryption Tool
Version Gate
Available as of v1.21.8+k3s1
K3s contains a CLI tool secrets-encrypt
, which enables automatic control over the following:
- Disabling/Enabling secrets encryption
- Adding new encryption keys
- Rotating and deleting encryption keys
- Reencrypting secrets
warning
Failure to follow proper procedure for rotating encryption keys can leave your cluster permanently corrupted. Proceed with caution.
New Encryption Key Rotation (Experimental)
Version Gate
Available as of v1.28.1+k3s1. This new version of the tool utilized K8s automatic config reloading which is currently in beta. GA is expected in v1.29.0
For older releases, see Encryption Key Rotation Classic
- Single-Server
- High-Availability
To rotate secrets encryption keys on a single-server cluster:
Start the K3s server with the flag
--secrets-encryption
note
Starting K3s without encryption and enabling it at a later time is currently not supported.
Rotate secrets encryption keys
k3s secrets-encrypt rotate-keys
Wait for reencryption to finish. Watch the server logs, or wait for:
$ k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
To rotate secrets encryption keys on HA setups:
Start up all three K3s servers with the
--secrets-encryption
flag. For brevity, the servers will be referred to as S1, S2, S3.note
Starting K3s without encryption and enabling it at a later time is currently not supported.
Rotate secrets encryption keys on S1
k3s secrets-encrypt rotate-keys
Wait for reencryption to finish. Watch the server logs, or wait for:
$ k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
info
K3s will reencrypt ~5 secrets per second. Clusters with large # of secrets can take several minutes to reencrypt. You can track progress in the server logs.
Restart K3s on S1 with same arguments. If running K3s as a service:
# If using systemd
systemctl restart k3s
# If using openrc
rc-service k3s restart
Once S1 is up, restart K3s on S2 and S3
Encryption Key Rotation Classic
- Single-Server
- High-Availability
To rotate secrets encryption keys on a single-server cluster:
Start the K3s server with the flag
--secrets-encryption
note
Starting K3s without encryption and enabling it at a later time is currently not supported.
Prepare
k3s secrets-encrypt prepare
Kill and restart the K3s server with same arguments. If running K3s as a service:
# If using systemd
systemctl restart k3s
# If using openrc
rc-service k3s restart
Rotate
k3s secrets-encrypt rotate
Kill and restart the K3s server with same arguments
Reencrypt
info
K3s will reencrypt ~5 secrets per second.
Clusters with large # of secrets can take several minutes to reencrypt.k3s secrets-encrypt reencrypt
The steps are the same for both embedded DB and external DB clusters.
To rotate secrets encryption keys on HA setups:
Start up all three K3s servers with the
--secrets-encryption
flag. For brevity, the servers will be referred to as S1, S2, S3.Notes
- Starting K3s without encryption and enabling it at a later time is currently not supported.
- While not required, it is recommended that you pick one server node from which to run the
secrets-encrypt
commands.
Prepare on S1
k3s secrets-encrypt prepare
Kill and restart S1 with same arguments. If running K3s as a service:
# If using systemd
systemctl restart k3s
# If using openrc
rc-service k3s restart
Once S1 is up, kill and restart the S2 and S3
Rotate on S1
k3s secrets-encrypt rotate
Kill and restart S1 with same arguments
Once S1 is up, kill and restart the S2 and S3
Reencrypt on S1
info
K3s will reencrypt ~5 secrets per second.
Clusters with large # of secrets can take several minutes to reencrypt.k3s secrets-encrypt reencrypt
Kill and restart S1 with same arguments
Once S1 is up, kill and restart the S2 and S3
Secrets Encryption Disable/Re-enable
- Single-Server
- High-Availability
After launching a server with --secrets-encryption
flag, secrets encryption can be disabled.
To disable secrets encryption on a single-node cluster:
Disable
k3s secrets-encrypt disable
Kill and restart the K3s server with same arguments. If running K3s as a service:
# If using systemd
systemctl restart k3s
# If using openrc
rc-service k3s restart
Reencrypt with flags
k3s secrets-encrypt reencrypt --force --skip
To re-enable secrets encryption on a single node cluster:
Enable
k3s secrets-encrypt enable
Kill and restart the K3s server with same arguments
Reencrypt with flags
k3s secrets-encrypt reencrypt --force --skip
After launching a HA cluster with --secrets-encryption
flags, secrets encryption can be disabled.
note
While not required, it is recommended that you pick one server node from which to run the secrets-encrypt
commands.
For brevity, the three servers used in this guide will be referred to as S1, S2, S3.
To disable secrets encryption on a HA cluster:
Disable on S1
k3s secrets-encrypt disable
Kill and restart S1 with same arguments. If running K3s as a service:
# If using systemd
systemctl restart k3s
# If using openrc
rc-service k3s restart
Once S1 is up, kill and restart the S2 and S3
Reencrypt with flags on S1
k3s secrets-encrypt reencrypt --force --skip
To re-enable secrets encryption on a HA cluster:
Enable on S1
k3s secrets-encrypt enable
Kill and restart S1 with same arguments
Once S1 is up, kill and restart the S2 and S3
Reencrypt with flags on S1
k3s secrets-encrypt reencrypt --force --skip
Secrets Encryption Status
The secrets-encrypt tool includes a status
command that displays information about the current status of secrets encryption on the node.
An example of the command on a single-server node:
$ k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match
Active Key Type Name
------ -------- ----
* AES-CBC aescbckey
Another example on HA cluster, after rotating the keys, but before restarting the servers:
$ k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: hash does not match between node-1 and node-2
Active Key Type Name
------ -------- ----
* AES-CBC aescbckey-2021-12-10T22:54:38Z
AES-CBC aescbckey
Details on each section are as follows:
- Encryption Status: Displayed whether secrets encryption is disabled or enabled on the node
- Current Rotation Stage: Indicates the current rotation stage on the node.
Stages are:start
,prepare
,rotate
,reencrypt_request
,reencrypt_active
,reencrypt_finished
- Server Encryption Hashes: Useful for HA clusters, this indicates whether all servers are on the same stage with their local files. This can be used to identify whether a restart of servers is required before proceeding to the next stage. In the HA example above, node-1 and node-2 have different hashes, indicating that they currently do not have the same encryption configuration. Restarting the servers will sync up their configuration.
- Key Table: Summarizes information about the secrets encryption keys found on the node.
- Active: The “*“ indicates which, if any, of the keys are currently used for secrets encryption. An active key is used by Kubernetes to encrypt any new secrets.
- Key Type: All keys using this tool are
AES-CBC
type. See more info here. - Name: Name of the encryption key.