Using the Container Linux Update Operator with Rook

When you are using Container Linux (CoreOS) and have the update engine enabled, it could be that a node reboots quickly after another not leaving enough time for the Rook cluster to rebuild. The Container Linux Update Operator is the solution for this, you can block your nodes to reboot until the Ceph cluster is healthy.

Prerequisites

  • An operational Container Linux Kubernetes cluster (Successfully tested with 1.8.4)
  • A working rook cluster
  • The update-engine.service systemd unit on each machine should be unmasked, enabled and started in systemd
  • The locksmithd.service systemd unit on each machine should be masked and stopped in systemd

Start the update operator

Proper reading of the README on the Container Linux Update Operator is necessary. Clone the repo and go in the examples directory.

Look for the file named update-operator.yaml and update the command part of the container from:

  1. command:
  2. - "/bin/update-operator"

to:

  1. command:
  2. - "/bin/update-operator"
  3. - "--before-reboot-annotations"
  4. - "ceph-before-reboot-check"
  5. - "--after-reboot-annotations"
  6. - "ceph-after-reboot-check"

You can also add the -v 6 argument for more extensive logging.

Now create the update-operator by invoking following commands:

  1. kubectl create -f namespace.yaml
  2. kubectl create -f cluster-role.yaml
  3. kubectl create -f cluster-role-binding.yaml
  4. kubectl create -f update-operator.yaml
  5. kubectl create -f update-agent.yaml

These files create a new namespace reboot-coordinator, configured to listen for the node annotation ceph-reboot-check. Now you can create both files in the cluster/examples/coreos folder, here’s a short description of what each file does:

  • rbac.yaml: This file contains the necessary RBAC settings.
  • ceph-after-reboot-script.yaml: This file creates a ConfigMap containing a bash script which will be mounted in the rook-toolbox image as executable file.
  • ceph-before-reboot-script.yaml: This file creates a ConfigMap containing a bash script which will be mounted in the rook-toolbox image as executable file.
  • before-reboot-daemonset.yaml: This file creates a DaemonSet which waits for a node being labeled before-reboot=true, runs and checks the Ceph status. If all is correct, it annotates the node with ceph-before-reboot-check=true.
  • after-reboot-daemonset.yaml: This file creates a DaemonSet which waits for a node being labeled after-reboot=true, runs and unsets the noout option for the ceph OSDs. If all is correct, it annotates the node with ceph-after-reboot-check=true.

The node annotation ceph-no-noout=true can be used to avoid ceph-before-reboot-check from setting the OSD noout flag. This annotation should only be used when deleting a node from a cluster, this way the cluster starts rebalancing immediately, not waiting for the node to come back up.

  1. kubectl create -f rbac.yaml
  2. kubectl create -f ceph-after-reboot-script.yaml
  3. kubectl create -f ceph-before-reboot-script.yaml
  4. kubectl create -f before-reboot-daemonset.yaml
  5. kubectl create -f after-reboot-daemonset.yaml

Destroy the update operator

To destroy all elements created in this file, run:

  1. kubectl delete -f before-reboot-daemonset.yaml
  2. kubectl delete -f after-reboot-daemonset.yaml
  3. kubectl delete -f ceph-after-reboot-script.yaml
  4. kubectl delete -f ceph-before-reboot-script.yaml
  5. kubectl delete -f rbac.yaml

Then you may safely delete the update operator itself: From the directory of the Container Linux Update Operator you cloned earlier, go again into the examples folder and run following commands:

  1. kubectl delete -f update-agent.yaml
  2. kubectl delete -f update-operator.yaml
  3. kubectl delete -f cluster-role-binding.yaml
  4. kubectl delete -f cluster-role.yaml
  5. kubectl delete -f namespace.yaml