Shutting down the cluster gracefully

This document describes the process to gracefully shut down your cluster. You might need to temporarily shut down your cluster for maintenance reasons, or to save on resource costs.

Prerequisites

  • Take an etcd backup prior to shutting down the cluster.

Shutting down the cluster

You can shut down your cluster in a graceful manner so that it can be restarted at a later date.

You can shut down a cluster until a year from the installation date and expect it to restart gracefully. After a year from the installation date, the cluster certificates expire.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

  • You have taken an etcd backup.

    It is important to take an etcd backup before performing this procedure so that your cluster can be restored if you encounter any issues when restarting the cluster.

Procedure

  1. If you are shutting the cluster down for an extended period, determine the date on which certificates expire.

    1. $ oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.metadata.annotations.auth\.openshift\.io/certificate-not-after}'

    Example output

    1. 2022-08-05T14:37:50Zuser@user:~ $ (1)
    1To ensure that the cluster can restart gracefully, plan to restart it on or before the specified date. As the cluster restarts, the process might require you to manually approve the pending certificate signing requests (CSRs) to recover kubelet certificates.

  2. Shut down all of the nodes in the cluster. You can do this from your cloud provider’s web console, or run the following loop:

    1. $ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do oc debug node/${node} -- chroot /host shutdown -h 1; done

    Example output

    1. Starting pod/ip-10-0-130-169us-east-2computeinternal-debug ...
    2. To use host binaries, run `chroot /host`
    3. Shutdown scheduled for Mon 2021-09-13 09:36:17 UTC, use 'shutdown -c' to cancel.
    5. Removing debug pod ...
    6. Starting pod/ip-10-0-150-116us-east-2computeinternal-debug ...
    7. To use host binaries, run `chroot /host`
    8. Shutdown scheduled for Mon 2021-09-13 09:36:29 UTC, use 'shutdown -c' to cancel.

    Shutting down the nodes using one of these methods allows pods to terminate gracefully, which reduces the chance for data corruption.

    It is not necessary to drain control plane nodes (also known as the master nodes) of the standard pods that ship with OKD prior to shutdown.

    Cluster administrators are responsible for ensuring a clean restart of their own workloads after the cluster is restarted. If you drained control plane nodes prior to shutdown because of custom workloads, you must mark the control plane nodes as schedulable before the cluster will be functional again after restart.

  3. Shut off any cluster dependencies that are no longer needed, such as external storage or an LDAP server. Be sure to consult your vendor’s documentation before doing so.

Additional resources