Adding compute machines to vSphere

You can add more compute machines to your OKD cluster on VMware vSphere.

Prerequisites

If you do not have access to the Fedora CoreOS (FCOS) images that were used to create your cluster, you can add more compute machines to your OKD cluster with newer versions of Fedora CoreOS (FCOS) images. For instructions, see Adding new nodes to UPI cluster fails after upgrading to OpenShift 4.6+.

Creating more Fedora CoreOS (FCOS) machines in vSphere

You can create more compute machines for your cluster that uses user-provisioned infrastructure on VMware vSphere.

Prerequisites

  • Obtain the base64-encoded Ignition file for your compute machines.

  • You have access to the vSphere template that you created for your cluster.

Procedure

  1. After the template deploys, deploy a VM for a machine in the cluster.

    1. Right-click the template’s name and click Clone → Clone to Virtual Machine.

    2. On the Select a name and folder tab, specify a name for the VM. You might include the machine type in the name, such as compute-1.

    3. On the Select a name and folder tab, select the name of the folder that you created for the cluster.

    4. On the Select a compute resource tab, select the name of a host in your datacenter.

    5. Optional: On the Select storage tab, customize the storage options.

    6. On the Select clone options, select Customize this virtual machine’s hardware.

    7. On the Customize hardware tab, click VM Options → Advanced.

      • From the Latency Sensitivity list, select High.

      • Click Edit Configuration, and on the Configuration Parameters window, click Add Configuration Params. Define the following parameter names and values:

        • guestinfo.ignition.config.data: Paste the contents of the base64-encoded compute Ignition config file for this machine type.

        • guestinfo.ignition.config.data.encoding: Specify base64.

        • disk.EnableUUID: Specify TRUE.

  1. 8.  In the **Virtual Hardware** panel of the **Customize hardware** tab, modify the specified values as required. Ensure that the amount of RAM, CPU, and disk storage meets the minimum requirements for the machine type. Also, make sure to select the correct network under **Add network adapter** if there are multiple networks available.
  3. 9.  Complete the configuration and power on the VM.
  1. Continue to create more compute machines for your cluster.

Approving the certificate signing requests for your machines

When you add machines to a cluster, two pending certificate signing requests (CSRs) are generated for each machine that you added. You must confirm that these CSRs are approved or, if necessary, approve them yourself. The client requests must be approved first, followed by the server requests.

Prerequisites

  • You added machines to your cluster.

Procedure

  1. Confirm that the cluster recognizes the machines:

    1. $ oc get nodes

    Example output

    1. NAME      STATUS    ROLES   AGE  VERSION
    2. master-0  Ready     master  63m  v1.19.0
    3. master-1  Ready     master  63m  v1.19.0
    4. master-2  Ready     master  64m  v1.19.0
    5. worker-0  NotReady  worker  76s  v1.19.0
    6. worker-1  NotReady  worker  70s  v1.19.0

    The output lists all of the machines that you created.

    The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

  2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

    1. $ oc get csr

    Example output

    1. NAME        AGE     REQUESTOR                                                                   CONDITION
    2. csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
    3. csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
    4. ...

    In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

  3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

    Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. Once the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

    For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

    • To approve them individually, run the following command for each valid CSR:

      1. $ oc adm certificate approve <csr_name> (1)
      1<csr_name> is the name of a CSR from the list of current CSRs.

    • To approve all pending CSRs, run the following command:

      1. $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

      Some Operators might not become available until some CSRs are approved.

  4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

    1. $ oc get csr

    Example output

    1. NAME        AGE     REQUESTOR                                                                   CONDITION
    2. csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
    3. csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
    4. ...

  5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

    • To approve them individually, run the following command for each valid CSR:

      1. $ oc adm certificate approve <csr_name> (1)
      1<csr_name> is the name of a CSR from the list of current CSRs.

    • To approve all pending CSRs, run the following command:

      1. $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

  6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

    1. $ oc get nodes

    Example output

    1. NAME      STATUS    ROLES   AGE  VERSION
    2. master-0  Ready     master  73m  v1.20.0
    3. master-1  Ready     master  73m  v1.20.0
    4. master-2  Ready     master  74m  v1.20.0
    5. worker-0  Ready     worker  11m  v1.20.0
    6. worker-1  Ready     worker  11m  v1.20.0

    It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Additional information