Scaling a user-provisioned cluster with the Bare Metal Operator

After deploying a user-provisioned infrastructure cluster, you can use the Bare Metal Operator (BMO) and other metal3 components to scale bare-metal hosts in the cluster. This approach helps you to scale a user-provisioned cluster in a more automated way.

About scaling a user-provisioned cluster with the Bare Metal Operator

You can scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO) and other metal3 components. User-provisioned infrastructure installations do not feature the Machine API Operator. The Machine API Operator typically manages the lifecycle of bare-metal hosts in a cluster. However, it is possible to use the BMO and other metal3 components to scale nodes in user-provisioned clusters without requiring the Machine API Operator.

Prerequisites for scaling a user-provisioned cluster

  • You installed a user-provisioned infrastructure cluster on bare metal.

  • You have baseboard management controller (BMC) access to the hosts.

Limitations for scaling a user-provisioned cluster

  • You cannot use a provisioning network to scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO).

    • Consequentially, you can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.

  • You cannot scale MachineSet objects in user-provisioned infrastructure clusters by using the BMO.

Configuring a provisioning resource to scale user-provisioned clusters

Create a Provisioning custom resource (CR) to enable Metal platform components on a user-provisioned infrastructure cluster.

Prerequisites

  • You installed a user-provisioned infrastructure cluster on bare metal.

Procedure

  1. Create a Provisioning CR.

    1. Save the following YAML in the provisioning.yaml file:

      1. apiVersion: metal3.io/v1alpha1
      2. kind: Provisioning
      3. metadata:
      4.   name: provisioning-configuration
      5. spec:
      6.   provisioningNetwork: "Disabled"
      7.   watchAllNamespaces: false

      OKD 4 does not support enabling a provisioning network when you scale a user-provisioned cluster by using the Bare Metal Operator.

  2. Create the Provisioning CR by running the following command:

    1. $ oc create -f provisioning.yaml

    Example output

    1. provisioning.metal3.io/provisioning-configuration created

Verification

  • Verify that the provisioning service is running by running the following command:

    1. $ oc get pods -n openshift-machine-api

    Example output

    1. NAME                                                  READY   STATUS    RESTARTS        AGE
    2. cluster-autoscaler-operator-678c476f4c-jjdn5          2/2     Running   0               5d21h
    3. cluster-baremetal-operator-6866f7b976-gmvgh           2/2     Running   0               5d21h
    4. control-plane-machine-set-operator-7d8566696c-bh4jz   1/1     Running   0               5d21h
    5. ironic-proxy-64bdw                                    1/1     Running   0               5d21h
    6. ironic-proxy-rbggf                                    1/1     Running   0               5d21h
    7. ironic-proxy-vj54c                                    1/1     Running   0               5d21h
    8. machine-api-controllers-544d6849d5-tgj9l              7/7     Running   1 (5d21h ago)   5d21h
    9. machine-api-operator-5c4ff4b86d-6fjmq                 2/2     Running   0               5d21h
    10. metal3-6d98f84cc8-zn2mx                               5/5     Running   0               5d21h
    11. metal3-image-customization-59d745768d-bhrp7           1/1     Running   0               5d21h

Provisioning new hosts in a user-provisioned cluster by using the BMO

You can use the Bare Metal Operator (BMO) to provision bare-metal hosts in a user-provisioned cluster by creating a BareMetalHost custom resource (CR).

To provision bare-metal hosts to the cluster by using the BMO, you must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to false.

Prerequisites

  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Create the Secret CR and the BareMetalHost CR.

    1. Save the following YAML in the bmh.yaml file:

      1. ---
      2. apiVersion: v1
      3. kind: Secret
      4. metadata:
      5.   name: worker1-bmc
      6.   namespace: openshift-machine-api
      7. type: Opaque
      8. data:
      9.   username: <base64_of_uid>
      10.   password: <base64_of_pwd>
      11. ---
      12. apiVersion: metal3.io/v1alpha1
      13. kind: BareMetalHost
      14. metadata:
      15.   name: worker1
      16.   namespace: openshift-machine-api
      17. spec:
      18.   bmc:
      19.     address: <protocol>://<bmc_url> (1)
      20.     credentialsName: "worker1-bmc"
      21.   bootMACAddress: <nic1_mac_address>
      22.   externallyProvisioned: false (2)
      23.   customDeploy:
      24.     method: install_coreos
      25.   online: true
      26.   userData:
      27.     name: worker-user-data-managed
      28.     namespace: openshift-machine-api
      1You can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.
      2You must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to false. The default value is false.

  2. Create the bare-metal host object by running the following command:

    1. $ oc create -f bmh.yaml

    Example output

    1. secret/worker1-bmc created
    2. baremetalhost.metal3.io/worker1 created

  3. Approve all certificate signing requests (CSRs).

    1. Verify that the provisioning state of the host is provisioned by running the following command:

      1. $ oc get bmh -A

      Example output

      1. NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
      2. openshift-machine-api   controller1   externally provisioned              true             5m25s
      3. openshift-machine-api   worker1       provisioned                         true             4m45s

    2. Get the list of pending CSRs by running the following command:

      1. $ oc get csr

      Example output

      1. NAME        AGE   SIGNERNAME                                    REQUESTOR                                         REQUESTEDDURATION CONDITION
      2. csr-gfm9f   33s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-o
      3. perator:node-bootstrapper   <none>              Pending

    3. Approve the CSR by running the following command:

      1. $ oc adm certificate approve <csr_name>

      Example output

      1. certificatesigningrequest.certificates.k8s.io/<csr_name> approved

Verification

  • Verify that the node is ready by running the following command:

    1. $ oc get nodes

    Example output

    1. NAME        STATUS   ROLES           AGE     VERSION
    2. app1        Ready    worker          47s     v1.24.0+dc5a2fd
    3. controller1 Ready    master,worker   2d22h   v1.24.0+dc5a2fd

Optional: Managing existing hosts in a user-provisioned cluster by using the BMO

Optionally, you can use the Bare Metal Operator (BMO) to manage existing bare-metal controller hosts in a user-provisioned cluster by creating a BareMetalHost object for the existing host. It is not a requirement to manage existing user-provisioned hosts; however, you can enroll them as externally-provisioned hosts for inventory purposes.

To manage existing hosts by using the BMO, you must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to true to prevent the BMO from re-provisioning the host.

Prerequisites

  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Create the Secret CR and the BareMetalHost CR.

    1. Save the following YAML in the controller.yaml file:

      1. ---
      2. apiVersion: v1
      3. kind: Secret
      4. metadata:
      5.   name: controller1-bmc
      6.   namespace: openshift-machine-api
      7. type: Opaque
      8. data:
      9.   username: <base64_of_uid>
      10.   password: <base64_of_pwd>
      11. ---
      12. apiVersion: metal3.io/v1alpha1
      13. kind: BareMetalHost
      14. metadata:
      15.   name: controller1
      16.   namespace: openshift-machine-api
      17. spec:
      18.   bmc:
      19.     address: <protocol>://<bmc_url> (1)
      20.     credentialsName: "controller1-bmc"
      21.   bootMACAddress: <nic1_mac_address>
      22.   customDeploy:
      23.     method: install_coreos
      24.   externallyProvisioned: true (2)
      25.   online: true
      26.   userData:
      27.     name: controller-user-data-managed
      28.     namespace: openshift-machine-api
      1You can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.
      2You must set the value to true to prevent the BMO from re-provisioning the bare-metal controller host.

  2. Create the bare-metal host object by running the following command:

    1. $ oc create -f controller.yaml

    Example output

    1. secret/controller1-bmc created
    2. baremetalhost.metal3.io/controller1 created

Verification

  • Verify that the BMO created the bare-metal host object by running the following command:

    1. $ oc get bmh -A

    Example output

    1. NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
    2. openshift-machine-api   controller1   externally provisioned              true             13s

Removing hosts from a user-provisioned cluster by using the BMO

You can use the Bare Metal Operator (BMO) to remove bare-metal hosts from a user-provisioned cluster.

Prerequisites

  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Cordon and drain the host by running the following command:

    1. $ oc adm drain app1 --force --ignore-daemonsets=true

    Example output

    1. node/app1 cordoned
    2. WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-tvthg, openshift-dns/dns-
    3. default-9q6rz, openshift-dns/node-resolver-zvt42, openshift-image-registry/node-ca-mzxth, openshift-ingress-cana
    4. ry/ingress-canary-qq5lf, openshift-machine-config-operator/machine-config-daemon-v79dm, openshift-monitoring/nod
    5. e-exporter-2vn59, openshift-multus/multus-additional-cni-plugins-wssvj, openshift-multus/multus-fn8tg, openshift
    6. -multus/network-metrics-daemon-5qv55, openshift-network-diagnostics/network-check-target-jqxn2, openshift-ovn-ku
    7. bernetes/ovnkube-node-rsvqg
    8. evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766965-258vp
    9. evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766950-kg5mk
    10. evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766935-stf4s
    11. pod/collect-profiles-27766965-258vp evicted
    12. pod/collect-profiles-27766950-kg5mk evicted
    13. pod/collect-profiles-27766935-stf4s evicted
    14. node/app1 drained

  2. Delete the customDeploy specification from the BareMetalHost CR.

    1. Edit the BareMetalHost CR for the host by running the following command:

      1. $ oc edit bmh -n openshift-machine-api <host_name>

    2. Delete the lines spec.customDeploy and spec.customDeploy.method:

      1. ...
      2.   customDeploy:
      3.     method: install_coreos

    3. Verify that the provisioning state of the host changes to deprovisioning by running the following command:

      1. $ oc get bmh -A

      Example output

      1. NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
      2. openshift-machine-api   controller1   externally provisioned              true             58m
      3. openshift-machine-api   worker1       deprovisioning                      true             57m

  3. Delete the node by running the following command:

    1. $ oc delete node <node_name>

Verification

  • Verify the node is deleted by running the following command:

    1. $ oc get nodes

    Example output

    1. NAME          STATUS   ROLES           AGE     VERSION
    2. controller1   Ready    master,worker   2d23h   v1.24.0+dc5a2fd