Configuring a heterogeneous cluster

A heterogeneous cluster is a cluster that supports worker machines with different architectures. You can deploy a heterogeneous cluster by creating an Azure installer-provisioned cluster using the heterogeneous installer binary. For Azure installation, see Installing on Azure with customizations.

The heterogeneous clusters Technology Preview feature has limited usability with installing, upgrading, and running payloads.

The following procedures explain how to generate an arm64 boot image and create an Azure machine set with the arm64 boot image. This will add arm64 worker nodes to your heterogeneous cluster and deploy the desired amount of ARM64 virtual machines (VM). This section also shows how to upgrade your existing cluster to a heterogeneous cluster. Heterogeneous clusters are only available on Azure installer-provisioned infrastructures with x86_64 control planes.

Heterogeneous clusters for OKD on Azure installer-provisioned infrastructure installations is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

To configure your heterogeneous cluster, you must create an arm64 boot image and add it to your Azure machine set. The following procedure describes how to manually generate an arm64 boot image.

Prerequisites

  • You installed the Azure CLI (az).

  • You created a single-architecture Azure installer-provisioned cluster with the heterogeneous installer binary.

Procedure

  1. Log in to your Azure account:

    1. $ az login

  2. Create a storage account and upload the arm64 virtual hard disk (VHD) to your storage account. The OKD installation program creates a resource group, however, the boot image can also be uploaded to a custom named resource group:

    1. $ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS (1)
    1The westus object is an example region.

  3. Create a storage container using the storage account you generated:

    1. $ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}

  4. You must use the OKD installation program JSON file to extract the URL and arch64 VHD name:

    1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:

      1. $ RHCOS_VHD_ORIGIN_URL=$(./openshift-install coreos print-stream-json | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".url')

    2. Extract the aarch64 VHD name and set it to BLOB_NAME as the file name by running the following command:

      1. $ BLOB_NAME=rhcos-$(./openshift-install coreos print-stream-json | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".release')-azure.aarch64.vhd

  5. Generate a shared access signature (SAS) token. Use this token to upload the FCOS VHD to your storage container with the following commands:

    1. $ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`
    1. $ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`

  6. Copy the FCOS VHD into the storage container:

    1. $ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
    2.  --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
    3.  --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

    You can check the status of the copying process with the following command:

    1. $ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy

    Example output

    1. {
    2.  "completionTime": null,
    3.  "destinationSnapshot": null,
    4.  "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
    5.  "incrementalCopy": null,
    6.  "progress": "17179869696/17179869696",
    7.  "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
    8.  "status": "success", (1)
    9.  "statusDescription": null
    10. }
    1If the status parameter displays the success object, the copying process is complete.

  7. Create an image gallery using the following command:

    1. $ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

    Use the image gallery to create an image definition. In the following example command, rhcos-arm64 is the name of the image definition.

    1. $ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --publisher RedHat --offer arm --sku arm64 --os-type linux --architecture Arm64 --hyper-v-generation V2

  8. To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:

    1. $ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)

  9. Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version. In the following example, 1.0.0 is the image version.

    1. $ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}

  10. Your arm64 boot image is now generated. You can access the ID of your image with the following command:

    1. $ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-arm64 -e 1.0.0

    The following example image ID is used in the recourseID parameter of the machine set:

    Example resourceID

    1. /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0

Adding a machine set to your cluster using the arm64 boot image

To add arm64 worker nodes to your heterogeneous cluster, you must create an Azure machine set that uses the arm64 boot image. To create your own custom machine set on Azure, see “Creating a machine set on Azure”.

Prerequisites

  • You installed the OpenShift CLI (oc).

Procedure

  • Create a machine set and modify the resourceID and vmSize parameters with the following command. This machine set will control the arm64 worker nodes in your cluster:

    1. $ oc create -f arm64-machine-set-0.yaml

    Sample YAML machine set with arm64 boot image

    1. apiVersion: machine.openshift.io/v1beta1
    2. kind: MachineSet
    3. metadata:
    4.   labels:
    5.     machine.openshift.io/cluster-api-cluster: <infrastructure_id>
    6.     machine.openshift.io/cluster-api-machine-role: worker
    7.     machine.openshift.io/cluster-api-machine-type: worker
    8.   name: <infrastructure_id>-arm64-machine-set-0
    9.   namespace: openshift-machine-api
    10. spec:
    11.   replicas: 2
    12.   selector:
    13.     matchLabels:
    14.       machine.openshift.io/cluster-api-cluster: <infrastructure_id>
    15.       machine.openshift.io/cluster-api-machineset: <infrastructure_id>-arm64-machine-set-0
    16.   template:
    17.     metadata:
    18.       labels:
    19.         machine.openshift.io/cluster-api-cluster: <infrastructure_id>
    20.         machine.openshift.io/cluster-api-machine-role: worker
    21.         machine.openshift.io/cluster-api-machine-type: worker
    22.         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-arm64-machine-set-0
    23.     spec:
    24.       lifecycleHooks: {}
    25.       metadata: {}
    26.       providerSpec:
    27.         value:
    28.           acceleratedNetworking: true
    29.           apiVersion: machine.openshift.io/v1beta1
    30.           credentialsSecret:
    31.             name: azure-cloud-credentials
    32.             namespace: openshift-machine-api
    33.           image:
    34.             offer: ""
    35.             publisher: ""
    36.             resourceID: /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0 (1)
    37.             sku: ""
    38.             version: ""
    39.           kind: AzureMachineProviderSpec
    40.           location: <region>
    41.           managedIdentity: <infrastructure_id>-identity
    42.           networkResourceGroup: <infrastructure_id>-rg
    43.           osDisk:
    44.             diskSettings: {}
    45.             diskSizeGB: 128
    46.             managedDisk:
    47.               storageAccountType: Premium_LRS
    48.             osType: Linux
    49.           publicIP: false
    50.           publicLoadBalancer: <infrastructure_id>
    51.           resourceGroup: <infrastructure_id>-rg
    52.           subnet: <infrastructure_id>-worker-subnet
    53.           userDataSecret:
    54.             name: worker-user-data
    55.           vmSize: Standard_D4ps_v5 (2)
    56.           vnet: <infrastructure_id>-vnet
    57.           zone: "<zone>"
    1Set the resourceID parameter to the arm64 boot image.
    2Set the vmSize parameter to the instance type used in your installation. Some example instance types are Standard_D4ps_v5 or D8ps.

Verification

  1. Verify that the new ARM64 machines are running by entering the following command:

    1. $ oc get machineset -n openshift-machine-api

    Example output

    1. NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
    2. <infrastructure_id>-arm64-machine-set-0                   2        2      2          2  10m

  2. You can check that the nodes are ready and scheduable with the following command:

    1. $ oc get nodes

Additional resources

Upgrading your heterogeneous cluster

You must perform an explicit upgrade command to upgrade your existing cluster to a heterogeneous cluster.

Prerequisites

  • You installed the OpenShift CLI (oc).

Procedure

  • To manually upgrade your cluster, use the following command:
  1. $ oc adm upgrade --allow-explicit-upgrade --to-image <image-pullspec> (1)
1You can access the image-pullspec object from the mixed-arch mirrors page in the release.txt file.