Installing managed clusters with RHACM and SiteConfig resources

You can provision OKD clusters at scale with Red Hat Advanced Cluster Management (RHACM) using the assisted service and the GitOps plugin policy generator with core-reduction technology enabled. The zero touch priovisioning (ZTP) pipeline performs the cluster installations. ZTP can be used in a disconnected environment.

GitOps ZTP and Topology Aware Lifecycle Manager

GitOps zero touch provisioning (ZTP) generates installation and configuration CRs from manifests stored in Git. These artifacts are applied to a centralized hub cluster where Red Hat Advanced Cluster Management (RHACM), the assisted service, and the Topology Aware Lifecycle Manager (TALM) use the CRs to install and configure the managed cluster. The configuration phase of the ZTP pipeline uses the TALM to orchestrate the application of the configuration CRs to the cluster. There are several key integration points between GitOps ZTP and the TALM.

Inform policies

By default, GitOps ZTP creates all policies with a remediation action of inform. These policies cause RHACM to report on compliance status of clusters relevant to the policies but does not apply the desired configuration. During the ZTP process, after OpenShift installation, the TALM steps through the created inform policies and enforces them on the target managed cluster(s). This applies the configuration to the managed cluster. Outside of the ZTP phase of the cluster lifecycle, this allows you to change policies without the risk of immediately rolling those changes out to affected managed clusters. You can control the timing and the set of remediated clusters by using TALM.

Automatic creation of ClusterGroupUpgrade CRs

To automate the initial configuration of newly deployed clusters, TALM monitors the state of all ManagedCluster CRs on the hub cluster. Any ManagedCluster CR that does not have a ztp-done label applied, including newly created ManagedCluster CRs, causes the TALM to automatically create a ClusterGroupUpgrade CR with the following characteristics:

  • The ClusterGroupUpgrade CR is created and enabled in the ztp-install namespace.

  • ClusterGroupUpgrade CR has the same name as the ManagedCluster CR.

  • The cluster selector includes only the cluster associated with that ManagedCluster CR.

  • The set of managed policies includes all policies that RHACM has bound to the cluster at the time the ClusterGroupUpgrade is created.

  • Pre-caching is disabled.

  • Timeout set to 4 hours (240 minutes).

The automatic creation of an enabled ClusterGroupUpgrade ensures that initial zero-touch deployment of clusters proceeds without the need for user intervention. Additionally, the automatic creation of a ClusterGroupUpgrade CR for any ManagedCluster without the ztp-done label allows a failed ZTP installation to be restarted by simply deleting the ClusterGroupUpgrade CR for the cluster.

Waves

Each policy generated from a PolicyGenTemplate CR includes a ztp-deploy-wave annotation. This annotation is based on the same annotation from each CR which is included in that policy. The wave annotation is used to order the policies in the auto-generated ClusterGroupUpgrade CR. The wave annotation is not used other than for the auto-generated ClusterGroupUpgrade CR.

All CRs in the same policy must have the same setting for the ztp-deploy-wave annotation. The default value of this annotation for each CR can be overridden in the PolicyGenTemplate. The wave annotation in the source CR is used for determining and setting the policy wave annotation. This annotation is removed from each built CR which is included in the generated policy at runtime.

The TALM applies the configuration policies in the order specified by the wave annotations. The TALM waits for each policy to be compliant before moving to the next policy. It is important to ensure that the wave annotation for each CR takes into account any prerequisites for those CRs to be applied to the cluster. For example, an Operator must be installed before or concurrently with the configuration for the Operator. Similarly, the CatalogSource for an Operator must be installed in a wave before or concurrently with the Operator Subscription. The default wave value for each CR takes these prerequisites into account.

Multiple CRs and policies can share the same wave number. Having fewer policies can result in faster deployments and lower CPU usage. It is a best practice to group many CRs into relatively few waves.

To check the default wave value in each source CR, run the following command against the out/source-crs directory that is extracted from the ztp-site-generate container image:

  1. $ grep -r "ztp-deploy-wave" out/source-crs

Phase labels

The ClusterGroupUpgrade CR is automatically created and includes directives to annotate the ManagedCluster CR with labels at the start and end of the ZTP process.

When ZTP configuration post-installation commences, the ManagedCluster has the ztp-running label applied. When all policies are remediated to the cluster and are fully compliant, these directives cause the TALM to remove the ztp-running label and apply the ztp-done label.

For deployments that make use of the informDuValidator policy, the ztp-done label is applied when the cluster is fully ready for deployment of applications. This includes all reconciliation and resulting effects of the ZTP applied configuration CRs. The ztp-done label affects automatic ClusterGroupUpgrade CR creation by TALM. Do not manipulate this label after the initial ZTP installation of the cluster.

Linked CRs

The automatically created ClusterGroupUpgrade CR has the owner reference set as the ManagedCluster from which it was derived. This reference ensures that deleting the ManagedCluster CR causes the instance of the ClusterGroupUpgrade to be deleted along with any supporting resources.

Overview of deploying managed clusters with ZTP

Red Hat Advanced Cluster Management (RHACM) uses zero touch provisioning (ZTP) to deploy single-node OKD clusters, three-node clusters, and standard clusters. You manage site configuration data as OKD custom resources (CRs) in a Git repository. ZTP uses a declarative GitOps approach for a develop once, deploy anywhere model to deploy the managed clusters.

The deployment of the clusters includes:

  • Installing the host operating system (RHCOS) on a blank server

  • Deploying OKD

  • Creating cluster policies and site subscriptions

  • Making the necessary network configurations to the server operating system

  • Deploying profile Operators and performing any needed software-related configuration, such as performance profile, PTP, and SR-IOV

Overview of the managed site installation process

After you apply the managed site custom resources (CRs) on the hub cluster, the following actions happen automatically:

  1. A Discovery image ISO file is generated and booted on the target host.

  2. When the ISO file successfully boots on the target host it reports the host hardware information to RHACM.

  3. After all hosts are discovered, OKD is installed.

  4. When OKD finishes installing, the hub installs the klusterlet service on the target cluster.

  5. The requested add-on services are installed on the target cluster.

The Discovery image ISO process is complete when the Agent CR for the managed cluster is created on the hub cluster.

The target bare-metal host must meet the networking, firmware, and hardware requirements listed in Recommended single-node OpenShift cluster configuration for vDU application workloads.

Creating the managed bare-metal host secrets

Add the required Secret custom resources (CRs) for the managed bare-metal host to the hub cluster. You need a secret for the ZTP pipeline to access the Baseboard Management Controller (BMC) and a secret for the assisted installer service to pull cluster installation images from the registry.

The secrets are referenced from the SiteConfig CR by name. The namespace must match the SiteConfig namespace.

Procedure

  1. Create a YAML secret file containing credentials for the host Baseboard Management Controller (BMC) and a pull secret required for installing OpenShift and all add-on cluster Operators:

    1. Save the following YAML as the file example-sno-secret.yaml:

      1. apiVersion: v1
      2. kind: Secret
      3. metadata:
      4.   name: example-sno-bmc-secret
      5.   namespace: example-sno (1)
      6. data: (2)
      7.   password: <base64_password>
      8.   username: <base64_username>
      9. type: Opaque
      10. ---
      11. apiVersion: v1
      12. kind: Secret
      13. metadata:
      14.   name: pull-secret
      15.   namespace: example-sno  (3)
      16. data:
      17.   .dockerconfigjson: <pull_secret> (4)
      18. type: kubernetes.io/dockerconfigjson
      1Must match the namespace configured in the related SiteConfig CR
      2Base64-encoded values for password and username
      3Must match the namespace configured in the related SiteConfig CR
      4Base64-encoded pull secret

  2. Add the relative path to example-sno-secret.yaml to the kustomization.yaml file that you use to install the cluster.

Deploying a managed cluster with SiteConfig and ZTP

Use the following procedure to create a SiteConfig custom resource (CR) and related files and initiate the zero touch provisioning (ZTP) cluster deployment.

Prerequisites

  • You have installed the OpenShift CLI (oc).

  • You have logged in to the hub cluster as a user with cluster-admin privileges.

  • You configured the hub cluster for generating the required installation and policy CRs.

  • You created a Git repository where you manage your custom site configuration data. The repository must be accessible from the hub cluster and you must configure it as a source repository for the ArgoCD application. See “Preparing the GitOps ZTP site configuration repository” for more information.

    When you create the source repository, ensure that you patch the ArgoCD application with the argocd/deployment/argocd-openshift-gitops-patch.json patch-file that you extract from the ztp-site-generate container. See “Configuring the hub cluster with ArgoCD”.

  • To be ready for provisioning managed clusters, you require the following for each bare-metal host:

    Network connectivity

    Your network requires DNS. Managed cluster hosts should be reachable from the hub cluster. Ensure that Layer 3 connectivity exists between the hub cluster and the managed cluster host.

    Baseboard Management Controller (BMC) details

    ZTP uses BMC username and password details to connect to the BMC during cluster installation. The GitOps ZTP plugin manages the ManagedCluster CRs on the hub cluster based on the SiteConfig CR in your site Git repo. You create individual BMCSecret CRs for each host manually.

    Procedure

    1. Create the required managed cluster secrets on the hub cluster. These resources must be in a namespace with a name matching the cluster name. For example, in out/argocd/example/siteconfig/example-sno.yaml, the cluster name and namespace is example-sno.

      1. Export the cluster namespace by running the following command:

        1. $ export CLUSTERNS=example-sno

      2. Create the namespace:

        1. $ oc create namespace $CLUSTERNS

    2. Create pull secret and BMC Secret CRs for the managed cluster. The pull secret must contain all the credentials necessary for installing OKD and all required Operators. See “Creating the managed bare-metal host secrets” for more information.

      The secrets are referenced from the SiteConfig custom resource (CR) by name. The namespace must match the SiteConfig namespace.

    3. Create a SiteConfig CR for your cluster in your local clone of the Git repository:

      1. Choose the appropriate example for your CR from the out/argocd/example/siteconfig/ folder. The folder includes example files for single node, three-node, and standard clusters:

        • example-sno.yaml

        • example-3node.yaml

        • example-standard.yaml

      2. Change the cluster and host details in the example file to match the type of cluster you want. For example:

        Example single-node OpenShift cluster SiteConfig CR

        1. apiVersion: ran.openshift.io/v1
        2. kind: SiteConfig
        3. metadata:
        4.   name: "<site_name>"
        5.   namespace: "<site_name>"
        6. spec:
        7.   baseDomain: "example.com"
        8.   pullSecretRef:
        9.     name: "assisted-deployment-pull-secret" (1)
        10.   clusterImageSetNameRef: "openshift-4.12" (2)
        11.   sshPublicKey: "ssh-rsa AAAA..." (3)
        12.   clusters:
        13.   - clusterName: "<site_name>"
        14.     networkType: "OVNKubernetes"
        15.     clusterLabels: (4)
        16.       common: true
        17.       group-du-sno: ""
        18.       sites : "<site_name>"
        19.     clusterNetwork:
        20.       - cidr: 1001:1::/48
        21.         hostPrefix: 64
        22.     machineNetwork:
        23.       - cidr: 1111:2222:3333:4444::/64
        24.     serviceNetwork:
        25.       - 1001:2::/112
        26.     additionalNTPSources:
        27.       - 1111:2222:3333:4444::2
        28.     #crTemplates:
        29.     #  KlusterletAddonConfig: "KlusterletAddonConfigOverride.yaml" (5)
        30.     nodes:
        31.       - hostName: "example-node.example.com" (6)
        32.         role: "master"
        33.         bmcAddress: idrac-virtualmedia://<out_of_band_ip>/<system_id>/ (7)
        34.         bmcCredentialsName:
        35.           name: "bmh-secret" (8)
        36.         bootMACAddress: "AA:BB:CC:DD:EE:11"
        37.         bootMode: "UEFI" (9)
        38.         rootDeviceHints:
        39.           wwn: "0x11111000000asd123"
        40.         cpuset: "0-1,52-53"  (10)
        41.         nodeNetwork: (11)
        42.           interfaces:
        43.             - name: eno1
        44.               macAddress: "AA:BB:CC:DD:EE:11"
        45.           config:
        46.             interfaces:
        47.               - name: eno1
        48.                 type: ethernet
        49.                 state: up
        50.                 macAddress: "AA:BB:CC:DD:EE:11"
        51.                 ipv4:
        52.                   enabled: false
        53.                 ipv6: (12)
        54.                   enabled: true
        55.                   address:
        56.                   - ip: 1111:2222:3333:4444::aaaa:1
        57.                     prefix-length: 64
        58.             dns-resolver:
        59.               config:
        60.                 search:
        61.                 - example.com
        62.                 server:
        63.                 - 1111:2222:3333:4444::2
        64.             routes:
        65.               config:
        66.               - destination: ::/0
        67.                 next-hop-interface: eno1
        68.                 next-hop-address: 1111:2222:3333:4444::1
        69.                 table-id: 254
        1Create the assisted-deployment-pull-secret CR with the same namespace as the SiteConfig CR.
        2clusterImageSetNameRef defines an image set available on the hub cluster. To see the list of supported versions on your hub cluster, run oc get clusterimagesets.
        3Configure the SSH public key used to access the cluster.
        4Cluster labels must correspond to the bindingRules field in the PolicyGenTemplate CRs that you define. For example, policygentemplates/common-ranGen.yaml applies to all clusters with common: true set, policygentemplates/group-du-sno-ranGen.yaml applies to all clusters with group-du-sno: “” set.
        5Optional. The CR specifed under KlusterletAddonConfig is used to override the default KlusterletAddonConfig that is created for the cluster.
        6For single-node deployments, define a single host. For three-node deployments, define three hosts. For standard deployments, define three hosts with role: master and two or more hosts defined with role: worker.
        7BMC address that you use to access the host. Applies to all cluster types. ZTP supports iPXE and virtual media booting by using Redfish or IPMI protocols.
        8Name of the bmh-secret CR that you separately create with the host BMC credentials. When creating the bmh-secret CR, use the same namespace as the SiteConfig CR that provisions the host.
        9Configures the boot mode for the host. The default value is UEFI. Use UEFISecureBoot to enable secure boot on the host.
        10cpuset must match the value set in the cluster PerformanceProfile CR spec.cpu.reserved field for workload partitioning.
        11Specifies the network settings for the node.
        12Configures the IPv6 address for the host. For single-node OpenShift clusters with static IP addresses, the node-specific API and Ingress IPs should be the same.

        For more information about BMC addressing, see the “Additional resources” section.

      3. You can inspect the default set of extra-manifest MachineConfig CRs in out/argocd/extra-manifest. It is automatically applied to the cluster when it is installed.

      4. Optional: To provision additional install-time manifests on the provisioned cluster, create a directory in your Git repository, for example, sno-extra-manifest/, and add your custom manifest CRs to this directory. If your SiteConfig.yaml refers to this directory in the extraManifestPath field, any CRs in this referenced directory are appended to the default set of extra manifests.

    4. Add the SiteConfig CR to the kustomization.yaml file in the generators section, similar to the example shown in out/argocd/example/siteconfig/kustomization.yaml.

    5. Commit the SiteConfig CR and associated kustomization.yaml changes in your Git repository and push the changes.

      The ArgoCD pipeline detects the changes and begins the managed cluster deployment.

Additional resources

Monitoring managed cluster installation progress

The ArgoCD pipeline uses the SiteConfig CR to generate the cluster configuration CRs and syncs it with the hub cluster. You can monitor the progress of the synchronization in the ArgoCD dashboard.

Prerequisites

  • You have installed the OpenShift CLI (oc).

  • You have logged in to the hub cluster as a user with cluster-admin privileges.

Procedure

When the synchronization is complete, the installation generally proceeds as follows:

  1. The Assisted Service Operator installs OKD on the cluster. You can monitor the progress of cluster installation from the RHACM dashboard or from the command line by running the following commands:

    1. Export the cluster name:

      1. $ export CLUSTER=<clusterName>

    2. Query the AgentClusterInstall CR for the managed cluster:

      1. $ oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.conditions[?(@.type=="Completed")]}' | jq

    3. Get the installation events for the cluster:

      1. $ curl -sk $(oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.debugInfo.eventsURL}')  | jq '.[-2,-1]'

Troubleshooting GitOps ZTP by validating the installation CRs

The ArgoCD pipeline uses the SiteConfig and PolicyGenTemplate custom resources (CRs) to generate the cluster configuration CRs and Red Hat Advanced Cluster Management (RHACM) policies. Use the following steps to troubleshoot issues that might occur during this process.

Prerequisites

  • You have installed the OpenShift CLI (oc).

  • You have logged in to the hub cluster as a user with cluster-admin privileges.

Procedure

  1. Check that the installation CRs were created by using the following command:

    1. $ oc get AgentClusterInstall -n <cluster_name>

    If no object is returned, use the following steps to troubleshoot the ArgoCD pipeline flow from SiteConfig files to the installation CRs.

  2. Verify that the ManagedCluster CR was generated using the SiteConfig CR on the hub cluster:

    1. $ oc get managedcluster

  3. If the ManagedCluster is missing, check if the clusters application failed to synchronize the files from the Git repository to the hub cluster:

    1. $ oc describe -n openshift-gitops application clusters

    1. Check for the Status.Conditions field to view the error logs for the managed cluster. For example, setting an invalid value for extraManifestPath: in the SiteConfig CR raises the following error:

      1. Status:
      2.   Conditions:
      3.     Last Transition Time:  2021-11-26T17:21:39Z
      4.     Message:               rpc error: code = Unknown desc = `kustomize build /tmp/https___git.com/ran-sites/siteconfigs/ --enable-alpha-plugins` failed exit status 1: 2021/11/26 17:21:40 Error could not create extra-manifest ranSite1.extra-manifest3 stat extra-manifest3: no such file or directory 2021/11/26 17:21:40 Error: could not build the entire SiteConfig defined by /tmp/kust-plugin-config-913473579: stat extra-manifest3: no such file or directory Error: failure in plugin configured via /tmp/kust-plugin-config-913473579; exit status 1: exit status 1
      5.     Type:  ComparisonError

    2. Check the Status.Sync field. If there are log errors, the Status.Sync field could indicate an Unknown error:

      1. Status:
      2.   Sync:
      3.     Compared To:
      4.       Destination:
      5.         Namespace:  clusters-sub
      6.         Server:     https://kubernetes.default.svc
      7.       Source:
      8.         Path:             sites-config
      9.         Repo URL:         https://git.com/ran-sites/siteconfigs/.git
      10.         Target Revision:  master
      11.     Status:               Unknown

Removing a managed cluster site from the ZTP pipeline

You can remove a managed site and the associated installation and configuration policy CRs from the ZTP pipeline.

Prerequisites

  • You have installed the OpenShift CLI (oc).

  • You have logged in to the hub cluster as a user with cluster-admin privileges.

Precedure

  1. Remove a site and the associated CRs by removing the associated SiteConfig and PolicyGenTemplate files from the kustomization.yaml file.

    When you run the ZTP pipeline again, the generated CRs are removed.

  2. Optional: If you want to permanently remove a site, you should also remove the SiteConfig and site-specific PolicyGenTemplate files from the Git repository.

  3. Optional: If you want to remove a site temporarily, for example when redeploying a site, you can leave the SiteConfig and site-specific PolicyGenTemplate CRs in the Git repository.

After removing the SiteConfig file from the Git repository, if the corresponding clusters get stuck in the detach process, check Red Hat Advanced Cluster Management (RHACM) on the hub cluster for information about cleaning up the detached cluster.

Additional resources

Removing obsolete content from the ZTP pipeline

If a change to the PolicyGenTemplate configuration results in obsolete policies, for example, if you rename policies, use the following procedure to remove the obsolete policies.

Prerequisites

  • You have installed the OpenShift CLI (oc).

  • You have logged in to the hub cluster as a user with cluster-admin privileges.

Procedure

  1. Remove the affected PolicyGenTemplate files from the Git repository, commit and push to the remote repository.

  2. Wait for the changes to synchronize through the application and the affected policies to be removed from the hub cluster.

  3. Add the updated PolicyGenTemplate files back to the Git repository, and then commit and push to the remote repository.

    Removing zero touch provisioning (ZTP) policies from the Git repository, and as a result also removing them from the hub cluster, does not affect the configuration of the managed cluster. The policy and CRs managed by that policy remains in place on the managed cluster.

  4. Optional: As an alternative, after making changes to PolicyGenTemplate CRs that result in obsolete policies, you can remove these policies from the hub cluster manually. You can delete policies from the RHACM console using the Governance tab or by running the following command:

    1. $ oc delete policy -n <namespace> <policy_name>

Tearing down the ZTP pipeline

You can remove the ArgoCD pipeline and all generated ZTP artifacts.

Prerequisites

  • You have installed the OpenShift CLI (oc).

  • You have logged in to the hub cluster as a user with cluster-admin privileges.

Procedure

  1. Detach all clusters from Red Hat Advanced Cluster Management (RHACM) on the hub cluster.

  2. Delete the kustomization.yaml file in the deployment directory using the following command:

    1. $ oc delete -k out/argocd/deployment

  3. Commit and push your changes to the site repository.