Monitoring

The following sections contain tips to troubleshoot Harvester Monitoring.

Monitoring is unusable

When the Harvester Dashboard is not showing any monitoring metrics, it can be caused by the following reasons.

Monitoring is unusable due to Pod being stuck in Terminating status

Harvester Monitoring pods are deployed randomly on the cluster Nodes. When the Node hosting the pods accidentally goes down, the related pods may become stuck in the Terminating status rendering the Monitoring unusable from the WebUI.

  1. $ kubectl get pods -n cattle-monitoring-system
  3. NAMESPACE                   NAME                                                     READY   STATUS        RESTARTS   AGE
  4. cattle-monitoring-system    prometheus-rancher-monitoring-prometheus-0               3/3     Terminating   0          3d23h
  5. cattle-monitoring-system    rancher-monitoring-admission-create-fwjn9                0/1     Terminating   0          137m
  6. cattle-monitoring-system    rancher-monitoring-crd-create-9wtzf                      0/1     Terminating   0          137m
  7. cattle-monitoring-system    rancher-monitoring-grafana-d9c56d79b-ph4nz               3/3     Terminating   0          3d23h
  8. cattle-monitoring-system    rancher-monitoring-grafana-d9c56d79b-t24sz               0/3     Init:0/2      0          132m
  10. cattle-monitoring-system    rancher-monitoring-kube-state-metrics-5bc8bb48bd-nbd92   1/1     Running       4          4d1h
  11. ...

Monitoring can be recovered using CLI commands to force delete the related pods. The cluster will redeploy new pods to replace them.

  1. # Delete each none-running Pod in namespace cattle-monitoring-system.
  3. $ kubectl delete pod --force -n cattle-monitoring-system prometheus-rancher-monitoring-prometheus-0
  5.  pod "prometheus-rancher-monitoring-prometheus-0" force deleted
  8. $ kubectl delete pod --force -n cattle-monitoring-system rancher-monitoring-admission-create-fwjn9
  10. $ kubectl delete pod --force -n cattle-monitoring-system rancher-monitoring-crd-create-9wtzf
  12. $ kubectl delete pod --force -n cattle-monitoring-system rancher-monitoring-grafana-d9c56d79b-ph4nz 
  14. $ kubectl delete pod --force -n cattle-monitoring-system rancher-monitoring-grafana-d9c56d79b-t24sz

Wait for a few minutes so that the new pods are created and readied for the Monitoring dashboard to be usable again.

  1. $ kubectl get pods -n cattle-monitoring-system 
  3. NAME                                                     READY   STATUS     RESTARTS   AGE
  4. prometheus-rancher-monitoring-prometheus-0               0/3     Init:0/1   0          98s
  5. rancher-monitoring-grafana-d9c56d79b-cp86w               0/3     Init:0/2   0          27s
  6. ...
  8. $ kubectl get pods -n cattle-monitoring-system 
  10. NAME                                                     READY   STATUS    RESTARTS   AGE
  11. prometheus-rancher-monitoring-prometheus-0               3/3     Running   0          7m57s
  12. rancher-monitoring-grafana-d9c56d79b-cp86w               3/3     Running   0          6m46s
  13. ...

Expand PV/Volume Size

Harvester integrates Longhorn as the default storage provider.

Harvester Monitoring uses Persistent Volume (PV) to store running data. When a cluster has been running for a certain time, the Persistent Volume may need to expand its size.

Based on the Longhorn Volume expansion guide, Harvester illustrates how to expand the volume size.

View Volume

From Embedded Longhorn WebUI

Access the embedded Longhorn WebUI according to this document.

The Longhorn dashboard default view.

Monitoring - 图1

Click Volume to list all existing volumes.

Monitoring - 图2

From CLI

You can also use kubectl to get all Volumes.

  1. # kubectl get pvc -A
  2. NAMESPACE                  NAME                                                                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
  3. cattle-monitoring-system   alertmanager-rancher-monitoring-alertmanager-db-alertmanager-rancher-monitoring-alertmanager-0   Bound    pvc-1b2fbbe9-14b1-4a65-941a-7d5645a89977   5Gi        RWO            harvester-longhorn   43h
  4. cattle-monitoring-system   prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0           Bound    pvc-7c6dcb61-51a9-4a38-b4c5-acaa11788978   50Gi       RWO            harvester-longhorn   43h
  5. cattle-monitoring-system   rancher-monitoring-grafana                                                                       Bound    pvc-b2b2c07c-f7cd-4965-90e6-ac3319597bf7   2Gi        RWO            harvester-longhorn   43h
  7. # kubectl get volume -A
  8. NAMESPACE         NAME                                       STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE     AGE
  9. longhorn-system   pvc-1b2fbbe9-14b1-4a65-941a-7d5645a89977   attached   degraded                 5368709120    harv31   43h
  10. longhorn-system   pvc-7c6dcb61-51a9-4a38-b4c5-acaa11788978   attached   degraded                 53687091200   harv31   43h
  11. longhorn-system   pvc-b2b2c07c-f7cd-4965-90e6-ac3319597bf7   attached   degraded                 2147483648    harv31   43h

Scale Down a Deployment

To detach the Volume, you need to scale down the deployment that uses the Volume.

The example below is against the PVC claimed by rancher-monitoring-grafana.

Find the deployment in the namespace cattle-monitoring-system.

  1. # kubectl get deployment -n cattle-monitoring-system
  2. NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
  3. rancher-monitoring-grafana              1/1     1            1           43h  // target deployment
  4. rancher-monitoring-kube-state-metrics   1/1     1            1           43h
  5. rancher-monitoring-operator             1/1     1            1           43h
  6. rancher-monitoring-prometheus-adapter   1/1     1            1           43h

Scale down the deployment rancher-monitoring-grafana to 0.

  1. # kubectl scale --replicas=0 deployment/rancher-monitoring-grafana -n cattle-monitoring-system

Check the deployment and the volume.

  1. # kubectl get deployment -n cattle-monitoring-system
  2. NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
  3. rancher-monitoring-grafana              0/0     0            0           43h  // scaled down
  4. rancher-monitoring-kube-state-metrics   1/1     1            1           43h
  5. rancher-monitoring-operator             1/1     1            1           43h
  6. rancher-monitoring-prometheus-adapter   1/1     1            1           43h
  8. # kubectl get volume -A
  9. NAMESPACE         NAME                                       STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE     AGE
  10. longhorn-system   pvc-1b2fbbe9-14b1-4a65-941a-7d5645a89977   attached   degraded                 5368709120    harv31   43h
  11. longhorn-system   pvc-7c6dcb61-51a9-4a38-b4c5-acaa11788978   attached   degraded                 53687091200   harv31   43h
  12. longhorn-system   pvc-b2b2c07c-f7cd-4965-90e6-ac3319597bf7   detached   unknown                  2147483648             43h  // volume is detached

Expand Volume

In the Longhorn WebUI, the related volume becomes Detached. Click the icon in the Operation column, and select Expand Volume.

Monitoring - 图3

Input a new size, and Longhorn will expand the volume to this size.

Monitoring - 图4

Scale Up a Deployment

After the Volume is expanded to target size, you need to scale up the aforementioned deployment to its original replicas. For the above example of rancher-monitoring-grafana, the original replicas is 1.

  1. # kubectl scale --replicas=1 deployment/rancher-monitoring-grafana -n cattle-monitoring-system

Check the deployment again.

  1. # kubectl get deployment -n cattle-monitoring-system
  2. NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
  3. rancher-monitoring-grafana              1/1     1            1           43h  // scaled up
  4. rancher-monitoring-kube-state-metrics   1/1     1            1           43h
  5. rancher-monitoring-operator             1/1     1            1           43h
  6. rancher-monitoring-prometheus-adapter   1/1     1            1           43h

The Volume is attached to the new POD.

Monitoring - 图5

To now, the Volume is expanded to the new size and the POD is using it smoothly.

Fail to Enable rancher-monitoring Addon

You may encounter this when you install the Harvester v1.3.0 or higher version cluster with the minimal 250 GB disk per hardware requirements.

Reproduce Steps

  1. Install the Harvester v1.3.0 cluster.

  2. Enable the rancher-monitoring addon, you will observe:

  • The POD prometheus-rancher-monitoring-prometheus-0 in cattle-monitoring-system namespace fails to start due to PVC attached failed.

    1. $ kubectl get pods -n cattle-monitoring-system
    2. NAME                                                     READY   STATUS      RESTARTS   AGE
    3. alertmanager-rancher-monitoring-alertmanager-0           2/2     Running     0          3m22s
    4. helm-install-rancher-monitoring-4b5mx                    0/1     Completed   0          3m41s
    5. prometheus-rancher-monitoring-prometheus-0               0/3     Init:0/1    0          3m21s // stuck in this status
    6. rancher-monitoring-grafana-d6f466988-hgpkb               4/4     Running     0          3m26s
    7. rancher-monitoring-kube-state-metrics-7659b76cc4-66sr7   1/1     Running     0          3m26s
    8. rancher-monitoring-operator-595476bc84-7hdxj             1/1     Running     0          3m25s
    9. rancher-monitoring-prometheus-adapter-55dc9ccd5d-pcrpk   1/1     Running     0          3m26s
    10. rancher-monitoring-prometheus-node-exporter-pbzv4        1/1     Running     0          3m26s
    12. $ kubectl describe pod -n cattle-monitoring-system prometheus-rancher-monitoring-prometheus-0
    13. Name:             prometheus-rancher-monitoring-prometheus-0
    14. Namespace:        cattle-monitoring-system
    15. Priority:         0
    16. Service Account:  rancher-monitoring-prometheus
    17. ...
    18. Events:
    19.   Type     Reason              Age                    From                     Message
    20.   ----     ------              ----                   ----                     -------
    21.   Warning  FailedScheduling    3m48s (x3 over 4m15s)  default-scheduler        0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
    22.   Normal   Scheduled           3m44s                  default-scheduler        Successfully assigned cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0 to harv41
    23.   Warning  FailedMount         101s                   kubelet                  Unable to attach or mount volumes: unmounted volumes=[prometheus-rancher-monitoring-prometheus-db], unattached volumes=[prometheus-rancher-monitoring-prometheus-db], failed to process volumes=[]: timed out waiting for the condition
    24.   Warning  FailedAttachVolume  90s (x9 over 3m42s)    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0" : rpc error: code = Aborted desc = volume pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0 is not ready for workloads
    26. $ kubectl get pvc -A
    27. NAMESPACE                  NAME                                                                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
    28. cattle-monitoring-system   prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0           Bound    pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0   50Gi       RWO            harvester-longhorn     7m12s
    30. $ kubectl get volume -A
    31. NAMESPACE         NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE     AGE
    32. longhorn-system   pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0   v1            detached   unknown                  53687091200            6m55s

  • The Longhorn manager is unable to schedule the replica.

    1. $ kubectl logs -n longhorn-system longhorn-manager-bf65b | grep "pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0"
    3. time="2024-02-19T10:12:56Z" level=error msg="There's no available disk for replica pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0-r-dcb129fd, size 53687091200" func="schedule
    4. r.(*ReplicaScheduler).ScheduleReplica" file="replica_scheduler.go:95"
    5. time="2024-02-19T10:12:56Z" level=warning msg="Failed to schedule replica" func="controller.(*VolumeController).reconcileVolumeCondition" file="volume_controller.go:169
    6. 4" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harv41 owner=harv41 replica=pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0-r-dcb129fd sta
    7. te= volume=pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0
    8. ...

Workaround

  1. Disable the rancher-monitoring addon if you have alreay enabled it.

    All pods in cattle-monitoring-system are deleted but the PVCs are retained. For more information, see [Addons].

    1. $ kubectl get pods -n cattle-monitoring-system
    2. No resources found in cattle-monitoring-system namespace.
    4. $ kubectl get pvc -n cattle-monitoring-system
    5. NAME                                                                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
    6. alertmanager-rancher-monitoring-alertmanager-db-alertmanager-rancher-monitoring-alertmanager-0   Bound    pvc-cea6316e-f74f-4771-870b-49edb5442819   5Gi        RWO            harvester-longhorn   14m
    7. prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0           Bound    pvc-bbe8760d-926c-484a-851c-b8ec29ae05c0   50Gi       RWO            harvester-longhorn   14m

  2. Delete the PVC named prometheus, but retain the PVC named alertmanager.

    1. $ kubectl delete pvc -n cattle-monitoring-system prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0
    2. persistentvolumeclaim "prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0" deleted
    4. $ kubectl get pvc -n cattle-monitoring-system
    5. NAME                                                                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
    6. alertmanager-rancher-monitoring-alertmanager-db-alertmanager-rancher-monitoring-alertmanager-0   Bound    pvc-cea6316e-f74f-4771-870b-49edb5442819   5Gi        RWO            harvester-longhorn   16m

  3. On the Addons screen of the Harvester UI, select ⋮ (menu icon) and then select Edit YAML.

    Monitoring - 图6

  4. As indicated below, change the two occurrences of the number 50 to 30 under prometheusSpec, and then save. The prometheus feature will use a 30GiB disk to store data.

    Monitoring - 图7

    Alternatively, you can use kubectl to edit the object.

    kubectl edit addons.harvesterhci.io -n cattle-monitoring-system rancher-monitoring

    1.         retentionSize: 50GiB // Change 50 to 30
    2.         storageSpec:
    3.           volumeClaimTemplate:
    4.             spec:
    5.               accessModes:
    6.                 - ReadWriteOnce
    7.               resources:
    8.                 requests:
    9.                   storage: 50Gi // Change 50 to 30
    10.               storageClassName: harvester-longhorn

  5. Enable the rancher-monitoring addon and wait for a few minutes..

  6. All pods are successfully deployed, and the rancher-monitoring feature is available.

    1. $ kubectl get pods -n cattle-monitoring-system
    2. NAME                                                     READY   STATUS      RESTARTS   AGE
    3. alertmanager-rancher-monitoring-alertmanager-0           2/2     Running     0          3m52s
    4. helm-install-rancher-monitoring-s55tq                    0/1     Completed   0          4m17s
    5. prometheus-rancher-monitoring-prometheus-0               3/3     Running     0          3m51s
    6. rancher-monitoring-grafana-d6f466988-hkv6f               4/4     Running     0          3m55s
    7. rancher-monitoring-kube-state-metrics-7659b76cc4-ght8x   1/1     Running     0          3m55s
    8. rancher-monitoring-operator-595476bc84-r96bp             1/1     Running     0          3m55s
    9. rancher-monitoring-prometheus-adapter-55dc9ccd5d-vtssc   1/1     Running     0          3m55s
    10. rancher-monitoring-prometheus-node-exporter-lgb88        1/1     Running     0          3m55s

rancher-monitoring-crd ManagedChart State is Modified

Issue Description

In certain situations, the state of the rancher-monitoring-crd ManagedChart object changes to Modified (with the message ...rancher-monitoring-crd-manager missing...).

Example:

  1. $ kubectl get managedchart rancher-monitoring-crd -n fleet-local -o yaml
  2. apiVersion: management.cattle.io/v3
  3. kind: ManagedChart
  4. ...
  5. spec:
  6.   chart: rancher-monitoring-crd
  7.   defaultNamespace: cattle-monitoring-system
  8.   paused: false
  9.   releaseName: rancher-monitoring-crd
  10.   repoName: harvester-charts
  11.   targets:
  12.   - clusterName: local
  13.     clusterSelector:
  14.       matchExpressions:
  15.       - key: provisioning.cattle.io/unmanaged-system-agent
  16.         operator: DoesNotExist
  17.   version: 102.0.0+up40.1.2
  18. ...
  19. status:
  20.   conditions:
  21.   - lastUpdateTime: "2024-02-22T14:03:11Z"
  22.     message: Modified(1) [Cluster fleet-local/local]; clusterrole.rbac.authorization.k8s.io
  23.       rancher-monitoring-crd-manager missing; clusterrolebinding.rbac.authorization.k8s.io
  24.       rancher-monitoring-crd-manager missing; configmap.v1 cattle-monitoring-system/rancher-monitoring-crd-manifest
  25.       missing; serviceaccount.v1 cattle-monitoring-system/rancher-monitoring-crd-manager
  26.       missing
  27.     status: "False"
  28.     type: Ready
  29.   - lastUpdateTime: "2024-02-22T14:03:11Z"
  30.     status: "True"
  31.     type: Processed
  32.   - lastUpdateTime: "2024-04-02T07:45:26Z"
  33.     status: "True"
  34.     type: Defined
  35.   display:
  36.     readyClusters: 0/1
  37.     state: Modified
  38. ...

The ManagedChart object has a downstream object named Bundle, which has similar information.

Example:

  1. $ kubectl get bundles -A
  2. NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
  3. fleet-local   fleet-agent-local                             1/1
  4. fleet-local   local-managed-system-agent                    1/1
  5. fleet-local   mcc-harvester                                 1/1
  6. fleet-local   mcc-harvester-crd                             1/1
  7. fleet-local   mcc-local-managed-system-upgrade-controller   1/1
  8. fleet-local   mcc-rancher-logging-crd                       1/1
  9. fleet-local   mcc-rancher-monitoring-crd                    0/1                       Modified(1) [Cluster fleet-local/local]; clusterrole.rbac.authorization.k8s.io rancher-monitoring-crd-manager missing; clusterrolebinding.rbac.authorization.k8s.io rancher-monitoring-crd-manager missing; configmap.v1 cattle-monitoring-system/rancher-monitoring-crd-manifest missing; serviceaccount.v1 cattle-monitoring-system/rancher-monitoring-crd-manager missing

When the issue exists and you start an upgrade, Harvester may return the following error message: admission webhook "validator.harvesterhci.io" denied the request: managed chart rancher-monitoring-crd is not ready, please wait for it to be ready.

Also, when you search for the objects marked as missing, you will find that they exist in the cluster.

Example:

  1. $ kubectl get clusterrole rancher-monitoring-crd-manager
  2. apiVersion: rbac.authorization.k8s.io/v1
  3. kind: ClusterRole
  4. metadata:
  5.   annotations:
  6.     meta.helm.sh/release-name: rancher-monitoring-crd
  7.     meta.helm.sh/release-namespace: cattle-monitoring-system
  8.   creationTimestamp: "2023-01-09T11:04:33Z"
  9.   labels:
  10.     app: rancher-monitoring-crd-manager
  11.     app.kubernetes.io/managed-by: Helm
  12.   name: rancher-monitoring-crd-manager
  13.   ...
  14. rules:
  15. - apiGroups:
  16.   - apiextensions.k8s.io
  17.   resources:
  18.   - customresourcedefinitions
  19.   verbs:
  20.   - create
  21.   - get
  22.   - patch
  23.   - delete
  25. $ kubectl get clusterrolebinding rancher-monitoring-crd-manager
  26. apiVersion: rbac.authorization.k8s.io/v1
  27. kind: ClusterRoleBinding
  28. metadata:
  29.   annotations:
  30.     meta.helm.sh/release-name: rancher-monitoring-crd
  31.     meta.helm.sh/release-namespace: cattle-monitoring-system
  32.   creationTimestamp: "2023-01-09T11:04:33Z"
  33.   labels:
  34.     app: rancher-monitoring-crd-manager
  35.     app.kubernetes.io/managed-by: Helm
  36.   name: rancher-monitoring-crd-manager
  37.   ...
  38. roleRef:
  39.   apiGroup: rbac.authorization.k8s.io
  40.   kind: ClusterRole
  41.   name: rancher-monitoring-crd-manager
  42. subjects:
  43. - kind: ServiceAccount
  44.   name: rancher-monitoring-crd-manager
  45.   namespace: cattle-monitoring-system
  47. $ kubectl get configmap -n cattle-monitoring-system rancher-monitoring-crd-manifest
  48. apiVersion: v1
  49. data:
  50.   crd-manifest.tgz.b64: ...
  51. kind: ConfigMap
  52. metadata:
  53.   annotations:
  54.     meta.helm.sh/release-name: rancher-monitoring-crd
  55.     meta.helm.sh/release-namespace: cattle-monitoring-system
  56.   creationTimestamp: "2023-01-09T11:04:33Z"
  57.   labels:
  58.     app.kubernetes.io/managed-by: Helm
  59.   name: rancher-monitoring-crd-manifest
  60.   namespace: cattle-monitoring-system
  61.   ...
  63. $ kubectl get ServiceAccount -n cattle-monitoring-system rancher-monitoring-crd-manager
  64. apiVersion: v1
  65. kind: ServiceAccount
  66. metadata:
  67.   annotations:
  68.     meta.helm.sh/release-name: rancher-monitoring-crd
  69.     meta.helm.sh/release-namespace: cattle-monitoring-system
  70.   creationTimestamp: "2023-01-09T11:04:33Z"
  71.   labels:
  72.     app: rancher-monitoring-crd-manager
  73.     app.kubernetes.io/managed-by: Helm
  74.   name: rancher-monitoring-crd-manager
  75.   namespace: cattle-monitoring-system
  76.   ...

Root Cause

The objects that are marked as missing do not have the related annotations and labels required by the ManagedChart object.

Example:

  1. One of the manually recreated object:
  3. apiVersion: rbac.authorization.k8s.io/v1
  4. kind: ClusterRole
  5. metadata:
  6.   annotations:
  7.     meta.helm.sh/release-name: rancher-monitoring-crd
  8.     meta.helm.sh/release-namespace: cattle-monitoring-system
  9.     objectset.rio.cattle.io/id: default-mcc-rancher-monitoring-crd-cattle-fleet-local-system   # This required item is not in the above object.
  10.   creationTimestamp: "2024-04-03T10:23:55Z"
  11.   labels:
  12.     app: rancher-monitoring-crd-manager
  13.     app.kubernetes.io/managed-by: Helm
  14.     objectset.rio.cattle.io/hash: 2da503261617e9ea2da822d2da7cdcfccad847a9    # This required item is not in the above object.
  15.   name: rancher-monitoring-crd-manager
  16. ...
  17. rules:
  18. - apiGroups:
  19.   - apiextensions.k8s.io
  20.   resources:
  21.   - customresourcedefinitions
  22.   verbs:
  23.   - create
  24.   - get
  25.   - patch
  26.   - delete
  27.   - update

Workaround

  1. Patch the ClusterRole object rancher-monitoring-crd-manager to add the update operation.

    1. $ cat > patchrules.yaml << EOF
    2. rules:
    3. - apiGroups:
    4.   - apiextensions.k8s.io
    5.   resources:
    6.   - customresourcedefinitions
    7.   verbs:
    8.   - create
    9.   - get
    10.   - patch
    11.   - delete
    12.   - update
    13. EOF
    15. $ kubectl patch ClusterRole rancher-monitoring-crd-manager --patch-file ./patchrules.yaml --type merge
    17. $ rm ./patchrules.yaml

  2. Patch the marked as missing objects to add the required annotations and labels.

    1. $ cat > patchhash.yaml << EOF
    2. metadata:
    3.   annotations:
    4.     objectset.rio.cattle.io/id: default-mcc-rancher-monitoring-crd-cattle-fleet-local-system
    5.   labels:
    6.     objectset.rio.cattle.io/hash: 2da503261617e9ea2da822d2da7cdcfccad847a9
    7. EOF
    9. $ kubectl patch ClusterRole rancher-monitoring-crd-manager --patch-file ./patchhash.yaml --type merge
    11. $ kubectl patch ClusterRoleBinding rancher-monitoring-crd-manager --patch-file ./patchhash.yaml --type merge
    13. $ kubectl patch ServiceAccount rancher-monitoring-crd-manager -n cattle-monitoring-system --patch-file ./patchhash.yaml --type merge
    15. $ kubectl patch ConfigMap rancher-monitoring-crd-manifest -n cattle-monitoring-system --patch-file ./patchhash.yaml --type merge
    17. $ rm ./patchhash.yaml

  3. Check the rancher-monitoring-crd ManagedChart object.

    After a few seconds, the status of the rancher-monitoring-crd ManagedChart object changes to Ready.

    1. $ kubectl get managedchart -n fleet-local rancher-monitoring-crd -oyaml
    2. apiVersion: management.cattle.io/v3
    3. kind: ManagedChart
    4. metadata:
    5. ...
    6.   name: rancher-monitoring-crd
    7.   namespace: fleet-local
    8. ...
    9. status:
    10.   conditions:
    11.   - lastUpdateTime: "2024-04-22T21:41:44Z"
    12.     status: "True"
    13.     type: Ready
    14. ...

    Also, error indicators are no longer displayed for the downstream objects.

    1. $ kubectl bundle -A
    2. NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
    3. fleet-local   fleet-agent-local                             1/1
    4. fleet-local   local-managed-system-agent                    1/1
    5. fleet-local   mcc-harvester                                 1/1
    6. fleet-local   mcc-harvester-crd                             1/1
    7. fleet-local   mcc-local-managed-system-upgrade-controller   1/1
    8. fleet-local   mcc-rancher-logging-crd                       1/1
    9. fleet-local   mcc-rancher-monitoring-crd                    1/1

  4. (Optional) Retry the upgrade (if previously unsuccessful because of this issue).

https://github.com/harvester/harvester/issues/5505