Scaling the Cluster Monitoring Operator

OKD exposes metrics that the Cluster Monitoring Operator collects and stores in the Prometheus-based monitoring stack. As an administrator, you can view dashboards for system resources, containers, and components metrics in the OKD web console by navigating to Observe → Dashboards.

Prometheus database storage requirements

Red Hat performed various tests for different scale sizes.

Approximately 20 percent of the expected size was added as overhead to ensure that the storage requirements do not exceed the calculated value.

The above calculation is for the default OKD Cluster Monitoring Operator.

Recommendations for OKD

• Use at least three infrastructure (infra) nodes.

• Use at least three openshift-container-storage nodes with non-volatile memory express (NVMe) drives.

Configuring cluster monitoring

You can increase the storage capacity for the Prometheus component in the cluster monitoring stack.

Procedure

To increase the storage capacity for Prometheus:

1. Create a YAML configuration file, cluster-monitoring-config.yaml. For example:

   apiVersion: v1
   kind: ConfigMap
   data:
     config.yaml: |
       prometheusK8s:
         retention: {{PROMETHEUS_RETENTION_PERIOD}} (1)
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} (2)
             resources:
               requests:
                 storage: {{PROMETHEUS_STORAGE_SIZE}} (3)
       alertmanagerMain:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} (2)
             resources:
               requests:
                 storage: {{ALERTMANAGER_STORAGE_SIZE}} (4)
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring

   1	A typical value is PROMETHEUS_RETENTION_PERIOD=15d. Units are measured in time using one of these suffixes: s, m, h, d.
   2	The storage class for your cluster.
   3	A typical value is PROMETHEUS_STORAGE_SIZE=2000Gi. Storage values can be a plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.
   4	A typical value is ALERTMANAGER_STORAGE_SIZE=20Gi. Storage values can be a plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

2. Add values for the retention period, storage class, and storage sizes.

3. Save the file.

4. Apply the changes by running:

   $ oc create -f cluster-monitoring-config.yaml