Scaling the Cluster Monitoring Operator

OKD exposes metrics that the Cluster Monitoring Operator collects and stores in the Prometheus-based monitoring stack. As an administrator, you can view system resources, containers and components metrics in one dashboard interface, Grafana.

If you are running cluster monitoring with an attached PVC for Prometheus, you might experience OOM kills during cluster upgrade. When persistent storage is in use for Prometheus, Prometheus memory usage doubles during cluster upgrade and for several hours after upgrade is complete. To avoid the OOM kill issue, allow worker nodes with double the size of memory that was available prior to the upgrade. For example, if you are running monitoring on the minimum recommended nodes, which is 2 cores with 8 GB of RAM, increase memory to 16 GB. For more information, see BZ#1925061.

Prometheus database storage requirements

Red Hat performed various tests for different scale sizes.

The Prometheus storage requirements below are not prescriptive. Higher resource consumption might be observed in your cluster depending on workload activity and resource use.

Table 1. Prometheus Database storage requirements based on number of nodes/pods in the cluster
Number of NodesNumber of podsPrometheus storage growth per dayPrometheus storage growth per 15 daysRAM Space (per scale size)Network (per tsdb chunk)

50

1800

6.3 GB

94 GB

6 GB

16 MB

100

3600

13 GB

195 GB

10 GB

26 MB

150

5400

19 GB

283 GB

12 GB

36 MB

200

7200

25 GB

375 GB

14 GB

46 MB

Approximately 20 percent of the expected size was added as overhead to ensure that the storage requirements do not exceed the calculated value.

The above calculation is for the default OKD Cluster Monitoring Operator.

CPU utilization has minor impact. The ratio is approximately 1 core out of 40 per 50 nodes and 1800 pods.

Recommendations for OKD

  • Use at least three infrastructure (infra) nodes.

  • Use at least three openshift-container-storage nodes with non-volatile memory express (NVMe) drives.

Configuring cluster monitoring

Procedure

To increase the storage capacity for Prometheus:

  1. Create a YAML configuration file, cluster-monitoring-config.yml. For example:

    1. apiVersion: v1
    2. kind: ConfigMap
    3. data:
    4.   config.yaml: |
    5.     prometheusOperator:
    6.       baseImage: quay.io/coreos/prometheus-operator
    7.       prometheusConfigReloaderBaseImage: quay.io/coreos/prometheus-config-reloader
    8.       configReloaderBaseImage: quay.io/coreos/configmap-reload
    9.       nodeSelector:
    10.         node-role.kubernetes.io/infra: ""
    11.     prometheusK8s:
    12.       retention: {{PROMETHEUS_RETENTION_PERIOD}} (1)
    13.       baseImage: openshift/prometheus
    14.       nodeSelector:
    15.         node-role.kubernetes.io/infra: ""
    16.       volumeClaimTemplate:
    17.         spec:
    18.           storageClassName: gp2
    19.           resources:
    20.             requests:
    21.               storage: {{PROMETHEUS_STORAGE_SIZE}} (2)
    22.     alertmanagerMain:
    23.       baseImage: openshift/prometheus-alertmanager
    24.       nodeSelector:
    25.         node-role.kubernetes.io/infra: ""
    26.       volumeClaimTemplate:
    27.         spec:
    28.           storageClassName: gp2
    29.           resources:
    30.             requests:
    31.               storage: {{ALERTMANAGER_STORAGE_SIZE}} (3)
    32.     nodeExporter:
    33.       baseImage: openshift/prometheus-node-exporter
    34.     kubeRbacProxy:
    35.       baseImage: quay.io/coreos/kube-rbac-proxy
    36.     kubeStateMetrics:
    37.       baseImage: quay.io/coreos/kube-state-metrics
    38.       nodeSelector:
    39.         node-role.kubernetes.io/infra: ""
    40.     grafana:
    41.       baseImage: grafana/grafana
    42.       nodeSelector:
    43.         node-role.kubernetes.io/infra: ""
    44.     auth:
    45.       baseImage: openshift/oauth-proxy
    46.     k8sPrometheusAdapter:
    47.       nodeSelector:
    48.         node-role.kubernetes.io/infra: ""
    49. metadata:
    50.   name: cluster-monitoring-config
    51. namespace: openshift-monitoring
    1A typical value is PROMETHEUS_RETENTION_PERIOD=15d. Units are measured in time using one of these suffixes: s, m, h, d.
    2A typical value is PROMETHEUS_STORAGE_SIZE=2000Gi. Storage values can be a plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.
    3A typical value is ALERTMANAGER_STORAGE_SIZE=20Gi. Storage values can be a plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

  2. Set the values like the retention period and storage sizes.

  3. Apply the changes by running:

    1. $ oc create -f cluster-monitoring-config.yml