Recommended infrastructure practices

This topic provides recommended performance and scalability practices for infrastructure in OKD.

Infrastructure node sizing

Infrastructure nodes are nodes that are labeled to run pieces of the OKD environment. The infrastructure node resource requirements depend on the cluster age, nodes, and objects in the cluster, as these factors can lead to an increase in the number of metrics or time series in Prometheus. The following infrastructure node size recommendations are based on the results observed in cluster-density testing detailed in the Control plane node sizing section, where the monitoring stack and the default ingress-controller were moved to these nodes.

In general, three infrastructure nodes are recommended per cluster.

Scaling the Cluster Monitoring Operator

OKD exposes metrics that the Cluster Monitoring Operator collects and stores in the Prometheus-based monitoring stack. As an administrator, you can view dashboards for system resources, containers, and components metrics in the OKD web console by navigating to Observe → Dashboards.

Prometheus database storage requirements

Red Hat performed various tests for different scale sizes.

Approximately 20 percent of the expected size was added as overhead to ensure that the storage requirements do not exceed the calculated value.

The above calculation is for the default OKD Cluster Monitoring Operator.

Recommendations for OKD

• Use at least three infrastructure (infra) nodes.

• Use at least three openshift-container-storage nodes with non-volatile memory express (NVMe) drives.

Configuring cluster monitoring

You can increase the storage capacity for the Prometheus component in the cluster monitoring stack.

Procedure

To increase the storage capacity for Prometheus:

1. Create a YAML configuration file, cluster-monitoring-config.yaml. For example:

   apiVersion: v1
   kind: ConfigMap
   data:
     config.yaml: |
       prometheusK8s:
         retention: {{PROMETHEUS_RETENTION_PERIOD}} (1)
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} (2)
             resources:
               requests:
                 storage: {{PROMETHEUS_STORAGE_SIZE}} (3)
       alertmanagerMain:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} (2)
             resources:
               requests:
                 storage: {{ALERTMANAGER_STORAGE_SIZE}} (4)
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring

   1	A typical value is PROMETHEUS_RETENTION_PERIOD=15d. Units are measured in time using one of these suffixes: s, m, h, d.
   2	The storage class for your cluster.
   3	A typical value is PROMETHEUS_STORAGE_SIZE=2000Gi. Storage values can be a plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.
   4	A typical value is ALERTMANAGER_STORAGE_SIZE=20Gi. Storage values can be a plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

2. Add values for the retention period, storage class, and storage sizes.

3. Save the file.

4. Apply the changes by running:

   $ oc create -f cluster-monitoring-config.yaml

Additional resources

• Infrastructure Nodes in OpenShift 4

• OKD cluster maximums

• Creating infrastructure machine sets