Postinstallation storage configuration

After installing OKD, you can further expand and customize your cluster to your requirements, including storage configuration.

Dynamic provisioning

About dynamic provisioning

The StorageClass resource object describes and classifies storage that can be requested, as well as provides a means for passing parameters for dynamically provisioned storage on demand. StorageClass objects can also serve as a management mechanism for controlling different levels of storage and access to the storage. Cluster Administrators (cluster-admin) or Storage Administrators (storage-admin) define and create the StorageClass objects that users can request without needing any detailed knowledge about the underlying storage volume sources.

The OKD persistent volume framework enables this functionality and allows administrators to provision a cluster with persistent storage. The framework also gives users a way to request those resources without having any knowledge of the underlying infrastructure.

Many storage types are available for use as persistent volumes in OKD. While all of them can be statically provisioned by an administrator, some types of storage are created dynamically using the built-in provider and plugin APIs.

Available dynamic provisioning plugins

OKD provides the following provisioner plugins, which have generic implementations for dynamic provisioning that use the cluster’s configured provider’s API to create new storage resources:

Defining a storage class

StorageClass objects are currently a globally scoped object and must be created by cluster-admin or storage-admin users.

The following sections describe the basic definition for a StorageClass object and specific examples for each of the supported plugin types.

Basic StorageClass object definition

The following resource shows the parameters and default values that you use to configure a storage class. This example uses the AWS ElasticBlockStore (EBS) object definition.

Sample StorageClass definition

kind: StorageClass (1)
apiVersion: storage.k8s.io/v1 (2)
metadata:
  name: <storage-class-name> (3)
  annotations: (4)
    storageclass.kubernetes.io/is-default-class: 'true'
    ...
provisioner: kubernetes.io/aws-ebs (5)
parameters: (6)
  type: gp3
...

Storage class annotations

To set a storage class as the cluster-wide default, add the following annotation to your storage class metadata:

storageclass.kubernetes.io/is-default-class: "true"

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
...

This enables any persistent volume claim (PVC) that does not specify a specific storage class to automatically be provisioned through the default storage class. However, your cluster can have more than one storage class, but only one of them can be the default storage class.

To set a storage class description, add the following annotation to your storage class metadata:

kubernetes.io/description: My Storage Class Description

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubernetes.io/description: My Storage Class Description
...

OpenStack Cinder object definition

cinder-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/cinder
parameters:
  type: fast  (2)
  availability: nova (3)
  fsType: ext4 (4)

AWS Elastic Block Store (EBS) object definition

aws-ebs-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1 (2)
  iopsPerGB: "10" (3)
  encrypted: "true" (4)
  kmsKeyId: keyvalue (5)
  fsType: ext4 (6)

Azure Disk object definition

azure-advanced-disk-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/azure-disk
volumeBindingMode: WaitForFirstConsumer (2)
allowVolumeExpansion: true
parameters:
  kind: Managed (3)
  storageaccounttype: Premium_LRS (4)
reclaimPolicy: Delete

Azure File object definition

The Azure File storage class uses secrets to store the Azure storage account name and the storage account key that are required to create an Azure Files share. These permissions are created as part of the following procedure.

Procedure

1. Define a ClusterRole object that allows access to create and view secrets:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
   #  name: system:azure-cloud-provider
     name: <persistent-volume-binder-role> (1)
   rules:
   - apiGroups: ['']
     resources: ['secrets']
     verbs:     ['get','create']

   1	The name of the cluster role to view and create secrets.

2. Add the cluster role to the service account:

   $ oc adm policy add-cluster-role-to-user <persistent-volume-binder-role> system:serviceaccount:kube-system:persistent-volume-binder

3. Create the Azure File StorageClass object:

   kind: StorageClass
   apiVersion: storage.k8s.io/v1
   metadata:
     name: <azure-file> (1)
   provisioner: kubernetes.io/azure-file
   parameters:
     location: eastus (2)
     skuName: Standard_LRS (3)
     storageAccount: <storage-account> (4)
   reclaimPolicy: Delete
   volumeBindingMode: Immediate

   1	Name of the storage class. The persistent volume claim uses this storage class for provisioning the associated persistent volumes.
   2	Location of the Azure storage account, such as eastus. Default is empty, meaning that a new Azure storage account will be created in the OKD cluster’s location.
   3	SKU tier of the Azure storage account, such as Standard_LRS. Default is empty, meaning that a new Azure storage account will be created with the Standard_LRS SKU.
   4	Name of the Azure storage account. If a storage account is provided, then skuName and location are ignored. If no storage account is provided, then the storage class searches for any storage account that is associated with the resource group for any accounts that match the defined skuName and location.

Considerations when using Azure File

The following file system features are not supported by the default Azure File storage class:

• Symlinks

• Hard links

• Extended attributes

• Sparse files

• Named pipes

Additionally, the owner user identifier (UID) of the Azure File mounted directory is different from the process UID of the container. The uid mount option can be specified in the StorageClass object to define a specific user identifier to use for the mounted directory.

The following StorageClass object demonstrates modifying the user and group identifier, along with enabling symlinks for the mounted directory.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azure-file
mountOptions:
  - uid=1500 (1)
  - gid=1500 (2)
  - mfsymlinks (3)
provisioner: kubernetes.io/azure-file
parameters:
  location: eastus
  skuName: Standard_LRS
reclaimPolicy: Delete
volumeBindingMode: Immediate

GCE PersistentDisk (gcePD) object definition

gce-pd-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard (2)
  replication-type: none
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

VMware vSphere object definition

vsphere-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <storage-class-name> (1)
provisioner: csi.vsphere.vmware.com (2)

Changing the default storage class

Use the following procedure to change the default storage class.

For example, if you have two defined storage classes, gp3 and standard, and you want to change the default storage class from gp3 to standard.

Prerequisites

• Access to the cluster with cluster-admin privileges.

Procedure

To change the default storage class:

1. List the storage classes:

   $ oc get storageclass

   Example output

   NAME                 TYPE
   gp3 (default)        kubernetes.io/aws-ebs (1)
   standard             kubernetes.io/aws-ebs

   1	(default) indicates the default storage class.

2. Make the desired storage class the default.

   For the desired storage class, set the storageclass.kubernetes.io/is-default-class annotation to true by running the following command:

   $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

   You can have multiple default storage classes for a short time. However, you should ensure that only one default storage class exists eventually. With multiple default storage classes present, any persistent volume claim (PVC) requesting the default storage class (pvc.spec.storageClassName=nil) gets the most recently created default storage class, regardless of the default status of that storage class, and the administrator receives an alert in the alerts dashboard that there are multiple default storage classes, MultipleDefaultStorageClasses.

3. Remove the default storage class setting from the old default storage class.

   For the old default storage class, change the value of the storageclass.kubernetes.io/is-default-class annotation to false by running the following command:

   $ oc patch storageclass gp3 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

4. Verify the changes:

   $ oc get storageclass

   Example output

   NAME                 TYPE
   gp3                  kubernetes.io/aws-ebs
   standard (default)   kubernetes.io/aws-ebs

Optimizing storage

Optimizing storage helps to minimize storage use across all resources. By optimizing storage, administrators help ensure that existing storage resources are working in an efficient manner.

Available persistent storage options

Understand your persistent storage options so that you can optimize your OKD environment.

1. NetApp NFS supports dynamic PV provisioning when using the Trident plugin.

Recommended configurable storage technology

The following table summarizes the recommended and configurable storage technologies for the given OKD cluster application.

Specific application storage recommendations

Registry

In a non-scaled/high-availability (HA) OpenShift image registry cluster deployment:

• The storage technology does not have to support RWX access mode.

• The storage technology must ensure read-after-write consistency.

• The preferred storage technology is object storage followed by block storage.

• File storage is not recommended for OpenShift image registry cluster deployment with production workloads.

Scaled registry

In a scaled/HA OpenShift image registry cluster deployment:

• The storage technology must support RWX access mode.

• The storage technology must ensure read-after-write consistency.

• The preferred storage technology is object storage.

• Red Hat OpenShift Data Foundation (ODF), Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS), Microsoft Azure Blob Storage, and OpenStack Swift are supported.

• Object storage should be S3 or Swift compliant.

• For non-cloud platforms, such as vSphere and bare metal installations, the only configurable technology is file storage.

• Block storage is not configurable.

Metrics

In an OKD hosted metrics cluster deployment:

• The preferred storage technology is block storage.

• Object storage is not configurable.

Logging

In an OKD hosted logging cluster deployment:

• Loki Operator:

  • The preferred storage technology is S3 compatible Object storage.

  • Block storage is not configurable.

• OpenShift Elasticsearch Operator:

  • The preferred storage technology is block storage.

  • Object storage is not supported.

Applications

Application use cases vary from application to application, as described in the following examples:

• Storage technologies that support dynamic PV provisioning have low mount time latencies, and are not tied to nodes to support a healthy cluster.

• Application developers are responsible for knowing and understanding the storage requirements for their application, and how it works with the provided storage to ensure that issues do not occur when an application scales or interacts with the storage layer.

Other specific application storage recommendations

• OpenStack Cinder: OpenStack Cinder tends to be adept in ROX access mode use cases.

• Databases: Databases (RDBMSs, NoSQL DBs, etc.) tend to perform best with dedicated block storage.

• The etcd database must have enough storage and adequate performance capacity to enable a large cluster. Information about monitoring and benchmarking tools to establish ample storage and a high-performance environment is described in Recommended etcd practices.

Additional resources

• Recommended etcd practices

Deploy Red Hat OpenShift Data Foundation

Red Hat OpenShift Data Foundation is a provider of agnostic persistent storage for OKD supporting file, block, and object storage, either in-house or in hybrid clouds. As a Red Hat storage solution, Red Hat OpenShift Data Foundation is completely integrated with OKD for deployment, management, and monitoring.

Additional resources

• Configuring the Elasticsearch log store