VMware vSphere CSI Driver Operator

Overview

OKD can provision persistent volumes (PVs) using the Container Storage Interface (CSI) VMware vSphere driver for Virtual Machine Disk (VMDK) volumes.

Familiarity with persistent storage and configuring CSI volumes is recommended when working with a CSI Operator and driver.

To create CSI-provisioned persistent volumes (PVs) that mount to vSphere storage assets, OKD installs the vSphere CSI Driver Operator and the vSphere CSI driver by default in the openshift-cluster-csi-drivers namespace.

  • vSphere CSI Driver Operator: The Operator provides a storage class, called thin-csi, that you can use to create persistent volumes claims (PVCs). The vSphere CSI Driver Operator supports dynamic volume provisioning by allowing storage volumes to be created on-demand, eliminating the need for cluster administrators to pre-provision storage.

  • vSphere CSI driver: The driver enables you to create and mount vSphere PVs. In OKD 4.12, the driver version is 2.6.1.

OKD defaults to using an in-tree (non-CSI) plugin to provision vSphere storage.

In future OKD versions, volumes provisioned using existing in-tree plugins are planned for migration to their equivalent CSI driver. CSI automatic migration should be seamless. Migration does not change how you use all existing API objects, such as persistent volumes, persistent volume claims, and storage classes. For more information about migration, see CSI automatic migration.

After full migration, in-tree plugins will eventually be removed in future versions of OKD.

The vSphere CSI Driver supports dynamic and static provisioning. When using static provisioning in the PV specifications, do not use the key storage.kubernetes.io/csiProvisionerIdentity in csi.volumeAttributes because this key indicates dynamically provisioned PVs.

About CSI

Storage vendors have traditionally provided storage drivers as part of Kubernetes. With the implementation of the Container Storage Interface (CSI), third-party providers can instead deliver storage plugins using a standard interface without ever having to change the core Kubernetes code.

CSI Operators give OKD users storage options, such as volume snapshots, that are not possible with in-tree volume plugins.

vSphere storage policy

The vSphere CSI Operator Driver storage class uses vSphere’s storage policy. OKD automatically creates a storage policy that targets datastore configured in cloud configuration:

  1. kind: StorageClass
  2. apiVersion: storage.k8s.io/v1
  3. metadata:
  4.   name: thin-csi
  5. provisioner: csi.vsphere.vmware.com
  6. parameters:
  7.   StoragePolicyName: "$openshift-storage-policy-xxxx"
  8. volumeBindingMode: WaitForFirstConsumer
  9. allowVolumeExpansion: false
  10. reclaimPolicy: Delete

ReadWriteMany vSphere volume support

If the underlying vSphere environment supports the vSAN file service, then vSphere Container Storage Interface (CSI) Driver Operator installed by OKD supports provisioning of ReadWriteMany (RWX) volumes. If vSAN file service is not configured, then ReadWriteOnce (RWO) is the only access mode available. If you do not have vSAN file service configured, and you request RWX, the volume fails to get created and an error is logged.

For more information about configuring the vSAN file service in your environment, see vSAN File Service.

You can request RWX volumes by making the following persistent volume claim (PVC):

  1. kind: PersistentVolumeClaim
  2. apiVersion: v1
  3. metadata:
  4.   name: myclaim
  5. spec:
  6.   resources:
  7.     requests:
  8.       storage: 1Gi
  9.   accessModes:
  10.      - ReadWriteMany
  11.   storageClassName: thin-csi

Requesting a PVC of the RWX volume type should result in provisioning of persistent volumes (PVs) backed by the vSAN file service.

VMware vSphere CSI Driver Operator requirements

To install the CSI Driver Operator, the following requirements must be met:

  • VMware vSphere version 7.0.2 or later

  • vCenter 7.0.2 or later

  • Virtual machines of hardware version 15 or later

  • No third-party CSI driver already installed in the cluster

If a third-party CSI driver is present in the cluster, OKD does not overwrite it. The presence of a third-party CSI driver prevents OKD from upgrading to OKD 4.13 or later.

To remove a third-party CSI driver, see Removing a third-party vSphere CSI Driver.

Removing a third-party vSphere CSI Operator Driver

OKD 4.10, and later, includes a built-in version of the vSphere Container Storage Interface (CSI) Operator Driver that is supported by Red Hat. If you have installed a vSphere CSI driver provided by the community or another vendor, updates to the next major version of OKD, such as 4.13, or later, might be disabled for your cluster.

OKD 4.12, and later, clusters are still fully supported, and updates to z-stream releases of 4.12, such as 4.12.z, are not blocked, but you must correct this state by removing the third-party vSphere CSI Driver before updates to next major version of OKD can occur. Removing the third-party vSphere CSI driver does not require deletion of associated persistent volume (PV) objects, and no data loss should occur.

These instructions may not be complete, so consult the vendor or community provider uninstall guide to ensure removal of the driver and components.

To uninstall the third-party vSphere CSI Driver:

  1. Delete the third-party vSphere CSI Driver (VMware vSphere Container Storage Plugin) Deployment and Daemonset objects.

  2. Delete the configmap and secret objects that were installed previously with the third-party vSphere CSI Driver.

  3. Delete the third-party vSphere CSI driver CSIDriver object:

    1. ~ $ oc delete CSIDriver csi.vsphere.vmware.com
    1. csidriver.storage.k8s.io "csi.vsphere.vmware.com" deleted

After you have removed the third-party vSphere CSI Driver from the OKD cluster, installation of Red Hat’s vSphere CSI Operator Driver automatically resumes, and any conditions that could block upgrades to OKD 4.11, or later, are automatically removed. If you had existing vSphere CSI PV objects, their lifecycle is now managed by Red Hat’s vSphere CSI Operator Driver.

Configuring vSphere CSI Topology

OKD provides the ability to deploy OKD for vSphere on different zones and regions, which allows you to deploy over multiple compute clusters, thus helping to avoid a single point of failure.

OKD on vSphere does not support multiple Datacenters.

This is accomplished by defining zone and region categories in vCenter, and then assigning these categories to different failure domains, such as a compute cluster, by creating tags for these zone and region categories. After you have created the appropriate categories, and assigned tags to vCenter objects, you can create additional machinesets that create virtual machines (VMs) that are responsible for scheduling pods in those failure domains.

Procedure

  1. In the VMware vCenter vSphere client GUI, define appropriate zone and region catagories and tags.

    While vSphere allows you to create categories with any arbitrary name, OKD strongly recommends use of openshift-region and openshift-zone names for defining topology.

    The following example defines two failure domains with one region and two zones:

    Table 1. vSphere topology with one region and two zones
    Compute clusterFailure domainDescription

    Compute cluster: ocp1, Datacenter: Atlanta

    openshift-region: us-east-1 (tag), openshift-zone: us-east-1a (tag)

    This defines a failure domain in region us-east-1 with zone us-east-1a.

    Computer cluster: ocp2, Datacenter: Atlanta

    openshift-region: us-east-1 (tag), openshift-zone: us-east-1b (tag)

    This defines a different failure domain within the same region called us-east-1b.

    For more information about vSphere categories and tags, see the VMware vSphere documentation.

  2. To allow the container storage interface (CSI) driver to detect this topology, edit the clusterCSIDriver object YAML file driverConfig section:

    • Specify the openshift-zone and openshift-region categories that you created earlier.

    • Set driverType to vSphere.

      1. ~ $ oc edit clustercsidriver csi.vsphere.vmware.com -o yaml

      Example output

      1. apiVersion: operator.openshift.io/v1
      2. kind: ClusterCSIDriver
      3. metadata:
      4.  name: csi.vsphere.vmware.com
      5. spec:
      6.  logLevel: Normal
      7.  managementState: Managed
      8.  observedConfig: null
      9.  operatorLogLevel: Normal
      10.  unsupportedConfigOverrides: null
      11.  driverConfig:
      12.   driverType: vSphere (1)
      13.     vSphere:
      14.       topologyCategories: (2)
      15.       - openshift-zone
      16.       - openshift-region
      1Ensure that driverType is set to vSphere.
      2openshift-zone and openshift-region categories created earlier in vCenter.

  3. Verify that CSINode object has topology keys by running the following commands:

    1. ~ $ oc get csinode

    Example output

    1. NAME                        DRIVERS     AGE
    2. co8-4s88d-infra-2m5vd       1           27m
    3. co8-4s88d-master-0          1           70m
    4. co8-4s88d-master-1          1           70m
    5. co8-4s88d-master-2          1           70m
    6. co8-4s88d-worker-j2hmg      1           47m
    7. co8-4s88d-worker-mbb46      1           47m
    8. co8-4s88d-worker-zlk7d      1           47m
    1. ~ $ oc get csinode co8-4s88d-worker-j2hmg -o yaml

    Example output

    1. ...
    2. spec:
    3.     drivers:
    4.     - allocatable:
    5.         count: 59
    6.     name: csi-vsphere.vmware.com
    7.     nodeID: co8-4s88d-worker-j2hmg
    8.     topologyKeys: (1)
    9.     - topology.csi.vmware.com/openshift-zone
    10.     - topology.csi.vmware.com/openshift-region
    1Topology keys from vSphere openshift-zone and openshift-region catagories.

    CSINode objects might take some time to receive updated topology information. After the driver is updated, CSINode objects should have topology keys in them.

  4. Create a tag to assign to datastores across failure domains:

    When an OKD spans more than one failure domain, the datastore might not be shared across those failure domains, which is where topology-aware provisioning of persistent volumes (PVs) is useful.

    1. In vCenter, create a category for tagging the datastores. For example, openshift-zonal-datastore-cat. You can use any other category name, provided the category uniquely is used for tagging datastores participating in OKD cluster. Also, ensure that StoragePod, Datastore, and Folder are selected as Associable Entities for the created category.

    2. In vCenter, create a tag that uses the previously created category. This example uses the tag name openshift-zonal-datastore.

    3. Assign the previously created tag (in this example openshift-zonal-datastore) to each datastore in a failure domain that would be considered for dynamic provisioning.

      You can use any names you like for categories and tags. The names used in this example are provided as recommendations. Ensure that the tags and categories that you define uniquely identify only datastores that are shared with all hosts in the OKD cluster.

  5. Create a storage policy that targets the tag-based datastores in each failure domain:

    1. In vCenter, from the main menu, click Policies and Profiles.

    2. On the Policies and Profiles page, in the navigation pane, click VM Storage Policies.

    3. Click CREATE.

    4. Type a name for the storage policy.

    5. For the rules, choose Tag Placement rules and select the tag and category that targets the desired datastores (in this example, the openshift-zonal-datastore tag).

      The datastores are listed in the storage compatibility table.

  6. Create a new storage class that uses the new zoned storage policy:

    1. Click Storage > StorageClasses.

    2. On the StorageClasses page, click Create StorageClass.

    3. Type a name for the new storage class in Name.

    4. Under Provisioner, select csi.vsphere.vmware.com.

    5. Under Additional parameters, for the StoragePolicyName parameter, set Value to the name of the new zoned storage policy that you created earlier.

    6. Click Create.

      Example output

      1. kind: StorageClass
      2. apiVersion: storage.k8s.io/v1
      3. metadata:
      4.   name: zoned-sc (1)
      5. provisioner: csi.vsphere.vmware.com
      6. parameters:
      7.   StoragePolicyName: zoned-storage-policy (2)
      8. reclaimPolicy: Delete
      9. allowVolumeExpansion: true
      10. volumeBindingMode: WaitForFirstConsumer
      1New topology aware storage class name.
      2Specify zoned storage policy.

      You can also create the storage class by editing the preceding YAML file and running the command oc create -f $FILE.

Results

Creating persistent volume claims (PVCs) and PVs from the topology aware storage class are truly zonal, and should use the datastore in their respective zone depending on how pods are scheduled:

  1. ~ $ oc get pv <pv-name> -o yaml

Example output

  1. ...
  2. nodeAffinity:
  3.   required:
  4.     nodeSelectorTerms:
  5.     - matchExpressions:
  6.       - key: topology.csi.vmware.com/openshift-zone (1)
  7.         operator: In
  8.         values:
  9.         - <openshift-zone>
  10.       -key: topology.csi.vmware.com/openshift-region (1)
  11.         operator: In
  12.         values:
  13.         - <openshift-region>
  14. ...
  15. peristentVolumeclaimPolicy: Delete
  16. storageClassName: <zoned-storage-class-name> (2)
  17. volumeMode: Filesystem
  18. ...
1PV has zoned keys.
2PV is using the zoned storage class.

Additional resources

Additional resources