Persistent Storage Using Container Storage Interface (CSI)

Overview

Container Storage Interface (CSI) allows OKD to consume storage from storage backends that implement the CSI interface as persistent storage.

CSI volumes are currently in Technology Preview and not for production workloads. CSI volumes may change in a future release of OKD.

OKD does not ship with any CSI drivers. It is recommended to use the CSI drivers provided by community or storage vendors.

OKD 3.11 supports version 0.2.0 of the CSI specification.

Architecture

CSI drivers are typically shipped as container images. These containers are not aware of OKD where they run. To use CSI-compatible storage backend in OKD, the cluster administrator must deploy several components that serve as a bridge between OKD and the storage driver.

The following diagram provides a high-level overview about the components running in pods in the OKD cluster.

Architecture of CSI components

It is possible to run multiple CSI drivers for different storage backends. Each driver needs its own external controllers’ deployment and DaemonSet with the driver and CSI registrar.

External CSI Controllers

External CSI Controllers is a deployment that deploys one or more pods with three containers:

  • External CSI attacher container that translates attach and detach calls from OKD to respective ControllerPublish and ControllerUnpublish calls to CSI driver

  • External CSI provisioner container that translates provision and delete calls from OKD to respective CreateVolume and DeleteVolume calls to CSI driver

  • CSI driver container

The CSI attacher and CSI provisioner containers talk to the CSI driver container using UNIX Domain Sockets, ensuring that no CSI communication leaves the pod. The CSI driver is not accessible from outside of the pod.

attach, detach, provision, and delete operations typically require the CSI driver to use credentials to the storage backend. Run the CSI controller pods on infrastructure nodes so the credentials never leak to user processes, even in the event of a catastrophic security breach on a compute node.

The external attacher must also run for CSI drivers that do not support third-party attach/detach operations. The external attacher will not issue any ControllerPublish or ControllerUnpublish operations to the CSI driver. However, it still must run to implement the necessary OKD attachment API.

CSI Driver DaemonSet

Finally, the CSI driver DaemonSet runs a pod on every node that allows OKD to mount storage provided by the CSI driver to the node and use it in user workloads (pods) as persistent volumes (PVs). The pod with the CSI driver installed contains the following containers:

  • CSI driver registrar, which registers the CSI driver into the openshift-node service running on the node. The openshift-node process running on the node then directly connects with the CSI driver using the UNIX Domain Socket available on the node.

  • CSI driver.

The CSI driver deployed on the node should have as few credentials to the storage backend as possible. OKD will only use the node plug-in set of CSI calls such as NodePublish/NodeUnpublish and NodeStage/NodeUnstage (if implemented).

Example Deployment

Since OKD does not ship with any CSI driver installed, this example shows how to deploy a community driver for OpenStack Cinder in OKD.

  1. Create a new project where the CSI components will run and a new service account that will run the components. Explicit node selector is used to run the Daemonset with the CSI driver also on master nodes.

    1. # oc adm new-project csi --node-selector=""
    2. Now using project "csi" on server "https://example.com:8443".
    4. # oc create serviceaccount cinder-csi
    5. serviceaccount "cinder-csi" created
    7. # oc adm policy add-scc-to-user privileged system:serviceaccount:csi:cinder-csi
    8. scc "privileged" added to: ["system:serviceaccount:csi:cinder-csi"]

  2. Apply this YAML file to create the deployment with the external CSI attacher and provisioner and DaemonSet with the CSI driver.

    1. # This YAML file contains all API objects that are necessary to run Cinder CSI
    2. # driver.
    3. #
    4. # In production, this needs to be in separate files, e.g. service account and
    5. # role and role binding needs to be created once.
    6. #
    7. # It serves as an example of how to use external attacher and external provisioner
    8. # images that are shipped with OpenShift Container Platform with a community CSI driver.
    10. kind: ClusterRole
    11. apiVersion: rbac.authorization.k8s.io/v1
    12. metadata:
    13.   name: cinder-csi-role
    14. rules:
    15.   - apiGroups: [""]
    16.     resources: ["persistentvolumes"]
    17.     verbs: ["create", "delete", "get", "list", "watch", "update", "patch"]
    18.   - apiGroups: [""]
    19.     resources: ["events"]
    20.     verbs: ["create", "get", "list", "watch", "update", "patch"]
    21.   - apiGroups: [""]
    22.     resources: ["persistentvolumeclaims"]
    23.     verbs: ["get", "list", "watch", "update", "patch"]
    24.   - apiGroups: [""]
    25.     resources: ["nodes"]
    26.     verbs: ["get", "list", "watch", "update", "patch"]
    27.   - apiGroups: ["storage.k8s.io"]
    28.     resources: ["storageclasses"]
    29.     verbs: ["get", "list", "watch"]
    30.   - apiGroups: ["storage.k8s.io"]
    31.     resources: ["volumeattachments"]
    32.     verbs: ["get", "list", "watch", "update", "patch"]
    33.   - apiGroups: [""]
    34.     resources: ["configmaps"]
    35.     verbs: ["get", "list", "watch", "create", "update", "patch"]
    37. ---
    39. kind: ClusterRoleBinding
    40. apiVersion: rbac.authorization.k8s.io/v1
    41. metadata:
    42.   name: cinder-csi-role
    43. subjects:
    44.   - kind: ServiceAccount
    45.     name: cinder-csi
    46.     namespace: csi
    47. roleRef:
    48.   kind: ClusterRole
    49.   name: cinder-csi-role
    50.   apiGroup: rbac.authorization.k8s.io
    52. ---
    53. apiVersion: v1
    54. data:
    55.   cloud.conf: W0dsb2JhbF0KYXV0aC11cmwgPSBodHRwczovL2V4YW1wbGUuY29tOjEzMDAwL3YyLjAvCnVzZXJuYW1lID0gYWxhZGRpbgpwYXNzd29yZCA9IG9wZW5zZXNhbWUKdGVuYW50LWlkID0gZTBmYTg1YjZhMDY0NDM5NTlkMmQzYjQ5NzE3NGJlZDYKcmVnaW9uID0gcmVnaW9uT25lCg== (1)
    56. kind: Secret
    57. metadata:
    58.   creationTimestamp: null
    59.   name: cloudconfig
    60. ---
    61. kind: Deployment
    62. apiVersion: apps/v1
    63. metadata:
    64.   name: cinder-csi-controller
    65. spec:
    66.   replicas: 2
    67.   selector:
    68.     matchLabels:
    69.       app: cinder-csi-controllers
    70.   template:
    71.     metadata:
    72.       labels:
    73.         app: cinder-csi-controllers
    74.     spec:
    75.       serviceAccount: cinder-csi
    76.       containers:
    77.         - name: csi-attacher
    78.           image: registry.redhat.io/openshift3/csi-attacher:v3.11
    79.           args:
    80.             - "--v=5"
    81.             - "--csi-address=$(ADDRESS)"
    82.             - "--leader-election"
    83.             - "--leader-election-namespace=$(MY_NAMESPACE)"
    84.             - "--leader-election-identity=$(MY_NAME)"
    85.           env:
    86.             - name: MY_NAME
    87.               valueFrom:
    88.                 fieldRef:
    89.                   fieldPath: metadata.name
    90.             - name: MY_NAMESPACE
    91.               valueFrom:
    92.                 fieldRef:
    93.                   fieldPath: metadata.namespace
    94.             - name: ADDRESS
    95.               value: /csi/csi.sock
    96.           volumeMounts:
    97.             - name: socket-dir
    98.               mountPath: /csi
    99.         - name: csi-provisioner
    100.           image: registry.redhat.io/openshift3/csi-provisioner:v3.11
    101.           args:
    102.             - "--v=5"
    103.             - "--provisioner=csi-cinderplugin"
    104.             - "--csi-address=$(ADDRESS)"
    105.           env:
    106.             - name: ADDRESS
    107.               value: /csi/csi.sock
    108.           volumeMounts:
    109.             - name: socket-dir
    110.               mountPath: /csi
    111.         - name: cinder-driver
    112.           image: quay.io/jsafrane/cinder-csi-plugin
    113.           command: [ "/bin/cinder-csi-plugin" ]
    114.           args:
    115.             - "--nodeid=$(NODEID)"
    116.             - "--endpoint=unix://$(ADDRESS)"
    117.             - "--cloud-config=/etc/cloudconfig/cloud.conf"
    118.           env:
    119.             - name: NODEID
    120.               valueFrom:
    121.                 fieldRef:
    122.                   fieldPath: spec.nodeName
    123.             - name: ADDRESS
    124.               value: /csi/csi.sock
    125.           volumeMounts:
    126.             - name: socket-dir
    127.               mountPath: /csi
    128.             - name: cloudconfig
    129.               mountPath: /etc/cloudconfig
    130.       volumes:
    131.         - name: socket-dir
    132.           emptyDir:
    133.         - name: cloudconfig
    134.           secret:
    135.             secretName: cloudconfig
    137. ---
    139. kind: DaemonSet
    140. apiVersion: apps/v1
    141. metadata:
    142.   name: cinder-csi-ds
    143. spec:
    144.   selector:
    145.     matchLabels:
    146.       app: cinder-csi-driver
    147.   template:
    148.     metadata:
    149.       labels:
    150.         app: cinder-csi-driver
    151.     spec:
    152.       (2)
    153.       serviceAccount: cinder-csi
    154.       containers:
    155.         - name: csi-driver-registrar
    156.           image: registry.redhat.io/openshift3/csi-driver-registrar:v3.11
    157.           securityContext:
    158.             privileged: true
    159.           args:
    160.             - "--v=5"
    161.             - "--csi-address=$(ADDRESS)"
    162.           env:
    163.             - name: ADDRESS
    164.               value: /csi/csi.sock
    165.             - name: KUBE_NODE_NAME
    166.               valueFrom:
    167.                 fieldRef:
    168.                   fieldPath: spec.nodeName
    169.           volumeMounts:
    170.             - name: socket-dir
    171.               mountPath: /csi
    172.         - name: cinder-driver
    173.           securityContext:
    174.             privileged: true
    175.             capabilities:
    176.               add: ["SYS_ADMIN"]
    177.             allowPrivilegeEscalation: true
    178.           image: quay.io/jsafrane/cinder-csi-plugin
    179.           command: [ "/bin/cinder-csi-plugin" ]
    180.           args:
    181.             - "--nodeid=$(NODEID)"
    182.             - "--endpoint=unix://$(ADDRESS)"
    183.             - "--cloud-config=/etc/cloudconfig/cloud.conf"
    184.           env:
    185.             - name: NODEID
    186.               valueFrom:
    187.                 fieldRef:
    188.                   fieldPath: spec.nodeName
    189.             - name: ADDRESS
    190.               value: /csi/csi.sock
    191.           volumeMounts:
    192.             - name: socket-dir
    193.               mountPath: /csi
    194.             - name: cloudconfig
    195.               mountPath: /etc/cloudconfig
    196.             - name: mountpoint-dir
    197.               mountPath: /var/lib/origin/openshift.local.volumes/pods/
    198.               mountPropagation: "Bidirectional"
    199.             - name: cloud-metadata
    200.               mountPath: /var/lib/cloud/data/
    201.             - name: dev
    202.               mountPath: /dev
    203.       volumes:
    204.         - name: cloud-metadata
    205.           hostPath:
    206.             path: /var/lib/cloud/data/
    207.         - name: socket-dir
    208.           hostPath:
    209.             path: /var/lib/kubelet/plugins/csi-cinderplugin
    210.             type: DirectoryOrCreate
    211.         - name: mountpoint-dir
    212.           hostPath:
    213.             path: /var/lib/origin/openshift.local.volumes/pods/
    214.             type: Directory
    215.         - name: cloudconfig
    216.           secret:
    217.             secretName: cloudconfig
    218.         - name: dev
    219.           hostPath:
    220.             path: /dev
    1Replace with cloud.conf for your OpenStack deployment, as described in OpenStack configuration. For example, the Secret can be generated using the oc create secret generic cloudconfig —from-file cloud.conf —dry-run -o yaml.
    2Optionally, add nodeSelector to the CSI driver pod template to configure the nodes on which the CSI driver starts. Only nodes matching the selector run pods that use volumes that are served by the CSI driver. Without nodeSelector, the driver runs on all nodes in the cluster.

Dynamic Provisioning

Dynamic provisioning of persistent storage depends on the capabilities of the CSI driver and underlying storage backend. The provider of the CSI driver should document how to create a StorageClass in OKD and the parameters available for configuration.

As seen in the OpenStack Cinder example, you can deploy this StorageClass to enable dynamic provisioning. The following example creates a new default storage class that ensures that all PVCs that do not require any special storage class are provisioned by the installed CSI driver:

  1. # oc create -f - << EOF
  2. apiVersion: storage.k8s.io/v1
  3. kind: StorageClass
  4. metadata:
  5.   name: cinder
  6.   annotations:
  7.     storageclass.kubernetes.io/is-default-class: "true"
  8. provisioner: csi-cinderplugin
  9. parameters:
  10. EOF

Usage

Once the CSI driver is deployed and the StorageClass for dynamic provisioning is created, OKD is ready to use CSI. The following example installs a default MySQL template without any changes to the template:

  1. # oc new-app mysql-persistent
  2. --> Deploying template "openshift/mysql-persistent" to project default
  3. ...
  5. # oc get pvc
  6. NAME              STATUS    VOLUME                                   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  7. mysql             Bound     kubernetes-dynamic-pv-3271ffcb4e1811e8   1Gi        RWO            cinder         3s