EdgeFS Rook integrated CSI driver, provisioner, attacher and snapshotter

Container Storage Interface (CSI) driver, provisioner, attacher and snapshotter for EdgeFS Scale-Out NFS/ISCSI services

Overview

EdgeFS CSI plugins implement an interface between CSI enabled Container Orchestrator (CO) and EdgeFS local cluster site. It allows dynamic and static provisioning of EdgeFS NFS exports and ISCSI LUNs, and attaching them to application workloads. With EdgeFS NFS/ISCSI implementation, I/O load can be spread-out across multiple PODs, thus eliminating I/O bottlenecks of classing single-node NFS/ISCSI and providing highly available persistent volumes. Current implementation of EdgeFS CSI plugins was tested in Kubernetes environment (requires Kubernetes 1.13+)

Prerequisites

  • Ensure your kubernetes cluster version is 1.13+
  • Kubernetes cluster must allow privileged pods, this flag must be set for the API server and the kubelet (instructions):
  1.   --allow-privileged=true
  • Required the API server and the kubelet feature gates (instructions):
  1.   --feature-gates=VolumeSnapshotDataSource=true,CSIDriverRegistry=true
  • Mount propagation must be enabled, the Docker daemon for the cluster must allow shared mounts (instructions)

  • Kubernetes CSI drivers require CSIDriver and CSINodeInfo resource types to be defined on the cluster. Check if they are already defined:

    1. kubectl get customresourcedefinition.apiextensions.k8s.io/csidrivers.csi.storage.k8s.io
    2. kubectl get customresourcedefinition.apiextensions.k8s.io/csinodeinfos.csi.storage.k8s.io

    If the cluster doesn’t have “csidrivers” and “csinodeinfos” resource types, create them:

    1. kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/release-1.13/pkg/crd/manifests/csidriver.yaml
    2. kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/release-1.13/pkg/crd/manifests/csinodeinfo.yaml

  • Depends on preferred CSI driver type, following utilities must be installed on each Kubernetes node (For Debian/Ubuntu based systems):

    1. # for NFS
    2. apt install -y nfs-common rpcbind
    3. # for ISCSI
    4. apt install -y open-iscsi

EdgeFS CSI drivers configuration

For each driver type (NFS/ISCSI) we have already prepared configuration files examples, there are:

Secret file configuration options example:

  1. # EdgeFS k8s cluster options
  2. k8sEdgefsNamespaces: ["rook-edgefs"]          # edgefs cluster namespace
  3. k8sEdgefsMgmtPrefix: rook-edgefs-mgr     # edgefs cluster management prefix
  5. # EdgeFS csi operations options
  6. cluster: cltest           # substitution edgefs cluster name for csi operations
  7. tenant: test              # substitution edgefs tenant name for csi operations
  8. #serviceFilter: "nfs01"   # comma delimited list of allowed service names for filtering
  10. # EdgeFS GRPC security options
  11. username: admin           # edgefs k8s cluster grpc service username
  12. password: admin           # edgefs k8s cluster grpc service password

Options for NFS and ISCSI configuration files

NameDescriptionDefault valueRequiredType
k8sEdgefsNamespacesArray of Kubernetes cluster’s namespaces for EdgeFS service discoveryrook-edgefstrueboth
k8sEdgefsMgmtPrefixRook EdgeFS cluster mgmt service prefixrook-edgefs-mgrtrueboth
usernameEdgeFS gRPC API server privileged useradmintrueboth
passwordEdgeFS gRPC API server passwordadmintrueboth
clusterEdgeFS cluster namespace also known as ‘region’ falseboth
tenantEdgeFS tenant isolated namespace falseboth
bucketEdgeFS tenant bucket to use as a default falseISCSI only
serviceFilterComma delimited list of allowed service names for filtering“” means all services allowedfalseboth
serviceBalancerPolicyService selection policy [minexportspolicy, randomservicepolicy]minexportspolicyfalseboth
chunksizeChunk size for actual volume, in bytes16384, should be power of twofalseboth
blocksizeBlock size for actual volume, in bytes4096, should be power of twofalseiSCSI only
fsTypeNew volume’s filesystem typeext4, ext3, xfsext4ISCSI only
forceVolumeDeletionAutomatically deletes EdgeFS volume after usagefalsefalseboth

By using k8sEdgefsNamespaces and k8sEdgefsMgmtPrefix parameters, driver is capable of detecting ClusterIPs and Endpoint IPs to provision and attach volumes.

Apply EdgeFS CSI NFS driver configuration

Check configuration options and create kubernetes secret for Edgefs CSI NFS plugin

  1. git clone --single-branch --branch release-1.2 https://github.com/rook/rook.git
  2. cd cluster/examples/kubernetes/edgefs/csi/nfs
  3. kubectl create secret generic edgefs-nfs-csi-driver-config --from-file=./edgefs-nfs-csi-driver-config.yaml

Deploy EdgeFS CSI NFS driver

After secret is created successfully, deploy EdgeFS CSI plugin, provisioner and attacher using the following command

  1. cd cluster/examples/kubernetes/edgefs/csi/nfs
  2. kubectl apply -f edgefs-nfs-csi-driver.yaml

There should be number of EdgeFS CSI plugin PODs available running as a DaemonSet:

  1. ...
  2. NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
  3. default       edgefs-nfs-csi-controller-0    4/4     Running   0          33s
  4. default       edgefs-nfs-csi-node-9st9n      2/2     Running   0          33s
  5. default       edgefs-nfs-csi-node-js7jp      2/2     Running   0          33s
  6. default       edgefs-nfs-csi-node-lhjgr      2/2     Running   0          33s
  7. ...

At this point configuration is all ready and available for consumption by applications.

Pre-provisioned volumes (NFS)

This method allows to use already created exports in EdgeFS services. This method keeps exports provisioned after application PODs terminated. Read more on how to create PersistentVolume specification for pre-provisioned volumes:

Link to Pre-provisioned volumes manifest specification

To test creation and mount pre-provisioned volume to pod execute example

NOTE: Make sure that volumeHandle: segment:service@cluster/tenant/bucket in nginx.yaml already exist on EdgeFS cluster and served via any Edgefs NFS service. Any volumeHandle’s parameters may be omitted and will be substituted via CSI configuration file parameters.

Examples:

  1. cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
  2. kubectl apply -f ./preprovisioned-edgefs-volume-nginx.yaml

Dynamically provisioned volumes (NFS)

To setup the system for dynamic provisioning, administrator needs to setup a StorageClass pointing to the CSI driver’s external-provisioner and specifying any parameters required by the driver

Link to dynamically provisioned volumes specification

Note

For dynamically provisioned volumes kubernetes will generate volume name automatically (for example pvc-871068ed-8b5d-11e8-9dae-005056b37cb2) Additional creation options should be passed as parameters in StorageClass definition i.e :

  1. apiVersion: storage.k8s.io/v1
  2. kind: StorageClass
  3. metadata:
  4.   name: edgefs-nfs-csi-storageclass
  5. provisioner: io.edgefs.csi.nfs
  6. parameters:
  7.   segment: rook-edgefs
  8.   service: nfs01
  9.   tenant: ten1
  10.   encryption: true

Parameters

NOTE: Parameters and their options are case sensitive and should be in lower case.

NameDescriptionAllowed valuesDefault value
segmentEdgefs cluster namespace for current StorageClass or PV. rook-edgefs
serviceEdgefs cluster service if not defined in secret  
clusterEdgefs cluster namespace if not defined in secret  
tenantEdgefs tenant namespace if not defined in secret  
chunksizeChunk size for actual volume, in bytesshould be power of two16384 bytes
blocksizeBlock size for actual volume, in bytesshould be power of two4096 bytes
aclVolume acl restrictions all
ecEnables ccow erasure coding for volumetrue, false, 0, 1false
ecmodeSet ccow erasure mode data mode (If ‘ec’ option enabled)3:1:xor, 2:2:rs, 4:2:rs, 6:2:rs, 9:3:rs6:2:rs
encryptionEnables encryption for volumetrue, false, 0, 1false

Example:

  1. cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
  2. kubectl apply -f ./dynamic-nginx.yaml

Apply Edgefs CSI ISCSI driver configuration

Check configuration options and create kubernetes secret for Edgefs CSI ISCSI plugin

  1. cd cluster/examples/kubernetes/edgefs/csi/iscsi
  2. kubectl create secret generic edgefs-iscsi-csi-driver-config --from-file=./edgefs-iscsi-csi-driver-config.yaml

Deploy Edgefs CSI ISCSI driver

After secret is created successfully, deploy EdgeFS CSI plugin, provisioner, attacher and snapshotter using the following command

  1. cd cluster/examples/kubernetes/edgefs/csi/iscsi
  2. kubectl apply -f edgefs-iscsi-csi-driver.yaml

There should be number of EdgeFS CSI ISCSI plugin PODs available running as a DaemonSet:

  1. ...
  2. NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
  3. default       edgefs-iscsi-csi-controller-0    4/4     Running   0          12s
  4. default       edgefs-iscsi-csi-node-26464      2/2     Running   0          12s
  5. default       edgefs-iscsi-csi-node-p5r58      2/2     Running   0          12s
  6. default       edgefs-iscsi-csi-node-ptn2m      2/2     Running   0          12s
  7. ...

At this point configuration is all ready and available for consumption by applications.

Pre-provisioned volumes (ISCSI)

This method allows to use already created exports in EdgeFS ISCSI services. This method keeps exports provisioned after application PODs terminated. Read more on how to create PersistentVolume specification for pre-provisioned volumes:

Link to Pre-provisioned volumes manifest specification

To test creation and mount pre-provisioned volume to pod execute example:

NOTE: Make sure that volumeHandle: segment:service@cluster/tenant/bucket/lun in nginx.yaml already exist on EdgeFS cluster and served via any Edgefs ISCSI service. Any volumeHandle’s parameters may be omitted and will be substituted via CSI configuration file parameters.

Example:

  1. cd cluster/examples/kubernetes/edgefs/csi/iscsi/examples
  2. kubectl apply -f ./preprovisioned-edgefs-volume-nginx.yaml

Dynamically provisioned volumes (ISCSI)

For dynamic volume provisioning, the administrator needs to set up a StorageClass pointing to the driver. In this case Kubernetes generates volume name automatically (for example pvc-ns-cfc67950-fe3c-11e8-a3ca-005056b857f8). Default driver configuration may be overwritten in parameters section:

Link to dynamically provisioned volumes specification

Note for dynamically provisioned volumes

For dynamically provisioned volumes, Kubernetes will generate volume names automatically (for example pvc-871068ed-8b5d-11e8-9dae-005056b37cb2). To pass additional creation parameters, you can add them as parameters to your StorageClass definition.

Example:

  1. apiVersion: storage.k8s.io/v1
  2. kind: StorageClass
  3. metadata:
  4.   name: edgefs-iscsi-csi-storageclass
  5. provisioner: io.edgefs.csi.nfs
  6. parameters:
  7.   segment: rook-edgefs
  8.   service: iscsi01
  9.   cluster: cltest
  10.   tenant: test
  11.   bucket: bk1
  12.   encryption: true

Parameters

NOTE: Parameters and their options are case sensitive and should be in lower case.

NameDescriptionAllowed valuesDefault value
segmentEdgefs cluster namespace for specific StorageClass or PV. rook-edgefs
serviceEdgefs cluster service if not defined in secret  
clusterEdgefs cluster namespace if not defined in secret  
tenantEdgefs tenant namespace if not defined in secret  
bucketEdgefs bucket namespace if not defined in secret  
chunksizeChunk size for actual volume, in bytesshould be power of two16384
blocksizeBlocksize size for actual volume, in bytesshould be power of two4096
fsTypeNew volume’s filesystem typeext4, ext3, xfsext4
aclVolume acl restrictions all
ecEnables ccow erasure coding for volumetrue, false, 0, 1false
ecmodeSet ccow erasure mode data mode (If ‘ec’ option enabled)4:2:rs ,6:2:rs, 4:2:rs, 9:3:rs4:2:rs
encryptionEnables encryption for volumetrue, false, 0, 1false

Example:

  1. cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
  2. kubectl apply -f ./dynamic-nginx.yaml

EdgeFS CSI ISCSI driver snapshots and clones

Getting information about existing snapshots

  1. # snapshot classes
  2. kubectl get volumesnapshotclasses.snapshot.storage.k8s.io
  4. # snapshot list
  5. kubectl get volumesnapshots.snapshot.storage.k8s.io
  7. # volumesnapshotcontents
  8. kubectl get volumesnapshotcontents.snapshot.storage.k8s.io

To create volume’s clone from existing snapshot you should:

Troubleshooting and log collection

For details about other configuration and deployment of NFS, ISCSI and EdgeFS CSI plugin, see Wiki pages:

Please submit an issue at: Issues

Troubleshooting

  • Show installed drivers:
  1. kubectl get csidrivers.csi.storage.k8s.io
  2. kubectl describe csidrivers.csi.storage.k8s.io
  • Error:
  1. MountVolume.MountDevice failed for volume "pvc-ns-<...>" :
  2. driver name io.edgefs.csi.iscsi not found in the list of registered CSI drivers

Make sure kubelet is configured with --root-dir=/var/lib/kubelet, otherwise update paths in the driver yaml file (all requirements).

  • “VolumeSnapshotDataSource” feature gate is disabled:
  1. vim /var/lib/kubelet/config.yaml
  2. # ...
  3. # featureGates:
  4. #   VolumeSnapshotDataSource: true
  5. # ...
  6. vim /etc/kubernetes/manifests/kube-apiserver.yaml
  7. # ...
  8. #     - --feature-gates=VolumeSnapshotDataSource=true
  9. # ...
  • Driver logs (for ISCSI driver, to get NFS driver logs substitute iscsi to nfs)
  1. kubectl logs -f edgefs-iscsi-csi-controller-0 driver
  2. kubectl logs -f $(kubectl get pods | awk '/edgefs-iscsi-csi-node-/ {print $1;exit}') driver
  3. # combine all pods:
  4. kubectl get pods | awk '/edgefs-iscsi-csi-node-/ {system("kubectl logs " $1 " driver &")}'
  • Show termination message in case driver failed to run:
  1. kubectl get pod edgefs-iscsi-csi-controller-0 -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"