Shared File System

A shared file system can be mounted with read/write permission from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.

This example runs a shared file system for the kube-registry.

Prerequisites

This guide assumes you have created a Rook cluster as explained in the main Kubernetes guide

Multiple File Systems Not Supported

By default only one shared file system can be created with Rook. Multiple file system support in Ceph is still considered experimental and can be enabled with the environment variable ROOK_ALLOW_MULTIPLE_FILESYSTEMS defined in operator.yaml.

Please refer to cephfs experimental features page for more information.

Create the File System

Create the file system by specifying the desired settings for the metadata pool, data pools, and metadata server in the CephFilesystem CRD. In this example we create the metadata pool with replication of three and a single data pool with erasure coding. For more options, see the documentation on creating shared file systems.

Save this shared file system definition as filesystem.yaml:

  1. apiVersion: ceph.rook.io/v1
  2. kind: CephFilesystem
  3. metadata:
  4. name: myfs
  5. namespace: rook-ceph
  6. spec:
  7. metadataPool:
  8. replicated:
  9. size: 3
  10. dataPools:
  11. - replicated:
  12. size: 3
  13. metadataServer:
  14. activeCount: 1
  15. activeStandby: true

The Rook operator will create all the pools and other resources necessary to start the service. This may take a minute to complete.

  1. # Create the file system
  2. $ kubectl create -f filesystem.yaml
  3. # To confirm the file system is configured, wait for the mds pods to start
  4. $ kubectl -n rook-ceph get pod -l app=rook-ceph-mds
  5. NAME READY STATUS RESTARTS AGE
  6. rook-ceph-mds-myfs-7d59fdfcf4-h8kw9 1/1 Running 0 12s
  7. rook-ceph-mds-myfs-7d59fdfcf4-kgkjp 1/1 Running 0 12s

To see detailed status of the file system, start and connect to the Rook toolbox. A new line will be shown with ceph status for the mds service. In this example, there is one active instance of MDS which is up, with one MDS instance in standby-replay mode in case of failover.

  1. $ ceph status
  2. ...
  3. services:
  4. mds: myfs-1/1/1 up {[myfs:0]=mzw58b=up:active}, 1 up:standby-replay

Consume the Shared File System: K8s Registry Sample

As an example, we will start the kube-registry pod with the shared file system as the backing store. Save the following spec as kube-registry.yaml:

  1. apiVersion: v1
  2. kind: ReplicationController
  3. metadata:
  4. name: kube-registry-v0
  5. namespace: kube-system
  6. labels:
  7. k8s-app: kube-registry
  8. version: v0
  9. kubernetes.io/cluster-service: "true"
  10. spec:
  11. replicas: 3
  12. selector:
  13. k8s-app: kube-registry
  14. version: v0
  15. template:
  16. metadata:
  17. labels:
  18. k8s-app: kube-registry
  19. version: v0
  20. kubernetes.io/cluster-service: "true"
  21. spec:
  22. containers:
  23. - name: registry
  24. image: registry:2
  25. resources:
  26. limits:
  27. cpu: 100m
  28. memory: 100Mi
  29. env:
  30. - name: REGISTRY_HTTP_ADDR
  31. value: :5000
  32. - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
  33. value: /var/lib/registry
  34. volumeMounts:
  35. - name: image-store
  36. mountPath: /var/lib/registry
  37. ports:
  38. - containerPort: 5000
  39. name: registry
  40. protocol: TCP
  41. volumes:
  42. - name: image-store
  43. flexVolume:
  44. driver: ceph.rook.io/rook
  45. fsType: ceph
  46. options:
  47. fsName: myfs # name of the filesystem specified in the filesystem CRD.
  48. clusterNamespace: rook-ceph # namespace where the Rook cluster is deployed
  49. # by default the path is /, but you can override and mount a specific path of the filesystem by using the path attribute
  50. # the path must exist on the filesystem, otherwise mounting the filesystem at that path will fail
  51. # path: /some/path/inside/cephfs

After creating it with kubectl create -f kube-registry.yaml, you now have a docker registry which is HA with persistent storage.

Kernel Version Requirement

If the Rook cluster has more than one filesystem and the application pod is scheduled to a node with kernel version older than 4.7, inconsistent results may arise since kernels older than 4.7 do not support specifying filesystem namespaces.

Consume the Shared File System: Toolbox

Once you have pushed an image to the registry (see the instructions to expose and use the kube-registry), verify that kube-registry is using the filesystem that was configured above by mounting the shared file system in the toolbox pod. See the Direct Filesystem topic for more details.

Teardown

To clean up all the artifacts created by the file system demo:

  1. kubectl delete -f kube-registry.yaml

To delete the filesystem components and backing data, delete the Filesystem CRD. Warning: Data will be deleted

  1. kubectl -n rook-ceph delete cephfilesystem myfs

Advanced Example: Erasure Coded Filesystem

The Ceph filesystem example can be found here: Ceph Shared File System - Samples - Erasure Coded.