Network File System (NFS)

NFS allows remote hosts to mount file systems over a network and interact with those file systems as though they are mounted locally. This enables system administrators to consolidate resources onto centralized servers on the network.

Prerequisites

  1. A Kubernetes cluster is necessary to run the Rook NFS operator. To make sure you have a Kubernetes cluster that is ready for Rook, you can follow these instructions.
  2. The desired volume to export needs to be attached to the NFS server pod via a PVC. Any type of PVC can be attached and exported, such as Host Path, AWS Elastic Block Store, GCP Persistent Disk, CephFS, Ceph RBD, etc. The limitations of these volumes also apply while they are shared by NFS. You can read further about the details and limitations of these volumes in the Kubernetes docs.
  3. NFS client packages must be installed on all nodes where Kubernetes might run pods with NFS mounted. Install nfs-utils on CentOS nodes or nfs-common on Ubuntu nodes.

Deploy NFS Operator

First deploy the Rook NFS operator using the following commands:

  1. cd cluster/examples/kubernetes/nfs
  2. kubectl create -f operator.yaml

You can check if the operator is up and running with:

  1. kubectl -n rook-nfs-system get pod
  3. NAME                                READY     STATUS    RESTARTS   AGE
  4. rook-nfs-operator-b8d6d955d-cdq2v   1/1       Running   0          1m

Create and Initialize NFS Server

Now that the operator is running, we can create an instance of a NFS server by creating an instance of the nfsservers.nfs.rook.io resource. The various fields and options of the NFS server resource can be used to configure the server and its volumes to export. Full details of the available configuration options can be found in the NFS CRD documentation.

This guide has 2 main examples that demonstrate exporting volumes with a NFS server:

  1. Default StorageClass example
  2. Rook Ceph volume example

Default StorageClass example

This first example will walk through creating a NFS server instance that exports storage that is backed by the default StorageClass for the environment you happen to be running in. In some environments, this could be a host path, in others it could be a cloud provider virtual disk. Either way, this example requires a default StorageClass to exist.

Start by saving the below NFS CRD instance definition to a file called nfs.yaml:

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name:  rook-nfs
  5. ---
  6. # A default StorageClass must be present
  7. apiVersion: v1
  8. kind: PersistentVolumeClaim
  9. metadata:
  10.   name: nfs-default-claim
  11.   namespace: rook-nfs
  12. spec:
  13.   accessModes:
  14.   - ReadWriteMany
  15.   resources:
  16.     requests:
  17.       storage: 1Gi
  18. ---
  19. apiVersion: nfs.rook.io/v1alpha1
  20. kind: NFSServer
  21. metadata:
  22.   name: rook-nfs
  23.   namespace: rook-nfs
  24. spec:
  25.   replicas: 1
  26.   exports:
  27.   - name: nfs-share
  28.     server:
  29.       accessMode: ReadWrite
  30.       squash: "none"
  31.     # A Persistent Volume Claim must be created before creating NFS CRD instance.
  32.     persistentVolumeClaim:
  33.       claimName: nfs-default-claim

With the nfs.yaml file saved, now create the NFS server as shown:

  1. kubectl create -f nfs.yaml

We can verify that a Kubernetes object has been created that represents our new NFS server and its export with the command below.

  1. kubectl -n rook-nfs get nfsservers.nfs.rook.io
  3. NAME       AGE
  4. rook-nfs   1m

Verify that the NFS server pod is up and running:

  1. kubectl -n rook-nfs get pod -l app=rook-nfs
  3. NAME         READY     STATUS    RESTARTS   AGE
  4. rook-nfs-0   1/1       Running   0          2m

If the NFS server pod is in the Running state, then we have successfully created an exported NFS share that clients can start to access over the network.

Accessing the Export

To access the export from another pod, you must first manually create a PersistentVolume with the connection information. This experience is a bit cumbersome, but will be improved in the future with dynamic provisioning support. First, find the current IP address of your NFS server pod using the following command:

  1. kubectl -n rook-nfs get service -l app=rook-nfs -o jsonpath='{.items[0].spec.clusterIP}'

Copy this IP address and insert it into the following content in place of the <Cluster IP> placeholder. Then also replace the /<Claim name> value with the claimName that was configured in the nfs.yaml. In this case, it will be nfs-default-claim and it will look like path: "/nfs-default-claim". Finally, save the updated content to a file called pv.yaml.

  1. apiVersion: v1
  2. kind: PersistentVolume
  3. metadata:
  4.   name: rook-nfs-pv
  5.   namespace: rook-nfs
  6. spec:
  7.   capacity:
  8.     storage: 1Mi
  9.   accessModes:
  10.     - ReadWriteMany
  11.   nfs:
  12.     # FIXME: replace <Cluster IP> with the current IP of the NFS server pod.
  13.     # Run the below command to get the current IP address:
  14.     # kubectl -n rook-nfs get service -l app=rook-nfs -o jsonpath='{.items[0].spec.clusterIP}'
  15.     server: <Cluster IP>
  16.     path: "/<Claim name>"

With the pv.yaml file saved, we can create the PV object:

  1. kubectl create -f pv.yaml

Consuming the Export

Now we can consume the PV that we just created by creating an example web server app that uses a PersistentVolumeClaim to claim the exported volume. There are 2 pods that comprise this example:

  1. A web server pod that will read and display the contents of the NFS share
  2. A writer pod that will write random data to the NFS share so the website will continually update

First, save the PVC definition in a file called pvc.yaml:

  1. apiVersion: v1
  2. kind: PersistentVolumeClaim
  3. metadata:
  4.   name: rook-nfs-pv-claim
  5. spec:
  6.   accessModes:
  7.     - ReadWriteMany
  8.   storageClassName: ""
  9.   resources:
  10.     requests:
  11.       storage: 1Mi

Then create the PVC:

  1. kubectl create -f pvc.yaml

Start both the busybox pod (writer) and the web server from the cluster/examples/kubernetes/nfs folder:

  1. kubectl create -f busybox-rc.yaml
  2. kubectl create -f web-rc.yaml

Let’s confirm that the expected busybox writer pods and web server pods are all up and in the Running state:

  1. kubectl get pod -l app=nfs-demo

In order to be able to reach the web server over the network, let’s create a service for it:

  1. kubectl create -f web-service.yaml

We can then use the busybox writer pod we launched before to check that nginx is serving the data appropriately. In the below 1-liner command, we use kubectl exec to run a command in the busybox writer pod that uses wget to retrieve the web page that the web server pod is hosting. As the busybox writer pod continues to write a new timestamp, we should see the returned output also update every ~10 seconds or so.

  1. > echo; kubectl exec $(kubectl get pod -l name=nfs-busybox -o jsonpath='{.items[0].metadata.name}') -- wget -qO- http://$(kubectl get services nfs-web -o jsonpath='{.spec.clusterIP}'); echo
  3. Thu Oct 22 19:28:55 UTC 2015
  4. nfs-busybox-w3s4t

Rook Ceph volume example

In this alternative example, we will use a different underlying volume as an export for the NFS server. These steps will walk us through exporting a Ceph RBD block volume so that clients can access it across the network.

First, you have to follow these instructions to deploy a sample Rook Ceph cluster that can be attached to the NFS server pod for sharing. After the Rook Ceph cluster is up and running, we can create proceed with creating the NFS server.

Save this PVC and NFS CRD instance as nfs-ceph.yaml:

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name:  rook-nfs
  5. ---
  6. # A rook ceph cluster must be running
  7. # Create a rook ceph cluster using examples in rook/cluster/examples/kubernetes/ceph
  8. # Refer to https://rook.io/docs/rook/master/ceph-quickstart.html for a quick rook cluster setup
  9. apiVersion: v1
  10. kind: PersistentVolumeClaim
  11. metadata:
  12.   name: nfs-ceph-claim
  13.   namespace: rook-nfs
  14. spec:
  15.   storageClassName: rook-ceph-block
  16.   accessModes:
  17.   - ReadWriteMany
  18.   resources:
  19.     requests:
  20.       storage: 2Gi
  21. ---
  22. apiVersion: nfs.rook.io/v1alpha1
  23. kind: NFSServer
  24. metadata:
  25.   name: rook-nfs
  26.   namespace: rook-nfs
  27. spec:
  28.   replicas: 1
  29.   exports:
  30.   - name: nfs-share
  31.     server:
  32.       accessMode: ReadWrite
  33.       squash: "none"
  34.     # A Persistent Volume Claim must be created before creating NFS CRD instance.
  35.     # Create a Ceph cluster for using this example
  36.     # Create a ceph PVC after creating the rook ceph cluster using ceph-pvc.yaml
  37.     persistentVolumeClaim:
  38.       claimName: nfs-ceph-claim

Create the NFS server instance that you saved in nfs-ceph.yaml:

  1. kubectl create -f nfs-ceph.yaml

After the NFS server pod is running, follow the same instructions from the previous example to access and consume the NFS share, with the following exception:

  • Replace the /<Claim name> value in the PV definition in the Accessing the Export section with the claimName that was configured in the nfs-ceph.yaml. In this case it will be nfs-ceph-claim and will look like path: "/nfs-ceph-claim".

After that, follow the rest of the instructions in the Accessing the Export section and then the Consuming the Export section to consume the NFS volume.

Teardown

To clean up all resources associated with this walk-through, you can run the commands below.

  1. kubectl delete -f web-service.yaml
  2. kubectl delete -f web-rc.yaml
  3. kubectl delete -f busybox-rc.yaml
  4. kubectl delete -f pvc.yaml
  5. kubectl delete -f pv.yaml
  6. kubectl delete -f nfs.yaml
  7. kubectl delete -f nfs-ceph.yaml
  8. kubectl delete -f operator.yaml

Troubleshooting

If the NFS server pod does not come up, the first step would be to examine the NFS operator’s logs:

  1. kubectl -n rook-nfs-system logs -l app=rook-nfs-operator