Persistent Storage Using FlexVolume Plug-ins

Overview

OKD has built-in volume plug-ins to use different storage technologies. To use storage from a back-end that does not have a built-in plug-in, you can extend OKD through FlexVolume drivers and provide persistent storage to applications.

FlexVolume drivers

A FlexVolume driver is an executable file that resides in a well-defined directory on all machines in the cluster, both masters and nodes. OKD calls the FlexVolume driver whenever it needs to attach, detach, mount, or unmount a volume represented by a PersistentVolume with flexVolume as the source.

The first command-line argument of the driver is always an operation name. Other parameters are specific to each operation. Most of the operations take a JavaScript Object Notation (JSON) string as a parameter. This parameter is a complete JSON string, and not the name of a file with the JSON data.

The FlexVolume driver contains:

• All flexVolume.options.

• Some options from flexVolume prefixed by kubernetes.io/, such as fsType and readwrite.

• The content of the referenced secret, if specified, prefixed by kubernetes.io/secret/.

FlexVolume driver JSON input example

{
    "fooServer": "192.168.0.1:1234", (1)
        "fooVolumeName": "bar",
    "kubernetes.io/fsType": "ext4", (2)
    "kubernetes.io/readwrite": "ro", (3)
    "kubernetes.io/secret/<key name>": "<key value>", (4)
    "kubernetes.io/secret/<another key name>": "<another key value>",
}

OKD expects JSON data on standard output of the driver. When not specified, the output describes the result of the operation.

FlexVolume Driver Default Output

{
    "status": "<Success/Failure/Not supported>",
    "message": "<Reason for success/failure>"
}

Exit code of the driver should be 0 for success and 1 for error.

Operations should be idempotent, which means that the attachment of an already attached volume or the mounting of an already mounted volume should result in a successful operation.

The FlexVolume driver can work in two modes:

• with the master-initated attach/detach operation, or

• without the master-initated attach/detach operation.

The attach/detach operation is used by the OKD master to attach a volume to a node and to detach it from a node. This is useful when a node becomes unresponsive for any reason. Then, the master can kill all pods on the node, detach all volumes from it, and attach the volumes to other nodes to resume the applications while the original node is still not reachable.

FlexVolume drivers with master-initiated attach/detach

A FlexVolume driver that supports master-controlled attach/detach must implement the following operations:

init

Initializes the driver. It is called during initialization of masters and nodes.

• Arguments: none

• Executed on: master, node

• Expected output: default JSON

getvolumename

Returns the unique name of the volume. This name must be consistent among all masters and nodes, because it is used in subsequent detach call as <volume-name>. Any / characters in the <volume-name> are automatically replaced by ~.

• Arguments: <json>

• Executed on: master, node

• Expected output: default JSON + volumeName:

  {
      "status": "Success",
      "message": "",
      "volumeName": "foo-volume-bar" (1)
  }

  1	The unique name of the volume in storage back-end foo.

attach

Attaches a volume represented by the JSON to a given node. This operation should return the name of the device on the node if it is known, that is, if it has been assigned by the storage back-end before it runs. If the device is not known, the device must be found on the node by the subsequent waitforattach operation.

• Arguments: <json> <node-name>

• Executed on: master

• Expected output: default JSON + device, if known:

  {
      "status": "Success",
      "message": "",
      "device": "/dev/xvda" (1)
  }

  1	The name of the device on the node, if known.

waitforattach

Waits until a volume is fully attached to a node and its device emerges. If the previous attach operation has returned <device-name>, it is provided as an input parameter. Otherwise, <device-name> is empty and the operation must find the device on the node.

• Arguments: <device-name> <json>

• Executed on: node

• Expected output: default JSON + device

  {
      "status": "Success",
      "message": "",
      "device": "/dev/xvda" (1)
  }

  1	The name of the device on the node.

detach

Detaches the given volume from a node. <volume-name> is the name of the device returned by the getvolumename operation. Any / characters in the <volume-name> are automatically replaced by ~.

• Arguments: <volume-name> <node-name>

• Executed on: master

• Expected output: default JSON

isattached

Checks that a volume is attached to a node.

• Arguments: <json> <node-name>

• Executed on: master

• Expected output: default JSON + attached

  {
      "status": "Success",
      "message": "",
      "attached": true (1)
  }

  1	The status of attachment of the volume to the node.

mountdevice

Mounts a volume’s device to a directory. <device-name> is name of the device as returned by the previous waitforattach operation.

• Arguments: <mount-dir> <device-name> <json>

• Executed on: node

• Expected output: default JSON

unmountdevice

Unmounts a volume’s device from a directory.

• Arguments: <mount-dir>

• Executed on: node

All other operations should return JSON with {"status": "Not supported"} and exit code 1.

FlexVolume drivers without master-initiated attach/detach

FlexVolume drivers that do not support master-controlled attach/detach are executed only on the node and must implement these operations:

init

Initializes the driver. It is called during initialization of all nodes.

• Arguments: none

• Executed on: node

• Expected output: default JSON

mount

Mounts a volume to directory. This can include anything that is necessary to mount the volume, including attaching the volume to the node, finding the its device, and then mounting the device.

• Arguments: <mount-dir> <json>

• Executed on: node

• Expected output: default JSON

unmount

Unmounts a volume from a directory. This can include anything that is necessary to clean up the volume after unmounting, such as detaching the volume from the node.

• Arguments: <mount-dir>

• Executed on: node

• Expected output: default JSON

All other operations should return JSON with {"status": "Not supported"} and exit code 1.

Installing FlexVolume drivers

To install the FlexVolume driver:

1. Ensure that the executable file exists on all masters and nodes in the cluster.

2. Place the executable file at the volume plug-in path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/~/.

For example, to install the FlexVolume driver for the storage foo, place the executable file at: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/openshift.com~foo/foo.

In OKD 3.11, since controller-manager runs as a static pod, the FlexVolume binary file that performs the attach and detach operations must be a self-contained executable file with no external dependencies.

On Atomic hosts, the default location of the FlexVolume plug-in directory is /etc/origin/kubelet-plugins/. You must place the FlexVolume executable file in the /etc/origin/kubelet-plugins/volume/exec/~/ directory on all master and nodes in the cluster.

Consuming storage using FlexVolume drivers

Use the PersistentVolume object to reference the installed storage. Each PersistentVolume object in OKD represents one storage asset, typically a volume, in the storage back-end.

Persistent volume object definition using FlexVolume drivers example

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001 (1)
spec:
  capacity:
    storage: 1Gi (2)
  accessModes:
    - ReadWriteOnce
  flexVolume:
    driver: openshift.com/foo (3)
    fsType: "ext4" (4)
    secretRef: foo-secret (5)
    readOnly: true (6)
    options: (7)
      fooServer: 192.168.0.1:1234
      fooVolumeName: bar

"fsType":"<FS type>",
"readwrite":"<rw>",
"secret/key1":"<secret1>"
...
"secret/keyN":"<secretN>"