Ephemeral local storage

Overview

In addition to persistent storage, pods and containers may require ephemeral or transient local storage for their operation. The lifetime of this ephemeral storage does not extend beyond the life of the individual pod, and this ephemeral storage cannot be shared across pods.

Prior to OKD 3.10, ephemeral local storage was exposed to pods using the container’s writable layer, logs directory, and EmptyDir volumes. Pods use ephemeral local storage for scratch space, caching, and logs. Issues related to the lack of local storage accounting and isolation include the following:

• Pods do not know how much local storage is available to them.

• Pods cannot request guaranteed local storage.

• Local storage is a best effort resource.

• Pods can be evicted due to other pods filling the local storage, after which new pods are not admitted until sufficient storage has been reclaimed.

Unlike persistent volumes, ephemeral storage is unstructured and shared, the space, not the actual data, between all pods running on a node, in addition to other uses by the system, the container runtime, and OKD. The ephemeral storage framework allows pods to specify their transient local storage needs, and OKD to schedule pods where appropriate and protect the node against excessive use of local storage.

While the ephemeral storage framework allows administrators and developers to better manage this local storage, it does not provide any promises related to I/O throughput and latency.

Types of ephemeral storage

Ephemeral local storage is always made available in the primary partition. There are two basic ways of creating the primary partition, root and runtime.

Root

This partition holds the kubelet’s root directory, /var/lib/origin/ by default, and /var/log/ directory. This partition may be shared between user pods, OS, and Kubernetes system daemons. This partition can be consumed by pods via EmptyDir volumes, container logs, image layers, and container writable layers. Kubelet manages shared access and isolation of this partition. This partition is ephemeral, and applications cannot expect any performance SLAs, disk IOPS for example, from this partition.

Runtime

This is an optional partition that runtimes can use for overlay file systems. OKD attempts to identify and provide shared access along with isolation to this partition. Container image layers and writable layers are stored here. If the runtime partition exists, the root partition does not hold any image layer or other writable storage.

Managing ephemeral storage

Cluster administrators can manage ephemeral storage within a project by setting quotas that define the limit ranges and number of requests for ephemeral storage across all Pods in a non-terminal state. Developers can also set requests and limits on this compute resource at the Pod and Container level.

Monitoring ephemeral storage

You can use /bin/df as a tool to monitor ephemeral storage usage on the volume where ephemeral container data is located, which is /var/lib/origin and /var/lib/docker. The available space for only /var/lib/origin is shown when you use the df command if /var/lib/docker is placed on a separate disk by the cluster administrator.

Use the df -h command to show the human-readable values of used and available space in /var/lib:

$ df -h /var/lib

The output shows the ephemeral storage usage in /var/lib:

Filesystem  Size  Used Avail Use% Mounted on
/dev/sda1    69G   32G   34G  49% /