Garbage Collection

Garbage Collection

Overview

The OKD node performs two types of garbage collection:

Container garbage collection: Removes terminated containers. Enabled by default.
Image garbage collection: Removes images not referenced by any running pods. Not enabled by default.

Container Garbage Collection

Container garbage collection is enabled by default and happens automatically in response to eviction thresholds being reached. The node tries to keep any container for any pod accessible from the API. If the pod has been deleted, the containers will be as well. Containers are preserved as long the pod is not deleted and the eviction threshold is not reached. If the node is under disk pressure, it will remove containers and their logs will no longer be accessible via oc logs.

The policy for container garbage collection is based on three node settings:

Setting	Description
`minimum-container-ttl-duration`	The minimum age that a container is eligible for garbage collection. The default is 0. Use 0 for no limit. Values for this setting can be specified using unit suffixes such as h for hour, m for minutes, s for seconds.
`maximum-dead-containers-per-container`	The number of instances to retain per pod container. The default is 1.
`maximum-dead-containers`	The maximum number of total dead containers in the node. The default is -1, which means unlimited.

The maximum-dead-containers setting takes precedence over the maximum-dead-containers-per-container setting when there is a conflict. For example, if retaining the number of maximum-dead-containers-per-container would result in a total number of containers that is greater than maximum-dead-containers, the oldest containers will be removed to satisfy the maximum-dead-containers limit.

When the node removes the dead containers, all files inside those containers are removed as well. Only containers created by the node will be garbage collected.

If you do not want to use the default settings, you can specify values for these settings in the **kubeletArguments** section of the appropriate node configuration map. Add the section if it does not already exist.

Container garbage collection is performed using default values if these parameters are not present in the node configuration map.

Container Garbage Collection Settings

kubeletArguments:
  minimum-container-ttl-duration:
    - "10s"
  maximum-dead-containers-per-container:
    - "2"
  maximum-dead-containers:
    - "240"

Detecting Containers for Deletion

Currently, Docker and rkt are supported. The following only applies to Docker; rkt has its own garbage collection.

Each spin of the garbage collector loop goes through the following steps:

Retrieve a list of available containers.
Filter out all containers that are running or are not alive longer than the **minimum-container-ttl-duration** parameter.
Classify all remaining containers into equivalence classes based on pod and image name membership.
Remove all unidentified containers (containers that are managed by kubelet but their name is malformed).
For each class that contains more containers than the **maximum-dead-containers-per-container** parameter, sort containers in the class by creation time.
Start removing containers from the oldest first until the **maximum-dead-containers-per-container** parameter is met.
If there are still more containers in the list than the **maximum-dead-containers** parameter, the collector starts removing containers from each class so the number of containers in each one is not greater than the average number of containers per class, or <all_remaining_containers>/<number_of_classes>.
If this is still not enough, sort all containers in the list and start removing containers from the oldest first until the **maximum-dead-containers** criterion is met.

Update the default settings to meet your needs.

Garbage collection only removes the containers that do not have a pod associated with it.

Image Garbage Collection

Image garbage collection relies on disk usage as reported by cAdvisor on the node to decide which images to remove from the node. It takes the following settings into consideration:

Setting	Description
`image-gc-high-threshold`	The percent of disk usage (expressed as an integer) which triggers image garbage collection.
`image-gc-low-threshold`	The percent of disk usage (expressed as an integer) to which image garbage collection attempts to free.

To enable image garbage collection, specify values for these settings in the **kubeletArguments** section of the appropriate node configuration map. Add the section if it does not already exist.

Image garbage collection is performed using default values if these parameters are not present in the node configuration map.

Image Garbage Collection Settings

kubeletArguments:
  image-gc-high-threshold:
    - "85"
  image-gc-low-threshold:
    - "80"

Detecting Images for Deletion

Two lists of images are retrieved in each garbage collector run:

A list of images currently running in at least one pod
A list of images available on a host

As new containers are run, new images appear. All images are marked with a time stamp. If the image is running (the first list above) or is newly detected (the second list above), it is marked with the current time. The remaining images are already marked from the previous spins. All images are then sorted by the time stamp.

Once the collection starts, the oldest images get deleted first until the stopping criterion is met.