Resource quotas per project

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per project. It can limit the quantity of objects that can be created in a project by type, as well as the total amount of compute resources and storage that might be consumed by resources in that project.

This guide describes how resource quotas work, how cluster administrators can set and manage resource quotas on a per project basis, and how developers and cluster administrators can view them.

Resources managed by quotas

The following describes the set of compute resources and object types that can be managed by a quota.

Quota scopes

Each quota can have an associated set of scopes. A quota only measures usage for a resource if it matches the intersection of enumerated scopes.

Adding a scope to a quota restricts the set of resources to which that quota can apply. Specifying a resource outside of the allowed set results in a validation error.

A BestEffort scope restricts a quota to limiting the following resources:

• pods

A Terminating, NotTerminating, and NotBestEffort scope restricts a quota to tracking the following resources:

• pods

• memory

• requests.memory

• limits.memory

• cpu

• requests.cpu

• limits.cpu

• ephemeral-storage

• requests.ephemeral-storage

• limits.ephemeral-storage

Quota enforcement

After a resource quota for a project is first created, the project restricts the ability to create any new resources that may violate a quota constraint until it has calculated updated usage statistics.

After a quota is created and usage statistics are updated, the project accepts the creation of new content. When you create or modify resources, your quota usage is incremented immediately upon the request to create or modify the resource.

When you delete a resource, your quota use is decremented during the next full recalculation of quota statistics for the project. A configurable amount of time determines how long it takes to reduce quota usage statistics to their current observed system value.

If project modifications exceed a quota usage limit, the server denies the action, and an appropriate error message is returned to the user explaining the quota constraint violated, and what their currently observed usage statistics are in the system.

Requests versus limits

When allocating compute resources, each container might specify a request and a limit value each for CPU, memory, and ephemeral storage. Quotas can restrict any of these values.

If the quota has a value specified for requests.cpu or requests.memory, then it requires that every incoming container make an explicit request for those resources. If the quota has a value specified for limits.cpu or limits.memory, then it requires that every incoming container specify an explicit limit for those resources.

Sample resource quota definitions

core-object-counts.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: core-object-counts
spec:
  hard:
    configmaps: "10" (1)
    persistentvolumeclaims: "4" (2)
    replicationcontrollers: "20" (3)
    secrets: "10" (4)
    services: "10" (5)
    services.loadbalancers: "2" (6)

openshift-object-counts.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: openshift-object-counts
spec:
  hard:
    openshift.io/imagestreams: "10" (1)

compute-resources.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    pods: "4" (1)
    requests.cpu: "1" (2)
    requests.memory: 1Gi (3)
    requests.ephemeral-storage: 2Gi (4)
    limits.cpu: "2" (5)
    limits.memory: 2Gi (6)
    limits.ephemeral-storage: 4Gi (7)

besteffort.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: besteffort
spec:
  hard:
    pods: "1" (1)
  scopes:
  - BestEffort (2)

compute-resources-long-running.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources-long-running
spec:
  hard:
    pods: "4" (1)
    limits.cpu: "4" (2)
    limits.memory: "2Gi" (3)
    limits.ephemeral-storage: "4Gi" (4)
  scopes:
  - NotTerminating (5)

compute-resources-time-bound.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources-time-bound
spec:
  hard:
    pods: "2" (1)
    limits.cpu: "1" (2)
    limits.memory: "1Gi" (3)
    limits.ephemeral-storage: "1Gi" (4)
  scopes:
  - Terminating (5)

storage-consumption.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-consumption
spec:
  hard:
    persistentvolumeclaims: "10" (1)
    requests.storage: "50Gi" (2)
    gold.storageclass.storage.k8s.io/requests.storage: "10Gi" (3)
    silver.storageclass.storage.k8s.io/requests.storage: "20Gi" (4)
    silver.storageclass.storage.k8s.io/persistentvolumeclaims: "5" (5)
    bronze.storageclass.storage.k8s.io/requests.storage: "0" (6)
    bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "0" (7)

Creating a quota

You can create a quota to constrain resource usage in a given project.

Procedure

1. Define the quota in a file.

2. Use the file to create the quota and apply it to a project:

   $ oc create -f <file> [-n <project_name>]

   For example:

   $ oc create -f core-object-counts.yaml -n demoproject

Creating object count quotas

You can create an object count quota for all standard namespaced resource types on OKD, such as BuildConfig and DeploymentConfig objects. An object quota count places a defined quota on all standard namespaced resource types.

When using a resource quota, an object is charged against the quota upon creation. These types of quotas are useful to protect against exhaustion of resources. The quota can only be created if there are enough spare resources within the project.

Procedure

To configure an object count quota for a resource:

1. Run the following command:

   $ oc create quota <name> \
       --hard=count/<resource>.<group>=<quota>,count/<resource>.<group>=<quota> (1)

   1	The <resource> variable is the name of the resource, and <group> is the API group, if applicable. Use the oc api-resources command for a list of resources and their associated API groups.

   For example:

   $ oc create quota test \
       --hard=count/deployments.extensions=2,count/replicasets.extensions=4,count/pods=3,count/secrets=4

   Example output

   resourcequota "test" created

   This example limits the listed resources to the hard limit in each project in the cluster.

2. Verify that the quota was created:

   $ oc describe quota test

   Example output

   Name:                         test
   Namespace:                    quota
   Resource                      Used  Hard
   --------                      ----  ----
   count/deployments.extensions  0     2
   count/pods                    0     3
   count/replicasets.extensions  0     4
   count/secrets                 0     4

Setting resource quota for extended resources

Overcommitment of resources is not allowed for extended resources, so you must specify requests and limits for the same extended resource in a quota. Currently, only quota items with the prefix requests. is allowed for extended resources. The following is an example scenario of how to set resource quota for the GPU resource nvidia.com/gpu.

Procedure

1. Determine how many GPUs are available on a node in your cluster. For example:

   # oc describe node ip-172-31-27-209.us-west-2.compute.internal | egrep 'Capacity|Allocatable|gpu'

   Example output

                       openshift.com/gpu-accelerator=true
   Capacity:
    nvidia.com/gpu:  2
   Allocatable:
    nvidia.com/gpu:  2
     nvidia.com/gpu  0           0

   In this example, 2 GPUs are available.

2. Set a quota in the namespace nvidia. In this example, the quota is 1:

   # cat gpu-quota.yaml

   Example output

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: gpu-quota
     namespace: nvidia
   spec:
     hard:
       requests.nvidia.com/gpu: 1

3. Create the quota:

   # oc create -f gpu-quota.yaml

   Example output

   resourcequota/gpu-quota created

4. Verify that the namespace has the correct quota set:

   # oc describe quota gpu-quota -n nvidia

   Example output

   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  0     1

5. Define a pod that asks for a single GPU. The following example definition file is called gpu-pod.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     generateName: gpu-pod-
     namespace: nvidia
   spec:
     restartPolicy: OnFailure
     containers:
     - name: rhel7-gpu-pod
       image: rhel7
       env:
         - name: NVIDIA_VISIBLE_DEVICES
           value: all
         - name: NVIDIA_DRIVER_CAPABILITIES
           value: "compute,utility"
         - name: NVIDIA_REQUIRE_CUDA
           value: "cuda>=5.0"
       command: ["sleep"]
       args: ["infinity"]
       resources:
         limits:
           nvidia.com/gpu: 1

6. Create the pod:

   # oc create -f gpu-pod.yaml

7. Verify that the pod is running:

   # oc get pods

   Example output

   NAME              READY     STATUS      RESTARTS   AGE
   gpu-pod-s46h7     1/1       Running     0          1m

8. Verify that the quota Used counter is correct:

   # oc describe quota gpu-quota -n nvidia

   Example output

   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  1     1

9. Attempt to create a second GPU pod in the nvidia namespace. This is technically available on the node because it has 2 GPUs:

   # oc create -f gpu-pod.yaml

   Example output

   Error from server (Forbidden): error when creating "gpu-pod.yaml": pods "gpu-pod-f7z2w" is forbidden: exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=1, limited: requests.nvidia.com/gpu=1

   This Forbidden error message is expected because you have a quota of 1 GPU and this pod tried to allocate a second GPU, which exceeds its quota.

Viewing a quota

You can view usage statistics related to any hard limits defined in a project’s quota by navigating in the web console to the project’s Quota page.

You can also use the CLI to view quota details.

Procedure

1. Get the list of quotas defined in the project. For example, for a project called demoproject:

   $ oc get quota -n demoproject

   Example output

   NAME                AGE
   besteffort          11m
   compute-resources   2m
   core-object-counts  29m

2. Describe the quota you are interested in, for example the core-object-counts quota:

   $ oc describe quota core-object-counts -n demoproject

   Example output

   Name:            core-object-counts
   Namespace:        demoproject
   Resource        Used    Hard
   --------        ----    ----
   configmaps        3    10
   persistentvolumeclaims    0    4
   replicationcontrollers    3    20
   secrets            9    10
   services        2    10

Configuring explicit resource quotas

Configure explicit resource quotas in a project request template to apply specific resource quotas in new projects.

Prerequisites

• Access to the cluster as a user with the cluster-admin role.

• Install the OpenShift CLI (oc).

Procedure

1. Add a resource quota definition to a project request template:

   • If a project request template does not exist in a cluster:

     1. Create a bootstrap project template and output it to a file called template.yaml:

        $ oc adm create-bootstrap-project-template -o yaml > template.yaml

     2. Add a resource quota definition to template.yaml. The following example defines a resource quota named ‘storage-consumption’. The definition must be added before the parameters: section in the template:

        - apiVersion: v1
          kind: ResourceQuota
          metadata:
            name: storage-consumption
          spec:
            hard:
              persistentvolumeclaims: "10" (1)
              requests.storage: "50Gi" (2)
              gold.storageclass.storage.k8s.io/requests.storage: "10Gi" (3)
              silver.storageclass.storage.k8s.io/requests.storage: "20Gi" (4)
              silver.storageclass.storage.k8s.io/persistentvolumeclaims: "5" (5)
              bronze.storageclass.storage.k8s.io/requests.storage: "0" (6)
              bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "0" (7)

        1	The total number of persistent volume claims in a project.
        2	Across all persistent volume claims in a project, the sum of storage requested cannot exceed this value.
        3	Across all persistent volume claims in a project, the sum of storage requested in the gold storage class cannot exceed this value.
        4	Across all persistent volume claims in a project, the sum of storage requested in the silver storage class cannot exceed this value.
        5	Across all persistent volume claims in a project, the total number of claims in the silver storage class cannot exceed this value.
        6	Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this value is set to 0, the bronze storage class cannot request storage.
        7	Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this value is set to 0, the bronze storage class cannot create claims.

     3. Create a project request template from the modified template.yaml file in the openshift-config namespace:

        $ oc create -f template.yaml -n openshift-config

        To include the configuration as a kubectl.kubernetes.io/last-applied-configuration annotation, add the —save-config option to the oc create command.

        By default, the template is called project-request.

   • If a project request template already exists within a cluster:

     If you declaratively or imperatively manage objects within your cluster by using configuration files, edit the existing project request template through those files instead.

     1. List templates in the openshift-config namespace:

        $ oc get templates -n openshift-config

     2. Edit an existing project request template:

        $ oc edit template <project_request_template> -n openshift-config

     3. Add a resource quota definition, such as the preceding storage-consumption example, into the existing template. The definition must be added before the parameters: section in the template.

1. If you created a project request template, reference it in the cluster’s project configuration resource:

   1. Access the project configuration resource for editing:

      • By using the web console:

        1. Navigate to the Administration → Cluster Settings page.

        2. Click Global Configuration to view all configuration resources.

        3. Find the entry for Project and click Edit YAML.

      • By using the CLI:

        1. Edit the project.config.openshift.io/cluster resource:

           $ oc edit project.config.openshift.io/cluster

2.  Update the `spec` section of the project configuration resource to include the `projectRequestTemplate` and `name` parameters. The following example references the default project request template name `project-request`:
    ```
    apiVersion: config.openshift.io/v1
    kind: Project
    metadata:
      ...
    spec:
      projectRequestTemplate:
        name: project-request
    ```

1. Verify that the resource quota is applied when projects are created:

   1. Create a project:

      $ oc new-project <project_name>

   2. List the project’s resource quotas:

      $ oc get resourcequotas

   3. Describe the resource quota in detail:

      $ oc describe resourcequotas <resource_quota_name>