Extending the Kubernetes API with custom resource definitions

This guide describes how cluster administrators can extend their OKD cluster by creating and managing custom resource definitions (CRDs).

Custom resource definitions

In the Kubernetes API, a resource is an endpoint that stores a collection of API objects of a certain kind. For example, the built-in Pods resource contains a collection of Pod objects.

A custom resource definition (CRD) object defines a new, unique object type, called a kind, in the cluster and lets the Kubernetes API server handle its entire lifecycle.

Custom resource (CR) objects are created from CRDs that have been added to the cluster by a cluster administrator, allowing all cluster users to add the new resource type into projects.

When a cluster administrator adds a new CRD to the cluster, the Kubernetes API server reacts by creating a new RESTful resource path that can be accessed by the entire cluster or a single project (namespace) and begins serving the specified CR.

Cluster administrators that want to grant access to the CRD to other users can use cluster role aggregation to grant access to users with the admin, edit, or view default cluster roles. Cluster role aggregation allows the insertion of custom policy rules into these cluster roles. This behavior integrates the new resource into the RBAC policy of the cluster as if it was a built-in resource.

Operators in particular make use of CRDs by packaging them with any required RBAC policy and other software-specific logic. Cluster administrators can also add CRDs manually to the cluster outside of the lifecycle of an Operator, making them available to all users.

While only cluster administrators can create CRDs, developers can create the CR from an existing CRD if they have read and write permission to it.

Creating a custom resource definition

To create custom resource (CR) objects, cluster administrators must first create a custom resource definition (CRD).

Prerequisites

  • Access to an OKD cluster with cluster-admin user privileges.

Procedure

To create a CRD:

  1. Create a YAML file that contains the following field types:

    Example YAML file for a CRD

    1. apiVersion: apiextensions.k8s.io/v1 (1)
    2. kind: CustomResourceDefinition
    3. metadata:
    4.   name: crontabs.stable.example.com (2)
    5. spec:
    6.   group: stable.example.com (3)
    7.   version: v1 (4)
    8.   scope: Namespaced (5)
    9.   names:
    10.     plural: crontabs (6)
    11.     singular: crontab (7)
    12.     kind: CronTab (8)
    13.     shortNames:
    14.     - ct (9)
    1Use the apiextensions.k8s.io/v1 API.
    2Specify a name for the definition. This must be in the <plural-name>.<group> format using the values from the group and plural fields.
    3Specify a group name for the API. An API group is a collection of objects that are logically related. For example, all batch objects like Job or ScheduledJob could be in the batch API group (such as batch.api.example.com). A good practice is to use a fully-qualified-domain name (FQDN) of your organization.
    4Specify a version name to be used in the URL. Each API group can exist in multiple versions, for example v1alpha, v1beta, v1.
    5Specify whether the custom objects are available to a project (Namespaced) or all projects in the cluster (Cluster).
    6Specify the plural name to use in the URL. The plural field is the same as a resource in an API URL.
    7Specify a singular name to use as an alias on the CLI and for display.
    8Specify the kind of objects that can be created. The type can be in CamelCase.
    9Specify a shorter string to match your resource on the CLI.

    By default, a CRD is cluster-scoped and available to all projects.

  2. Create the CRD object:

    1. $ oc create -f <file_name>.yaml

    A new RESTful API endpoint is created at:

    1. /apis/<spec:group>/<spec:version>/<scope>/*/<names-plural>/...

    For example, using the example file, the following endpoint is created:

    1. /apis/stable.example.com/v1/namespaces/*/crontabs/...

    You can now use this endpoint URL to create and manage CRs. The object kind is based on the spec.kind field of the CRD object you created.

Creating cluster roles for custom resource definitions

Cluster administrators can grant permissions to existing cluster-scoped custom resource definitions (CRDs). If you use the admin, edit, and view default cluster roles, you can take advantage of cluster role aggregation for their rules.

You must explicitly assign permissions to each of these roles. The roles with more permissions do not inherit rules from roles with fewer permissions. If you assign a rule to a role, you must also assign that verb to roles that have more permissions. For example, if you grant the get crontabs permission to the view role, you must also grant it to the edit and admin roles. The admin or edit role is usually assigned to the user that created a project through the project template.

Prerequisites

  • Create a CRD.

Procedure

  1. Create a cluster role definition file for the CRD. The cluster role definition is a YAML file that contains the rules that apply to each cluster role. A OKD controller adds the rules that you specify to the default cluster roles.

    Example YAML file for a cluster role definition

    1. kind: ClusterRole
    2. apiVersion: rbac.authorization.k8s.io/v1 (1)
    3. metadata:
    4.   name: aggregate-cron-tabs-admin-edit (2)
    5.   labels:
    6.     rbac.authorization.k8s.io/aggregate-to-admin: "true" (3)
    7.     rbac.authorization.k8s.io/aggregate-to-edit: "true" (4)
    8. rules:
    9. - apiGroups: ["stable.example.com"] (5)
    10.   resources: ["crontabs"] (6)
    11.   verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"] (7)
    12. ---
    13. kind: ClusterRole
    14. apiVersion: rbac.authorization.k8s.io/v1
    15. metadata:
    16.   name: aggregate-cron-tabs-view (2)
    17.   labels:
    18.     # Add these permissions to the "view" default role.
    19.     rbac.authorization.k8s.io/aggregate-to-view: "true" (8)
    20.     rbac.authorization.k8s.io/aggregate-to-cluster-reader: "true" (9)
    21. rules:
    22. - apiGroups: ["stable.example.com"] (5)
    23.   resources: ["crontabs"] (6)
    24.   verbs: ["get", "list", "watch"] (7)
    1Use the rbac.authorization.k8s.io/v1 API.
    2Specify a name for the definition.
    3Specify this label to grant permissions to the admin default role.
    4Specify this label to grant permissions to the edit default role.
    5Specify the group name of the CRD.
    6Specify the plural name of the CRD that these rules apply to.
    7Specify the verbs that represent the permissions that are granted to the role. For example, apply read and write permissions to the admin and edit roles and only read permission to the view role.
    8Specify this label to grant permissions to the view default role.
    9Specify this label to grant permissions to the cluster-reader default role.

  2. Create the cluster role:

    1. $ oc create -f <file_name>.yaml

Creating custom resources from a file

After a custom resource definitions (CRD) has been added to the cluster, custom resources (CRs) can be created with the CLI from a file using the CR specification.

Prerequisites

  • CRD added to the cluster by a cluster administrator.

Procedure

  1. Create a YAML file for the CR. In the following example definition, the cronSpec and image custom fields are set in a CR of Kind: CronTab. The Kind comes from the spec.kind field of the CRD object:

    Example YAML file for a CR

    1. apiVersion: "stable.example.com/v1" (1)
    2. kind: CronTab (2)
    3. metadata:
    4.   name: my-new-cron-object (3)
    5.   finalizers: (4)
    6.   - finalizer.stable.example.com
    7. spec: (5)
    8.   cronSpec: "* * * * /5"
    9.   image: my-awesome-cron-image
    1Specify the group name and API version (name/version) from the CRD.
    2Specify the type in the CRD.
    3Specify a name for the object.
    4Specify the finalizers for the object, if any. Finalizers allow controllers to implement conditions that must be completed before the object can be deleted.
    5Specify conditions specific to the type of object.

  2. After you create the file, create the object:

    1. $ oc create -f <file_name>.yaml

Inspecting custom resources

You can inspect custom resource (CR) objects that exist in your cluster using the CLI.

Prerequisites

  • A CR object exists in a namespace to which you have access.

Procedure

  1. To get information on a specific kind of a CR, run:

    1. $ oc get <kind>

    For example:

    1. $ oc get crontab

    Example output

    1. NAME                 KIND
    2. my-new-cron-object   CronTab.v1.stable.example.com

    Resource names are not case-sensitive, and you can use either the singular or plural forms defined in the CRD, as well as any short name. For example:

    1. $ oc get crontabs
    1. $ oc get crontab
    1. $ oc get ct

  2. You can also view the raw YAML data for a CR:

    1. $ oc get <kind> -o yaml

    For example:

    1. $ oc get ct -o yaml

    Example output

    1. apiVersion: v1
    2. items:
    3. - apiVersion: stable.example.com/v1
    4.   kind: CronTab
    5.   metadata:
    6.     clusterName: ""
    7.     creationTimestamp: 2017-05-31T12:56:35Z
    8.     deletionGracePeriodSeconds: null
    9.     deletionTimestamp: null
    10.     name: my-new-cron-object
    11.     namespace: default
    12.     resourceVersion: "285"
    13.     selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
    14.     uid: 9423255b-4600-11e7-af6a-28d2447dc82b
    15.   spec:
    16.     cronSpec: '* * * * /5' (1)
    17.     image: my-awesome-cron-image (1)
    1Custom data from the YAML that you used to create the object displays.