Versions in CustomResourceDefinitions

This page explains how to add versioning information to CustomResourceDefinitions, to indicate the stability level of your CustomResourceDefinitions or advance your API to a new version with conversion between API representations. It also describes how to upgrade an object from one version to another.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

You should have an initial understanding of custom resources.

Your Kubernetes server must be at or later than version v1.16. To check the version, enter kubectl version.

Overview

The CustomResourceDefinition API provides a workflow for introducing and upgrading to new versions of a CustomResourceDefinition.

When a CustomResourceDefinition is created, the first version is set in the CustomResourceDefinition spec.versions list to an appropriate stability level and a version number. For example v1beta1 would indicate that the first version is not yet stable. All custom resource objects will initially be stored at this version.

Once the CustomResourceDefinition is created, clients may begin using the v1beta1 API.

Later it might be necessary to add new version such as v1.

Adding a new version:

  1. Pick a conversion strategy. Since custom resource objects need the ability to be served at both versions, that means they will sometimes be served in a different version than the one stored. To make this possible, the custom resource objects must sometimes be converted between the version they are stored at and the version they are served at. If the conversion involves schema changes and requires custom logic, a conversion webhook should be used. If there are no schema changes, the default None conversion strategy may be used and only the apiVersion field will be modified when serving different versions.
  2. If using conversion webhooks, create and deploy the conversion webhook. See the Webhook conversion for more details.
  3. Update the CustomResourceDefinition to include the new version in the spec.versions list with served:true. Also, set spec.conversion field to the selected conversion strategy. If using a conversion webhook, configure spec.conversion.webhookClientConfig field to call the webhook.

Once the new version is added, clients may incrementally migrate to the new version. It is perfectly safe for some clients to use the old version while others use the new version.

Migrate stored objects to the new version:

  1. See the upgrade existing objects to a new stored version section.

It is safe for clients to use both the old and new version before, during and after upgrading the objects to a new stored version.

Removing an old version:

  1. Ensure all clients are fully migrated to the new version. The kube-apiserver logs can be reviewed to help identify any clients that are still accessing via the old version.
  2. Set served to false for the old version in the spec.versions list. If any clients are still unexpectedly using the old version they may begin reporting errors attempting to access the custom resource objects at the old version. If this occurs, switch back to using served:true on the old version, migrate the remaining clients to the new version and repeat this step.
  3. Ensure the upgrade of existing objects to the new stored version step has been completed.
    1. Verify that the storage is set to true for the new version in the spec.versions list in the CustomResourceDefinition.
    2. Verify that the old version is no longer listed in the CustomResourceDefinition status.storedVersions.
  4. Remove the old version from the CustomResourceDefinition spec.versions list.
  5. Drop conversion support for the old version in conversion webhooks.

Specify multiple versions

The CustomResourceDefinition API versions field can be used to support multiple versions of custom resources that you have developed. Versions can have different schemas, and conversion webhooks can convert custom resources between versions. Webhook conversions should follow the Kubernetes API conventions wherever applicable. Specifically, See the API change documentation for a set of useful gotchas and suggestions.

Note: In apiextensions.k8s.io/v1beta1, there was a version field instead of versions. The version field is deprecated and optional, but if it is not empty, it must match the first item in the versions field.

This example shows a CustomResourceDefinition with two versions. For the first example, the assumption is all versions share the same schema with no conversion between them. The comments in the YAML provide more context.

  1. apiVersion: apiextensions.k8s.io/v1
  2. kind: CustomResourceDefinition
  3. metadata:
  4.   # name must match the spec fields below, and be in the form: <plural>.<group>
  5.   name: crontabs.example.com
  6. spec:
  7.   # group name to use for REST API: /apis/<group>/<version>
  8.   group: example.com
  9.   # list of versions supported by this CustomResourceDefinition
  10.   versions:
  11.   - name: v1beta1
  12.     # Each version can be enabled/disabled by Served flag.
  13.     served: true
  14.     # One and only one version must be marked as the storage version.
  15.     storage: true
  16.     # A schema is required
  17.     schema:
  18.       openAPIV3Schema:
  19.         type: object
  20.         properties:
  21.           host:
  22.             type: string
  23.           port:
  24.             type: string
  25.   - name: v1
  26.     served: true
  27.     storage: false
  28.     schema:
  29.       openAPIV3Schema:
  30.         type: object
  31.         properties:
  32.           host:
  33.             type: string
  34.           port:
  35.             type: string
  36.   # The conversion section is introduced in Kubernetes 1.13+ with a default value of
  37.   # None conversion (strategy sub-field set to None).
  38.   conversion:
  39.     # None conversion assumes the same schema for all versions and only sets the apiVersion
  40.     # field of custom resources to the proper value
  41.     strategy: None
  42.   # either Namespaced or Cluster
  43.   scope: Namespaced
  44.   names:
  45.     # plural name to be used in the URL: /apis/<group>/<version>/<plural>
  46.     plural: crontabs
  47.     # singular name to be used as an alias on the CLI and for display
  48.     singular: crontab
  49.     # kind is normally the CamelCased singular type. Your resource manifests use this.
  50.     kind: CronTab
  51.     # shortNames allow shorter string to match your resource on the CLI
  52.     shortNames:
  53.     - ct
  1. # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   # name must match the spec fields below, and be in the form: <plural>.<group>
  6.   name: crontabs.example.com
  7. spec:
  8.   # group name to use for REST API: /apis/<group>/<version>
  9.   group: example.com
  10.   # list of versions supported by this CustomResourceDefinition
  11.   versions:
  12.   - name: v1beta1
  13.     # Each version can be enabled/disabled by Served flag.
  14.     served: true
  15.     # One and only one version must be marked as the storage version.
  16.     storage: true
  17.   - name: v1
  18.     served: true
  19.     storage: false
  20.   validation:
  21.     openAPIV3Schema:
  22.       type: object
  23.       properties:
  24.         host:
  25.           type: string
  26.         port:
  27.           type: string
  28.   # The conversion section is introduced in Kubernetes 1.13+ with a default value of
  29.   # None conversion (strategy sub-field set to None).
  30.   conversion:
  31.     # None conversion assumes the same schema for all versions and only sets the apiVersion
  32.     # field of custom resources to the proper value
  33.     strategy: None
  34.   # either Namespaced or Cluster
  35.   scope: Namespaced
  36.   names:
  37.     # plural name to be used in the URL: /apis/<group>/<version>/<plural>
  38.     plural: crontabs
  39.     # singular name to be used as an alias on the CLI and for display
  40.     singular: crontab
  41.     # kind is normally the PascalCased singular type. Your resource manifests use this.
  42.     kind: CronTab
  43.     # shortNames allow shorter string to match your resource on the CLI
  44.     shortNames:
  45.     - ct

You can save the CustomResourceDefinition in a YAML file, then use kubectl apply to create it.

  1. kubectl apply -f my-versioned-crontab.yaml

After creation, the API server starts to serve each enabled version at an HTTP REST endpoint. In the above example, the API versions are available at /apis/example.com/v1beta1 and /apis/example.com/v1.

Version priority

Regardless of the order in which versions are defined in a CustomResourceDefinition, the version with the highest priority is used by kubectl as the default version to access objects. The priority is determined by parsing the name field to determine the version number, the stability (GA, Beta, or Alpha), and the sequence within that stability level.

The algorithm used for sorting the versions is designed to sort versions in the same way that the Kubernetes project sorts Kubernetes versions. Versions start with a v followed by a number, an optional beta or alpha designation, and optional additional numeric versioning information. Broadly, a version string might look like v2 or v2beta1. Versions are sorted using the following algorithm:

  • Entries that follow Kubernetes version patterns are sorted before those that do not.
  • For entries that follow Kubernetes version patterns, the numeric portions of the version string is sorted largest to smallest.
  • If the strings beta or alpha follow the first numeric portion, they sorted in that order, after the equivalent string without the beta or alpha suffix (which is presumed to be the GA version).
  • If another number follows the beta, or alpha, those numbers are also sorted from largest to smallest.
  • Strings that don’t fit the above format are sorted alphabetically and the numeric portions are not treated specially. Notice that in the example below, foo1 is sorted above foo10. This is different from the sorting of the numeric portion of entries that do follow the Kubernetes version patterns.

This might make sense if you look at the following sorted version list:

  1. - v10
  2. - v2
  3. - v1
  4. - v11beta2
  5. - v10beta3
  6. - v3beta1
  7. - v12alpha1
  8. - v11alpha2
  9. - foo1
  10. - foo10

For the example in Specify multiple versions, the version sort order is v1, followed by v1beta1. This causes the kubectl command to use v1 as the default version unless the provided object specifies the version.

Version deprecation

FEATURE STATE: Kubernetes v1.19 [stable]

Starting in v1.19, a CustomResourceDefinition can indicate a particular version of the resource it defines is deprecated. When API requests to a deprecated version of that resource are made, a warning message is returned in the API response as a header. The warning message for each deprecated version of the resource can be customized if desired.

A customized warning message should indicate the deprecated API group, version, and kind, and should indicate what API group, version, and kind should be used instead, if applicable.

  1. apiVersion: apiextensions.k8s.io/v1
  2. kind: CustomResourceDefinition
  3.   name: crontabs.example.com
  4. spec:
  5.   group: example.com
  6.   names:
  7.     plural: crontabs
  8.     singular: crontab
  9.     kind: CronTab
  10.   scope: Namespaced
  11.   versions:
  12.   - name: v1alpha1
  13.     served: true
  14.     storage: false
  15.     # This indicates the v1alpha1 version of the custom resource is deprecated.
  16.     # API requests to this version receive a warning header in the server response.
  17.     deprecated: true
  18.     # This overrides the default warning returned to API clients making v1alpha1 API requests.
  19.     deprecationWarning: "example.com/v1alpha1 CronTab is deprecated; see http://example.com/v1alpha1-v1 for instructions to migrate to example.com/v1 CronTab"
  21.     schema: ...
  22.   - name: v1beta1
  23.     served: true
  24.     # This indicates the v1beta1 version of the custom resource is deprecated.
  25.     # API requests to this version receive a warning header in the server response.
  26.     # A default warning message is returned for this version.
  27.     deprecated: true
  28.     schema: ...
  29.   - name: v1
  30.     served: true
  31.     storage: true
  32.     schema: ...
  1. # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   name: crontabs.example.com
  6. spec:
  7.   group: example.com
  8.   names:
  9.     plural: crontabs
  10.     singular: crontab
  11.     kind: CronTab
  12.   scope: Namespaced
  13.   validation: ...
  14.   versions:
  15.   - name: v1alpha1
  16.     served: true
  17.     storage: false
  18.     # This indicates the v1alpha1 version of the custom resource is deprecated.
  19.     # API requests to this version receive a warning header in the server response.
  20.     deprecated: true
  21.     # This overrides the default warning returned to API clients making v1alpha1 API requests.
  22.     deprecationWarning: "example.com/v1alpha1 CronTab is deprecated; see http://example.com/v1alpha1-v1 for instructions to migrate to example.com/v1 CronTab"
  23.   - name: v1beta1
  24.     served: true
  25.     # This indicates the v1beta1 version of the custom resource is deprecated.
  26.     # API requests to this version receive a warning header in the server response.
  27.     # A default warning message is returned for this version.
  28.     deprecated: true
  29.   - name: v1
  30.     served: true
  31.     storage: true

Version removal

An older API version cannot be dropped from a CustomResourceDefinition manifest until existing stored data has been migrated to the newer API version for all clusters that served the older version of the custom resource, and the old version is removed from the status.storedVersions of the CustomResourceDefinition.

  1. apiVersion: apiextensions.k8s.io/v1
  2. kind: CustomResourceDefinition
  3.   name: crontabs.example.com
  4. spec:
  5.   group: example.com
  6.   names:
  7.     plural: crontabs
  8.     singular: crontab
  9.     kind: CronTab
  10.   scope: Namespaced
  11.   versions:
  12.   - name: v1beta1
  13.     # This indicates the v1beta1 version of the custom resource is no longer served.
  14.     # API requests to this version receive a not found error in the server response.
  15.     served: false
  16.     schema: ...
  17.   - name: v1
  18.     served: true
  19.     # The new served version should be set as the storage version
  20.     storage: true
  21.     schema: ...

Webhook conversion

FEATURE STATE: Kubernetes v1.16 [stable]

Note: Webhook conversion is available as beta since 1.15, and as alpha since Kubernetes 1.13. The CustomResourceWebhookConversion feature must be enabled, which is the case automatically for many clusters for beta features. Please refer to the feature gate documentation for more information.

The above example has a None conversion between versions which only sets the apiVersion field on conversion and does not change the rest of the object. The API server also supports webhook conversions that call an external service in case a conversion is required. For example when:

  • custom resource is requested in a different version than stored version.
  • Watch is created in one version but the changed object is stored in another version.
  • custom resource PUT request is in a different version than storage version.

To cover all of these cases and to optimize conversion by the API server, the conversion requests may contain multiple objects in order to minimize the external calls. The webhook should perform these conversions independently.

Write a conversion webhook server

Please refer to the implementation of the custom resource conversion webhook server that is validated in a Kubernetes e2e test. The webhook handles the ConversionReview requests sent by the API servers, and sends back conversion results wrapped in ConversionResponse. Note that the request contains a list of custom resources that need to be converted independently without changing the order of objects. The example server is organized in a way to be reused for other conversions. Most of the common code are located in the framework file that leaves only one function to be implemented for different conversions.

Note: The example conversion webhook server leaves the ClientAuth field empty, which defaults to NoClientCert. This means that the webhook server does not authenticate the identity of the clients, supposedly API servers. If you need mutual TLS or other ways to authenticate the clients, see how to authenticate API servers.

Permissible mutations

A conversion webhook must not mutate anything inside of metadata of the converted object other than labels and annotations. Attempted changes to name, UID and namespace are rejected and fail the request which caused the conversion. All other changes are ignored.

Deploy the conversion webhook service

Documentation for deploying the conversion webhook is the same as for the admission webhook example service. The assumption for next sections is that the conversion webhook server is deployed to a service named example-conversion-webhook-server in default namespace and serving traffic on path /crdconvert.

Note: When the webhook server is deployed into the Kubernetes cluster as a service, it has to be exposed via a service on port 443 (The server itself can have an arbitrary port but the service object should map it to port 443). The communication between the API server and the webhook service may fail if a different port is used for the service.

Configure CustomResourceDefinition to use conversion webhooks

The None conversion example can be extended to use the conversion webhook by modifying conversion section of the spec:

  1. apiVersion: apiextensions.k8s.io/v1
  2. kind: CustomResourceDefinition
  3. metadata:
  4.   # name must match the spec fields below, and be in the form: <plural>.<group>
  5.   name: crontabs.example.com
  6. spec:
  7.   # group name to use for REST API: /apis/<group>/<version>
  8.   group: example.com
  9.   # list of versions supported by this CustomResourceDefinition
  10.   versions:
  11.   - name: v1beta1
  12.     # Each version can be enabled/disabled by Served flag.
  13.     served: true
  14.     # One and only one version must be marked as the storage version.
  15.     storage: true
  16.     # Each version can define its own schema when there is no top-level
  17.     # schema is defined.
  18.     schema:
  19.       openAPIV3Schema:
  20.         type: object
  21.         properties:
  22.           hostPort:
  23.             type: string
  24.   - name: v1
  25.     served: true
  26.     storage: false
  27.     schema:
  28.       openAPIV3Schema:
  29.         type: object
  30.         properties:
  31.           host:
  32.             type: string
  33.           port:
  34.             type: string
  35.   conversion:
  36.     # a Webhook strategy instruct API server to call an external webhook for any conversion between custom resources.
  37.     strategy: Webhook
  38.     # webhook is required when strategy is `Webhook` and it configures the webhook endpoint to be called by API server.
  39.     webhook:
  40.       # conversionReviewVersions indicates what ConversionReview versions are understood/preferred by the webhook.
  41.       # The first version in the list understood by the API server is sent to the webhook.
  42.       # The webhook must respond with a ConversionReview object in the same version it received.
  43.       conversionReviewVersions: ["v1","v1beta1"]
  44.       clientConfig:
  45.         service:
  46.           namespace: default
  47.           name: example-conversion-webhook-server
  48.           path: /crdconvert
  49.         caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle>...tLS0K"
  50.   # either Namespaced or Cluster
  51.   scope: Namespaced
  52.   names:
  53.     # plural name to be used in the URL: /apis/<group>/<version>/<plural>
  54.     plural: crontabs
  55.     # singular name to be used as an alias on the CLI and for display
  56.     singular: crontab
  57.     # kind is normally the CamelCased singular type. Your resource manifests use this.
  58.     kind: CronTab
  59.     # shortNames allow shorter string to match your resource on the CLI
  60.     shortNames:
  61.     - ct
  1. # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   # name must match the spec fields below, and be in the form: <plural>.<group>
  6.   name: crontabs.example.com
  7. spec:
  8.   # group name to use for REST API: /apis/<group>/<version>
  9.   group: example.com
  10.   # prunes object fields that are not specified in OpenAPI schemas below.
  11.   preserveUnknownFields: false
  12.   # list of versions supported by this CustomResourceDefinition
  13.   versions:
  14.   - name: v1beta1
  15.     # Each version can be enabled/disabled by Served flag.
  16.     served: true
  17.     # One and only one version must be marked as the storage version.
  18.     storage: true
  19.     # Each version can define its own schema when there is no top-level
  20.     # schema is defined.
  21.     schema:
  22.       openAPIV3Schema:
  23.         type: object
  24.         properties:
  25.           hostPort:
  26.             type: string
  27.   - name: v1
  28.     served: true
  29.     storage: false
  30.     schema:
  31.       openAPIV3Schema:
  32.         type: object
  33.         properties:
  34.           host:
  35.             type: string
  36.           port:
  37.             type: string
  38.   conversion:
  39.     # a Webhook strategy instruct API server to call an external webhook for any conversion between custom resources.
  40.     strategy: Webhook
  41.     # webhookClientConfig is required when strategy is `Webhook` and it configures the webhook endpoint to be called by API server.
  42.     webhookClientConfig:
  43.       service:
  44.         namespace: default
  45.         name: example-conversion-webhook-server
  46.         path: /crdconvert
  47.       caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle>...tLS0K"
  48.   # either Namespaced or Cluster
  49.   scope: Namespaced
  50.   names:
  51.     # plural name to be used in the URL: /apis/<group>/<version>/<plural>
  52.     plural: crontabs
  53.     # singular name to be used as an alias on the CLI and for display
  54.     singular: crontab
  55.     # kind is normally the CamelCased singular type. Your resource manifests use this.
  56.     kind: CronTab
  57.     # shortNames allow shorter string to match your resource on the CLI
  58.     shortNames:
  59.     - ct

You can save the CustomResourceDefinition in a YAML file, then use kubectl apply to apply it.

  1. kubectl apply -f my-versioned-crontab-with-conversion.yaml

Make sure the conversion service is up and running before applying new changes.

Contacting the webhook

Once the API server has determined a request should be sent to a conversion webhook, it needs to know how to contact the webhook. This is specified in the webhookClientConfig stanza of the webhook configuration.

Conversion webhooks can either be called via a URL or a service reference, and can optionally include a custom CA bundle to use to verify the TLS connection.

URL

url gives the location of the webhook, in standard URL form (scheme://host:port/path).

The host should not refer to a service running in the cluster; use a service reference by specifying the service field instead. The host might be resolved via external DNS in some apiservers (i.e., kube-apiserver cannot resolve in-cluster DNS as that would be a layering violation). host may also be an IP address.

Please note that using localhost or 127.0.0.1 as a host is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installations are likely to be non-portable or not readily run in a new cluster.

The scheme must be “https”; the URL must begin with “https://“.

Attempting to use a user or basic auth (for example “user:password@”) is not allowed. Fragments (“#…”) and query parameters (“?…”) are also not allowed.

Here is an example of a conversion webhook configured to call a URL (and expects the TLS certificate to be verified using system trust roots, so does not specify a caBundle):

  1. apiVersion: apiextensions.k8s.io/v1
  2. kind: CustomResourceDefinition
  3. ...
  4. spec:
  5.   ...
  6.   conversion:
  7.     strategy: Webhook
  8.     webhook:
  9.       clientConfig:
  10.         url: "https://my-webhook.example.com:9443/my-webhook-path"
  11. ...
  1. # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. ...
  5. spec:
  6.   ...
  7.   conversion:
  8.     strategy: Webhook
  9.     webhookClientConfig:
  10.       url: "https://my-webhook.example.com:9443/my-webhook-path"
  11. ...

Service Reference

The service stanza inside webhookClientConfig is a reference to the service for a conversion webhook. If the webhook is running within the cluster, then you should use service instead of url. The service namespace and name are required. The port is optional and defaults to 443. The path is optional and defaults to “/“.

Here is an example of a webhook that is configured to call a service on port “1234” at the subpath “/my-path”, and to verify the TLS connection against the ServerName my-service-name.my-service-namespace.svc using a custom CA bundle.

  1. apiVersion: apiextensions.k8s.io/v1
  2. kind: CustomResourceDefinition
  3. ...
  4. spec:
  5.   ...
  6.   conversion:
  7.     strategy: Webhook
  8.     webhook:
  9.       clientConfig:
  10.         service:
  11.           namespace: my-service-namespace
  12.           name: my-service-name
  13.           path: /my-path
  14.           port: 1234
  15.         caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle>...tLS0K"
  16. ...
  1. # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. ...
  5. spec:
  6.   ...
  7.   conversion:
  8.     strategy: Webhook
  9.     webhookClientConfig:
  10.       service:
  11.         namespace: my-service-namespace
  12.         name: my-service-name
  13.         path: /my-path
  14.         port: 1234
  15.       caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle>...tLS0K"
  16. ...

Webhook request and response

Request

Webhooks are sent a POST request, with Content-Type: application/json, with a ConversionReview API object in the apiextensions.k8s.io API group serialized to JSON as the body.

Webhooks can specify what versions of ConversionReview objects they accept with the conversionReviewVersions field in their CustomResourceDefinition:

  1. apiVersion: apiextensions.k8s.io/v1
  2. kind: CustomResourceDefinition
  3. ...
  4. spec:
  5.   ...
  6.   conversion:
  7.     strategy: Webhook
  8.     webhook:
  9.       conversionReviewVersions: ["v1", "v1beta1"]
  10.       ...

conversionReviewVersions is a required field when creating apiextensions.k8s.io/v1 custom resource definitions. Webhooks are required to support at least one ConversionReview version understood by the current and previous API server.

  1. # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. ...
  5. spec:
  6.   ...
  7.   conversion:
  8.     strategy: Webhook
  9.     conversionReviewVersions: ["v1", "v1beta1"]
  10.     ...

If no conversionReviewVersions are specified, the default when creating apiextensions.k8s.io/v1beta1 custom resource definitions is v1beta1.

API servers send the first ConversionReview version in the conversionReviewVersions list they support. If none of the versions in the list are supported by the API server, the custom resource definition will not be allowed to be created. If an API server encounters a conversion webhook configuration that was previously created and does not support any of the ConversionReview versions the API server knows how to send, attempts to call to the webhook will fail.

This example shows the data contained in an ConversionReview object for a request to convert CronTab objects to example.com/v1:

  1. {
  2.   "apiVersion": "apiextensions.k8s.io/v1",
  3.   "kind": "ConversionReview",
  4.   "request": {
  5.     # Random uid uniquely identifying this conversion call
  6.     "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
  8.     # The API group and version the objects should be converted to
  9.     "desiredAPIVersion": "example.com/v1",
  11.     # The list of objects to convert.
  12.     # May contain one or more objects, in one or more versions.
  13.     "objects": [
  14.       {
  15.         "kind": "CronTab",
  16.         "apiVersion": "example.com/v1beta1",
  17.         "metadata": {
  18.           "creationTimestamp": "2019-09-04T14:03:02Z",
  19.           "name": "local-crontab",
  20.           "namespace": "default",
  21.           "resourceVersion": "143",
  22.           "uid": "3415a7fc-162b-4300-b5da-fd6083580d66"
  23.         },
  24.         "hostPort": "localhost:1234"
  25.       },
  26.       {
  27.         "kind": "CronTab",
  28.         "apiVersion": "example.com/v1beta1",
  29.         "metadata": {
  30.           "creationTimestamp": "2019-09-03T13:02:01Z",
  31.           "name": "remote-crontab",
  32.           "resourceVersion": "12893",
  33.           "uid": "359a83ec-b575-460d-b553-d859cedde8a0"
  34.         },
  35.         "hostPort": "example.com:2345"
  36.       }
  37.     ]
  38.   }
  39. }
  1. {
  2.   # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  3.   "apiVersion": "apiextensions.k8s.io/v1beta1",
  4.   "kind": "ConversionReview",
  5.   "request": {
  6.     # Random uid uniquely identifying this conversion call
  7.     "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
  9.     # The API group and version the objects should be converted to
  10.     "desiredAPIVersion": "example.com/v1",
  12.     # The list of objects to convert.
  13.     # May contain one or more objects, in one or more versions.
  14.     "objects": [
  15.       {
  16.         "kind": "CronTab",
  17.         "apiVersion": "example.com/v1beta1",
  18.         "metadata": {
  19.           "creationTimestamp": "2019-09-04T14:03:02Z",
  20.           "name": "local-crontab",
  21.           "namespace": "default",
  22.           "resourceVersion": "143",
  23.           "uid": "3415a7fc-162b-4300-b5da-fd6083580d66"
  24.         },
  25.         "hostPort": "localhost:1234"
  26.       },
  27.       {
  28.         "kind": "CronTab",
  29.         "apiVersion": "example.com/v1beta1",
  30.         "metadata": {
  31.           "creationTimestamp": "2019-09-03T13:02:01Z",
  32.           "name": "remote-crontab",
  33.           "resourceVersion": "12893",
  34.           "uid": "359a83ec-b575-460d-b553-d859cedde8a0"
  35.         },
  36.         "hostPort": "example.com:2345"
  37.       }
  38.     ]
  39.   }
  40. }

Response

Webhooks respond with a 200 HTTP status code, Content-Type: application/json, and a body containing a ConversionReview object (in the same version they were sent), with the response stanza populated, serialized to JSON.

If conversion succeeds, a webhook should return a response stanza containing the following fields:

  • uid, copied from the request.uid sent to the webhook
  • result, set to {"status":"Success"}
  • convertedObjects, containing all of the objects from request.objects, converted to request.desiredVersion

Example of a minimal successful response from a webhook:

  1. {
  2.   "apiVersion": "apiextensions.k8s.io/v1",
  3.   "kind": "ConversionReview",
  4.   "response": {
  5.     # must match <request.uid>
  6.     "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
  7.     "result": {
  8.       "status": "Success"
  9.     },
  10.     # Objects must match the order of request.objects, and have apiVersion set to <request.desiredAPIVersion>.
  11.     # kind, metadata.uid, metadata.name, and metadata.namespace fields must not be changed by the webhook.
  12.     # metadata.labels and metadata.annotations fields may be changed by the webhook.
  13.     # All other changes to metadata fields by the webhook are ignored.
  14.     "convertedObjects": [
  15.       {
  16.         "kind": "CronTab",
  17.         "apiVersion": "example.com/v1",
  18.         "metadata": {
  19.           "creationTimestamp": "2019-09-04T14:03:02Z",
  20.           "name": "local-crontab",
  21.           "namespace": "default",
  22.           "resourceVersion": "143",
  23.           "uid": "3415a7fc-162b-4300-b5da-fd6083580d66"
  24.         },
  25.         "host": "localhost",
  26.         "port": "1234"
  27.       },
  28.       {
  29.         "kind": "CronTab",
  30.         "apiVersion": "example.com/v1",
  31.         "metadata": {
  32.           "creationTimestamp": "2019-09-03T13:02:01Z",
  33.           "name": "remote-crontab",
  34.           "resourceVersion": "12893",
  35.           "uid": "359a83ec-b575-460d-b553-d859cedde8a0"
  36.         },
  37.         "host": "example.com",
  38.         "port": "2345"
  39.       }
  40.     ]
  41.   }
  42. }
  1. {
  2.   # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  3.   "apiVersion": "apiextensions.k8s.io/v1beta1",
  4.   "kind": "ConversionReview",
  5.   "response": {
  6.     # must match <request.uid>
  7.     "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
  8.     "result": {
  9.       "status": "Failed"
  10.     },
  11.     # Objects must match the order of request.objects, and have apiVersion set to <request.desiredAPIVersion>.
  12.     # kind, metadata.uid, metadata.name, and metadata.namespace fields must not be changed by the webhook.
  13.     # metadata.labels and metadata.annotations fields may be changed by the webhook.
  14.     # All other changes to metadata fields by the webhook are ignored.
  15.     "convertedObjects": [
  16.       {
  17.         "kind": "CronTab",
  18.         "apiVersion": "example.com/v1",
  19.         "metadata": {
  20.           "creationTimestamp": "2019-09-04T14:03:02Z",
  21.           "name": "local-crontab",
  22.           "namespace": "default",
  23.           "resourceVersion": "143",
  24.           "uid": "3415a7fc-162b-4300-b5da-fd6083580d66"
  25.         },
  26.         "host": "localhost",
  27.         "port": "1234"
  28.       },
  29.       {
  30.         "kind": "CronTab",
  31.         "apiVersion": "example.com/v1",
  32.         "metadata": {
  33.           "creationTimestamp": "2019-09-03T13:02:01Z",
  34.           "name": "remote-crontab",
  35.           "resourceVersion": "12893",
  36.           "uid": "359a83ec-b575-460d-b553-d859cedde8a0"
  37.         },
  38.         "host": "example.com",
  39.         "port": "2345"
  40.       }
  41.     ]
  42.   }
  43. }

If conversion fails, a webhook should return a response stanza containing the following fields:

  • uid, copied from the request.uid sent to the webhook
  • result, set to {"status":"Failed"}

Warning: Failing conversion can disrupt read and write access to the custom resources, including the ability to update or delete the resources. Conversion failures should be avoided whenever possible, and should not be used to enforce validation constraints (use validation schemas or webhook admission instead).

Example of a response from a webhook indicating a conversion request failed, with an optional message:

  1. {
  2.   "apiVersion": "apiextensions.k8s.io/v1",
  3.   "kind": "ConversionReview",
  4.   "response": {
  5.     "uid": "<value from request.uid>",
  6.     "result": {
  7.       "status": "Failed",
  8.       "message": "hostPort could not be parsed into a separate host and port"
  9.     }
  10.   }
  11. }
  1. {
  2.   # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
  3.   "apiVersion": "apiextensions.k8s.io/v1beta1",
  4.   "kind": "ConversionReview",
  5.   "response": {
  6.     "uid": "<value from request.uid>",
  7.     "result": {
  8.       "status": "Failed",
  9.       "message": "hostPort could not be parsed into a separate host and port"
  10.     }
  11.   }
  12. }

Writing, reading, and updating versioned CustomResourceDefinition objects

When an object is written, it is stored at the version designated as the storage version at the time of the write. If the storage version changes, existing objects are never converted automatically. However, newly-created or updated objects are written at the new storage version. It is possible for an object to have been written at a version that is no longer served.

When you read an object, you specify the version as part of the path. You can request an object at any version that is currently served. If you specify a version that is different from the object’s stored version, Kubernetes returns the object to you at the version you requested, but the stored object is not changed on disk.

What happens to the object that is being returned while serving the read request depends on what is specified in the CRD’s spec.conversion:

  • if the default strategy value None is specified, the only modifications to the object are changing the apiVersion string and perhaps pruning unknown fields (depending on the configuration). Note that this is unlikely to lead to good results if the schemas differ between the storage and requested version. In particular, you should not use this strategy if the same data is represented in different fields between versions.
  • if webhook conversion is specified, then this mechanism controls the conversion.

If you update an existing object, it is rewritten at the version that is currently the storage version. This is the only way that objects can change from one version to another.

To illustrate this, consider the following hypothetical series of events:

  1. The storage version is v1beta1. You create an object. It is stored at version v1beta1
  2. You add version v1 to your CustomResourceDefinition and designate it as the storage version. Here the schemas for v1 and v1beta1 are identical, which is typically the case when promoting an API to stable in the Kubernetes ecosystem.
  3. You read your object at version v1beta1, then you read the object again at version v1. Both returned objects are identical except for the apiVersion field.
  4. You create a new object. It is stored at version v1. You now have two objects, one of which is at v1beta1, and the other of which is at v1.
  5. You update the first object. It is now stored at version v1 since that is the current storage version.

Previous storage versions

The API server records each version which has ever been marked as the storage version in the status field storedVersions. Objects may have been stored at any version that has ever been designated as a storage version. No objects can exist in storage at a version that has never been a storage version.

Upgrade existing objects to a new stored version

When deprecating versions and dropping support, select a storage upgrade procedure.

Option 1: Use the Storage Version Migrator

  1. Run the storage Version migrator
  2. Remove the old version from the CustomResourceDefinition status.storedVersions field.

Option 2: Manually upgrade the existing objects to a new stored version

The following is an example procedure to upgrade from v1beta1 to v1.

  1. Set v1 as the storage in the CustomResourceDefinition file and apply it using kubectl. The storedVersions is now v1beta1, v1.
  2. Write an upgrade procedure to list all existing objects and write them with the same content. This forces the backend to write objects in the current storage version, which is v1.
  3. Remove v1beta1 from the CustomResourceDefinition status.storedVersions field.

Note:

The flag --subresource is used with the kubectl get, patch, edit, and replace commands to fetch and update the subresources, status and scale, for all the API resources that support them. This flag is available starting from kubectl version v1.24. Previously, reading subresources (like status) via kubectl involved using kubectl --raw, and updating subresources using kubectl was not possible at all. Starting from v1.24, the kubectl tool can be used to edit or patch the status subresource on a CRD object. See How to patch a Deployment using the subresource flag.

This page is part of the documentation for Kubernetes v1.27. If you are running a different version of Kubernetes, consult the documentation for that release.

Here is an example of how to patch the status subresource for a CRD object using kubectl:

  1. kubectl patch customresourcedefinitions <CRD_Name> --subresource='status' --type='merge' -p '{"status":{"storedVersions":["v1"]}}'