CustomResourceDefinition

CustomResourceDefinition（CRD）是 v1.7 + 新增的无需改变代码就可以扩展 Kubernetes API 的机制，用来管理自定义对象。它实际上是 ThirdPartyResources（TPR） 的升级版本，而 TPR 已经在 v1.8 中删除。

API 版本对照表

Kubernetes 版本 CRD API 版本
v1.8+ apiextensions.k8s.io/v1beta1

CRD 示例

下面的例子会创建一个 /apis/stable.example.com/v1/namespaces/<namespace>/crontabs/… 的自定义 API：

  1. apiVersion: apiextensions.k8s.io/v1beta1
  2. kind: CustomResourceDefinition
  3. metadata:
  4.   # name must match the spec fields below, and be in the form: <plural>.<group>
  5.   name: crontabs.stable.example.com
  6. spec:
  7.   # group name to use for REST API: /apis/<group>/<version>
  8.   group: stable.example.com
  9.   # version name to use for REST API: /apis/<group>/<version>
  10.   version: v1
  11.   # either Namespaced or Cluster
  12.   scope: Namespaced
  13.   names:
  14.     # plural name to be used in the URL: /apis/<group>/<version>/<plural>
  15.     plural: crontabs
  16.     # singular name to be used as an alias on the CLI and for display
  17.     singular: crontab
  18.     # kind is normally the CamelCased singular type. Your resource manifests use this.
  19.     kind: CronTab
  20.     # shortNames allow shorter string to match your resource on the CLI
  21.     shortNames:
  22.     - ct

API 创建好后，就可以创建具体的 CronTab 对象了

  1. $ cat my-cronjob.yaml
  2. apiVersion: "stable.example.com/v1"
  3. kind: CronTab
  4. metadata:
  5.   name: my-new-cron-object
  6. spec:
  7.   cronSpec: "* * * * /5"
  8.   image: my-awesome-cron-image
  10. $ kubectl create -f my-crontab.yaml
  11. crontab "my-new-cron-object" created
  13. $ kubectl get crontab
  14. NAME                 KIND
  15. my-new-cron-object   CronTab.v1.stable.example.com
  16. $ kubectl get crontab my-new-cron-object -o yaml
  17. apiVersion: stable.example.com/v1
  18. kind: CronTab
  19. metadata:
  20.   creationTimestamp: 2017-07-03T19:00:56Z
  21.   name: my-new-cron-object
  22.   namespace: default
  23.   resourceVersion: "20630"
  24.   selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
  25.   uid: 5c82083e-5fbd-11e7-a204-42010a8c0002
  26. spec:
  27.   cronSpec: '* * * * /5'
  28.   image: my-awesome-cron-image

Finalizer

Finalizer 用于实现控制器的异步预删除钩子，可以通过 metadata.finalizers 来指定 Finalizer。

  1. apiVersion: "stable.example.com/v1"
  2. kind: CronTab
  3. metadata:
  4.   finalizers:
  5.   - finalizer.stable.example.com

Finalizer 指定后，客户端删除对象的操作只会设置 metadata.deletionTimestamp 而不是直接删除。这会触发正在监听 CRD 的控制器，控制器执行一些删除前的清理操作，从列表中删除自己的 finalizer，然后再重新发起一个删除操作。此时，被删除的对象才会真正删除。

Validation

v1.8 开始新增了实验性的基于 OpenAPI v3 schema 的验证（Validation）机制，可以用来提前验证用户提交的资源是否符合规范。使用该功能需要配置 kube-apiserver 的 --feature-gates=CustomResourceValidation=true。

比如下面的 CRD 要求

  • spec.cronSpec 必须是匹配正则表达式的字符串
  • spec.replicas 必须是从 1 到 10 的整数
  1. apiVersion: apiextensions.k8s.io/v1beta1
  2. kind: CustomResourceDefinition
  3. metadata:
  4.   name: crontabs.stable.example.com
  5. spec:
  6.   group: stable.example.com
  7.   version: v1
  8.   scope: Namespaced
  9.   names:
  10.     plural: crontabs
  11.     singular: crontab
  12.     kind: CronTab
  13.     shortNames:
  14.     - ct
  15.   validation:
  16.    # openAPIV3Schema is the schema for validating custom objects.
  17.     openAPIV3Schema:
  18.       properties:
  19.         spec:
  20.           properties:
  21.             cronSpec:
  22.               type: string
  23.               pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
  24.             replicas:
  25.               type: integer
  26.               minimum: 1
  27.               maximum: 10

这样，在创建下面的 CronTab 时

  1. apiVersion: "stable.example.com/v1"
  2. kind: CronTab
  3. metadata:
  4.   name: my-new-cron-object
  5. spec:
  6.   cronSpec: "* * * *"
  7.   image: my-awesome-cron-image
  8.   replicas: 15

会报验证失败的错误：

  1. The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "selfLink":"","clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
  2. validation failure list:
  3. spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
  4. spec.replicas in body should be less than or equal to 10

Subresources

v1.10 开始 CRD 还支持 /status 和 /scale 等两个子资源（Alpha）。使用前需要在 kube-apiserver 开启 --feature-gates=CustomResourceSubresources=true。

  1. # resourcedefinition.yaml
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   name: crontabs.stable.example.com
  6. spec:
  7.   group: stable.example.com
  8.   version: v1
  9.   scope: Namespaced
  10.   names:
  11.     plural: crontabs
  12.     singular: crontab
  13.     kind: CronTab
  14.     shortNames:
  15.     - ct
  16.   # subresources describes the subresources for custom resources.
  17.   subresources:
  18.     # status enables the status subresource.
  19.     status: {}
  20.     # scale enables the scale subresource.
  21.     scale:
  22.       # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
  23.       specReplicasPath: .spec.replicas
  24.       # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
  25.       statusReplicasPath: .status.replicas
  26.       # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
  27.       labelSelectorPath: .status.labelSelector
  1. $ kubectl create -f resourcedefinition.yaml
  2. $ kubectl create -f- <<EOF
  3. apiVersion: "stable.example.com/v1"
  4. kind: CronTab
  5. metadata:
  6.   name: my-new-cron-object
  7. spec:
  8.   cronSpec: "* * * * */5"
  9.   image: my-awesome-cron-image
  10.   replicas: 3
  11. EOF
  13. $ kubectl scale --replicas=5 crontabs/my-new-cron-object
  14. crontabs "my-new-cron-object" scaled
  16. $ kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
  17. 5

Categories

Categories 用来将 CRD 对象分组，这样就可以使用 kubectl get <category-name> 来查询属于该组的所有对象。

  1. # resourcedefinition.yaml
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   name: crontabs.stable.example.com
  6. spec:
  7.   group: stable.example.com
  8.   version: v1
  9.   scope: Namespaced
  10.   names:
  11.     plural: crontabs
  12.     singular: crontab
  13.     kind: CronTab
  14.     shortNames:
  15.     - ct
  16.     # categories is a list of grouped resources the custom resource belongs to.
  17.     categories:
  18.     - all
  1. # my-crontab.yaml
  2. apiVersion: "stable.example.com/v1"
  3. kind: CronTab
  4. metadata:
  5.   name: my-new-cron-object
  6. spec:
  7.   cronSpec: "* * * * */5"
  8.   image: my-awesome-cron-image
  1. $ kubectl create -f resourcedefinition.yaml
  2. $ kubectl create -f my-crontab.yaml
  3. $ kubectl get all
  4. NAME                          AGE
  5. crontabs/my-new-cron-object   3s

CRD 控制器

在使用 CRD 扩展 Kubernetes API 时，通常还需要实现一个新建资源的控制器，监听改资源的变化情况，并作进一步的处理。

https://github.com/kubernetes/sample-controller 提供了一个 CRD 控制器的示例，包括

  • 如何注册资源 Foo
  • 如何创建、删除和查询 Foo 对象
  • 如何监听 Foo 资源对象的变化情况

参考文档