Operators and CRD scope with Operator SDK

Overview

A namespace-scoped operator watches and manages resources in a single namespace, whereas a cluster-scoped operator watches and manages resources cluster-wide. Namespace-scoped operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.

However, there are use cases where a cluster-scoped operator may make sense. For example, the cert-manager operator is often deployed with cluster-scoped permissions and watches so that it can manage issuing certificates for an entire cluster.

Namespace-scoped operator usage

This scope is ideal for operator projects which will control resources just in one namespace, which is where the operator is deployed.

NOTE: Initial projects created by operator-sdk are namespace-scoped by default which means that it will NOT have a ClusterRole defined in the deploy/role_binding.yaml.

Cluster-scoped operator usage

This scope is ideal for operator projects which will control resources in more than one namespace.

Changes required for a cluster-scoped operator

The SDK scaffolds operators to be namespaced by default but with a few modifications to the default manifests the operator can be run as cluster-scoped.

  • deploy/operator.yaml:
    • Set WATCH_NAMESPACE="" to watch all namespaces instead of setting it to the pod’s namespace
    • Set metadata.namespace to define the namespace where the operator will be deployed.
  • deploy/role.yaml:
    • Use ClusterRole instead of Role
  • deploy/role_binding.yaml:
    • Use ClusterRoleBinding instead of RoleBinding
    • Use ClusterRole instead of Role for roleRef
    • Set the subject namespace to the namespace in which the operator is deployed.
  • deploy/service_account.yaml:
    • Set metadata.namespace to the namespace where the operator is deployed.

Example for cluster-scoped operator

With the above changes the specified manifests should look as follows:

  • deploy/operator.yaml:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   name: memcached-operator
    5.   namespace: <operator-namespace>
    6. ...
    7. spec:
    8.   ...
    9.   template:
    10.     ...
    11.     spec:
    12.       ...
    13.       serviceAccountName: memcached-operator
    14.       containers:
    15.       - name: memcached-operator
    16.         ...
    17.         env:
    18.         - name: WATCH_NAMESPACE
    19.           value: ""

  • deploy/role.yaml:

    1. apiVersion: rbac.authorization.k8s.io/v1
    2. kind: ClusterRole
    3. metadata:
    4.   name: memcached-operator
    5. ...

  • deploy/role_binding.yaml:

    1. kind: ClusterRoleBinding
    2. apiVersion: rbac.authorization.k8s.io/v1
    3. metadata:
    4.   name: memcached-operator
    5. subjects:
    6. - kind: ServiceAccount
    7.   name: memcached-operator
    8.   namespace: <operator-namespace>
    9. roleRef:
    10.   kind: ClusterRole
    11.   name: memcached-operator
    12.   apiGroup: rbac.authorization.k8s.io

  • deploy/service_account.yaml

    1. apiVersion: v1
    2. kind: ServiceAccount
    3. metadata:
    4.   name: memcached-operator
    5.   namespace: <operator-namespace>

CRD scope

Additionally the CustomResourceDefinition (CRD) scope can also be changed for cluster-scoped operators so that there is only a single instance (for a given name) of the CRD to manage across the cluster.

NOTE: Cluster-scoped CRDs are NOT supported with the Helm operator. While Helm releases can create cluster-scoped resources, Helm’s design requires the release itself to be created in a specific namespace. Since the Helm operator uses a 1-to-1 mapping between a CR and a Helm release, Helm’s namespace-scoped release requirement extends to Helm operator’s namespace-scoped CR requirement.

For each CRD that needs to be cluster-scoped, update its manifest to be cluster-scoped.

  • deploy/crds/<full group>_<resource>_crd.yaml
    • Set spec.scope: Cluster

To ensure that the CRD is always generated with scope: Cluster, add the tag // +kubebuilder:resource:path=<resource>,scope=Cluster, or if already present replace scope={Namespaced -> Cluster}, above the CRD’s Go type definition in pkg/apis/<group>/<version>/<kind>_types.go. Note that the <resource> element must be the same lower-case plural value of the CRD’s Kind, spec.names.plural.

CRD cluster-scoped usage

This scope is ideal for the cases where an instance(CR) of some Kind(CRD) will be used in more than one namespace instead of a specific one.

NOTE: When a Manager instance is created in the main.go file, it receives the namespace(s) as Options. These namespace(s) should be watched and cached for the Client which is provided by the Controllers. Only clients provided by cluster-scoped projects where the Namespace attribute is "" will be able to manage cluster-scoped CRD’s. For more information see the Manager topic in the user guide and the Manager Options.

Example for changing the CRD scope from namespace to cluster

The following example is for Go based-operators. Note that for Helm and Ansible based-operators the changes in the CRD required to be done manually.

  • Check the spec.names.plural in the CRD’s Kind YAML file

  • deploy/crds/cache_v1alpha1_memcached_crd.yaml

    1. apiVersion: apiextensions.k8s.io/v1beta1
    2. kind: CustomResourceDefinition
    3. metadata:
    4.   name: memcacheds.cache.example.com
    5. spec:
    6.   group: cache.example.com
    7.   names:
    8.     kind: Memcached
    9.     listKind: MemcachedList
    10.     plural: memcacheds
    11.     singular: memcached
    12.   scope: Namespaced

  • Update the pkg/apis/<group>/<version>/<kind>_types.go by adding the tag // +kubebuilder:resource:path=<resource>,scope=Cluster

  • pkg/apis/cache/v1alpha1/memcached_types.go

    1. // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
    3. // Memcached is the Schema for the memcacheds API
    4. // +kubebuilder:resource:path=memcacheds,scope=Cluster
    5. type Memcached struct {
    6.   metav1.TypeMeta   `json:",inline"`
    7.   metav1.ObjectMeta `json:"metadata,omitempty"`
    9.   Spec   MemcachedSpec   `json:"spec,omitempty"`
    10.   Status MemcachedStatus `json:"status,omitempty"`
    11. }

  • Execute the command operator-sdk generate crds, then you should be able to check that the CRD was updated with the cluster scope as in the following example:

  • deploy/crds/cache.example.com_memcacheds_crd.yaml

    1. apiVersion: apiextensions.k8s.io/v1beta1
    2. kind: CustomResourceDefinition
    3. metadata:
    4.   name: memcacheds.cache.example.com
    5. spec:
    6.   group: cache.example.com
    7.   ...
    8.   scope: Cluster

Last modified January 1, 0001