Operator SDK tutorial for Hybrid Helm Operators

The standard Helm-based Operator support in the Operator SDK has limited functionality compared to the Go-based and Ansible-based Operator support that has reached the Auto Pilot capability (level V) in the Operator maturity model.

The Hybrid Helm Operator enhances the existing Helm-based support’s abilities through Go APIs. With this hybrid approach of Helm and Go, the Operator SDK enables Operator authors to use the following process:

• Generate a default structure for, or scaffold, a Go API in the same project as Helm.

• Configure the Helm reconciler in the main.go file of the project, through the libraries provided by the Hybrid Helm Operator.

This tutorial walks through the following process using the Hybrid Helm Operator:

• Create a Memcached deployment through a Helm chart if it does not exist

• Ensure that the deployment size is the same as specified by Memcached custom resource (CR) spec

• Create a MemcachedBackup deployment by using the Go API

Prerequisites

• Operator SDK CLI installed

• OpenShift CLI (oc) 4.14+ installed

• Logged into an OKD 4.14 cluster with oc with an account that has cluster-admin permissions

• To allow the cluster to pull the image, the repository where you push your image must be set as public, or you must configure an image pull secret

Additional resources

• Installing the Operator SDK CLI

• Getting started with the OpenShift CLI

Creating a project

Use the Operator SDK CLI to create a project called memcached-operator.

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/github.com/example/memcached-operator

2. Change to the directory:

   $ cd $HOME/github.com/example/memcached-operator

3. Run the operator-sdk init command to initialize the project. Use a domain of example.com so that all API groups are <group>.example.com:

   $ operator-sdk init \
       --plugins=hybrid.helm.sdk.operatorframework.io \
       --project-version="3" \
       --domain example.com \
       --repo=github.com/example/memcached-operator

   The init command generates the RBAC rules in the config/rbac/role.yaml file based on the resources that would be deployed by the chart’s default manifests. Verify that the rules generated in the config/rbac/role.yaml file meet your Operator’s permission requirements.

Additional resources

• This procedure creates a project structure that is compatible with both Helm and Go APIs. To learn more about the project directory structure, see Project layout.

Creating a Helm API

Use the Operator SDK CLI to create a Helm API.

Procedure

• Run the following command to create a Helm API with group cache, version v1, and kind Memcached:

  $ operator-sdk create api \
      --plugins helm.sdk.operatorframework.io/v1 \
      --group cache \
      --version v1 \
      --kind Memcached

$ operator-sdk create api —plugins helm.sdk.operatorframework.io/v1 —help

Additional resources

• Existing Helm charts

Operator logic for the Helm API

By default, your scaffolded Operator project watches Memcached resource events as shown in the watches.yaml file and executes Helm releases using the specified chart.

Example watches.yaml file

# Use the 'create api' subcommand to add watches to this file.
- group: cache.my.domain
  version: v1
  kind: Memcached
  chart: helm-charts/memcached
#+kubebuilder:scaffold:watch

Additional resources

• For detailed documentation on customizing the Helm Operator logic through the chart, see Understanding the Operator logic.

Custom Helm reconciler configurations using provided library APIs

A disadvantage of existing Helm-based Operators is the inability to configure the Helm reconciler, because it is abstracted from users. For a Helm-based Operator to reach the Seamless Upgrades capability (level II and later) that reuses an already existing Helm chart, a hybrid between the Go and Helm Operator types adds value.

The APIs provided in the helm-operator-plugins library allow Operator authors to make the following configurations:

• Customize value mapping based on cluster state

• Execute code in specific events by configuring the reconciler’s event recorder

• Customize the reconciler’s logger

• Setup Install, Upgrade, and Uninstall annotations to enable Helm’s actions to be configured based on the annotations found in custom resources watched by the reconciler

• Configure the reconciler to run with Pre and Post hooks

The above configurations to the reconciler can be done in the main.go file:

Details

Example main.go file

// Operator's main.go
// With the help of helpers provided in the library, the reconciler can be
// configured here before starting the controller with this reconciler.
reconciler := reconciler.New(
 reconciler.WithChart(*chart),
 reconciler.WithGroupVersionKind(gvk),
)
if err := reconciler.SetupWithManager(mgr); err != nil {
 panic(fmt.Sprintf("unable to create reconciler: %s", err))
}

Creating a Go API

Use the Operator SDK CLI to create a Go API.

Procedure

1. Run the following command to create a Go API with group cache, version v1, and kind MemcachedBackup:

   $ operator-sdk create api \
       --group=cache \
       --version v1 \
       --kind MemcachedBackup \
       --resource \
       --controller \
       --plugins=go/v3

2. When prompted, enter y for creating both resource and controller:

   $ Create Resource [y/n]
   y
   Create Controller [y/n]
   y

This procedure generates the MemcachedBackup resource API at api/v1/memcachedbackup_types.go and the controller at controllers/memcachedbackup_controller.go.

Defining the API

Define the API for the MemcachedBackup custom resource (CR).

Represent this Go API by defining the MemcachedBackup type, which will have a MemcachedBackupSpec.Size field to set the quantity of Memcached backup instances (CRs) to be deployed, and a MemcachedBackupStatus.Nodes field to store a CR’s pod names.

Procedure

1. Define the API for the MemcachedBackup CR by modifying the Go type definitions in the api/v1/memcachedbackup_types.go file to have the following spec and status:

   Example api/v1/memcachedbackup_types.go file

   // MemcachedBackupSpec defines the desired state of MemcachedBackup
   type MemcachedBackupSpec struct {
       // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
       // Important: Run "make" to regenerate code after modifying this file
       //+kubebuilder:validation:Minimum=0
       // Size is the size of the memcached deployment
       Size int32 `json:"size"`
   }
   // MemcachedBackupStatus defines the observed state of MemcachedBackup
   type MemcachedBackupStatus struct {
       // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
       // Important: Run "make" to regenerate code after modifying this file
       // Nodes are the names of the memcached pods
       Nodes []string `json:"nodes"`
   }

2. Update the generated code for the resource type:

   $ make generate

   After you modify a *_types.go file, you must run the make generate command to update the generated code for that resource type.

3. After the API is defined with spec and status fields and CRD validation markers, generate and update the CRD manifests:

   $ make manifests

This Makefile target invokes the controller-gen utility to generate the CRD manifests in the config/crd/bases/cache.my.domain_memcachedbackups.yaml file.

Controller implementation

The controller in this tutorial performs the following actions:

• Create a Memcached deployment if it does not exist.

• Ensure that the deployment size is the same as specified by the Memcached CR spec.

• Update the Memcached CR status with the names of the memcached pods.

For a detailed explanation on how to configure the controller to perform the above mentioned actions, see Implementing the controller in the Operator SDK tutorial for standard Go-based Operators.

Differences in main.go

For standard Go-based Operators and the Hybrid Helm Operator, the main.go file handles the scaffolding the initialization and running of the Manager program for the Go API. For the Hybrid Helm Operator, however, the main.go file also exposes the logic for loading the watches.yaml file and configuring the Helm reconciler.

Example main.go file

...
    for _, w := range ws {
        // Register controller with the factory
        reconcilePeriod := defaultReconcilePeriod
        if w.ReconcilePeriod != nil {
            reconcilePeriod = w.ReconcilePeriod.Duration
        }
        maxConcurrentReconciles := defaultMaxConcurrentReconciles
        if w.MaxConcurrentReconciles != nil {
            maxConcurrentReconciles = *w.MaxConcurrentReconciles
        }
        r, err := reconciler.New(
            reconciler.WithChart(*w.Chart),
            reconciler.WithGroupVersionKind(w.GroupVersionKind),
            reconciler.WithOverrideValues(w.OverrideValues),
            reconciler.SkipDependentWatches(w.WatchDependentResources != nil && !*w.WatchDependentResources),
            reconciler.WithMaxConcurrentReconciles(maxConcurrentReconciles),
            reconciler.WithReconcilePeriod(reconcilePeriod),
            reconciler.WithInstallAnnotations(annotation.DefaultInstallAnnotations...),
            reconciler.WithUpgradeAnnotations(annotation.DefaultUpgradeAnnotations...),
            reconciler.WithUninstallAnnotations(annotation.DefaultUninstallAnnotations...),
        )
...

The manager is initialized with both Helm and Go reconcilers:

Example Helm and Go reconcilers

...
// Setup manager with Go API
   if err = (&controllers.MemcachedBackupReconciler{
        Client: mgr.GetClient(),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller", "controller", "MemcachedBackup")
        os.Exit(1)
    }
   ...
// Setup manager with Helm API
    for _, w := range ws {
      ...
        if err := r.SetupWithManager(mgr); err != nil {
            setupLog.Error(err, "unable to create controller", "controller", "Helm")
            os.Exit(1)
        }
        setupLog.Info("configured watch", "gvk", w.GroupVersionKind, "chartPath", w.ChartPath, "maxConcurrentReconciles", maxConcurrentReconciles, "reconcilePeriod", reconcilePeriod)
    }
// Start the manager
   if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }

Permissions and RBAC manifests

The controller requires certain role-based access control (RBAC) permissions to interact with the resources it manages. For the Go API, these are specified with RBAC markers, as shown in the Operator SDK tutorial for standard Go-based Operators.

For the Helm API, the permissions are scaffolded by default in roles.yaml. Currently, however, due to a known issue when the Go API is scaffolded, the permissions for the Helm API are overwritten. As a result of this issue, ensure that the permissions defined in roles.yaml match your requirements.

The following is an example role.yaml for a Memcached Operator:

Example Helm and Go reconcilers

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: manager-role
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - cache.my.domain
  resources:
  - memcachedbackups
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - cache.my.domain
  resources:
  - memcachedbackups/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - services/finalizers
  - endpoints
  - persistentvolumeclaims
  - events
  - configmaps
  - secrets
  - serviceaccounts
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - cache.my.domain
  resources:
  - memcachedbackups/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - policy
  resources:
  - events
  - poddisruptionbudgets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - cache.my.domain
  resources:
  - memcacheds
  - memcacheds/status
  - memcacheds/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

Additional resources

• RBAC markers for Go-based Operators

Running locally outside the cluster

You can run your Operator project as a Go program outside of the cluster. This is useful for development purposes to speed up deployment and testing.

Procedure

• Run the following command to install the custom resource definitions (CRDs) in the cluster configured in your ~/.kube/config file and run the Operator locally:

  $ make install run

Running as a deployment on the cluster

You can run your Operator project as a deployment on your cluster.

Procedure

1. Run the following make commands to build and push the Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

   1. Build the image:

      $ make docker-build IMG=<registry>/<user>/<image_name>:<tag>

      The Dockerfile generated by the SDK for the Operator explicitly references GOARCH=amd64 for go build. This can be amended to GOARCH=$TARGETARCH for non-AMD64 architectures. Docker will automatically set the environment variable to the value specified by –platform. With Buildah, the –build-arg will need to be used for the purpose. For more information, see Multiple Architectures.

   2. Push the image to a repository:

      $ make docker-push IMG=<registry>/<user>/<image_name>:<tag>

      The name and tag of the image, for example IMG=<registry>/<user>/<image_name>:<tag>, in both the commands can also be set in your Makefile. Modify the IMG ?= controller:latest value to set your default image name.

2. Run the following command to deploy the Operator:

   $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

   By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system and is used for the deployment. This command also installs the RBAC manifests from config/rbac.

3. Run the following command to verify that the Operator is running:

   $ oc get deployment -n <project_name>-system

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   <project_name>-controller-manager       1/1     1            1           8m

Creating custom resources

After your Operator is installed, you can test it by creating custom resources (CRs) that are now provided on the cluster by the Operator.

Procedure

1. Change to the namespace where your Operator is installed:

   $ oc project <project_name>-system

2. Update the sample Memcached CR manifest at the config/samples/cache_v1_memcached.yaml file by updating the replicaCount field to 3:

   Example config/samples/cache_v1_memcached.yaml file

   apiVersion: cache.my.domain/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   spec:
     # Default values copied from <project_dir>/helm-charts/memcached/values.yaml
     affinity: {}
     autoscaling:
       enabled: false
       maxReplicas: 100
       minReplicas: 1
       targetCPUUtilizationPercentage: 80
     fullnameOverride: ""
     image:
       pullPolicy: IfNotPresent
       repository: nginx
       tag: ""
     imagePullSecrets: []
     ingress:
       annotations: {}
       className: ""
       enabled: false
       hosts:
       - host: chart-example.local
         paths:
         - path: /
           pathType: ImplementationSpecific
       tls: []
     nameOverride: ""
     nodeSelector: {}
     podAnnotations: {}
     podSecurityContext: {}
     replicaCount: 3
     resources: {}
     securityContext: {}
     service:
       port: 80
       type: ClusterIP
     serviceAccount:
       annotations: {}
       create: true
       name: ""
     tolerations: []

3. Create the Memcached CR:

   $ oc apply -f config/samples/cache_v1_memcached.yaml

4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

   $ oc get pods

   Example output

   NAME                                  READY     STATUS    RESTARTS   AGE
   memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          18m
   memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          18m
   memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          18m

5. Update the sample MemcachedBackup CR manifest at the config/samples/cache_v1_memcachedbackup.yaml file by updating the size to 2:

   Example config/samples/cache_v1_memcachedbackup.yaml file

   apiVersion: cache.my.domain/v1
   kind: MemcachedBackup
   metadata:
     name: memcachedbackup-sample
   spec:
     size: 2

6. Create the MemcachedBackup CR:

   $ oc apply -f config/samples/cache_v1_memcachedbackup.yaml

7. Ensure that the count of memcachedbackup pods is the same as specified in the CR:

   $ oc get pods

   Example output

   NAME                                        READY     STATUS    RESTARTS   AGE
   memcachedbackup-sample-8649699989-4bbzg     1/1       Running   0          22m
   memcachedbackup-sample-8649699989-mq6mx     1/1       Running   0          22m

8. You can update the spec in each of the above CRs, and then apply them again. The controller reconciles again and ensures that the size of the pods is as specified in the spec of the respective CRs.

9. Clean up the resources that have been created as part of this tutorial:

   1. Delete the Memcached resource:

      $ oc delete -f config/samples/cache_v1_memcached.yaml

   2. Delete the MemcachedBackup resource:

      $ oc delete -f config/samples/cache_v1_memcachedbackup.yaml

   3. If you used the make deploy command to test the Operator, run the following command:

      $ make undeploy

Project layout

The Hybrid Helm Operator scaffolding is customized to be compatible with both Helm and Go APIs.