Getting started with the Operator SDK

This guide walks through an example of building a simple Memcached Operator (memcached-operator) using tools and libraries provided by the Operator SDK, and shows how to manage the new Operator’s lifecycle from installation through updating to a new version.

As an administrator of a Kubernetes-based cluster, you can accomplish this using two centerpieces of the Operator Framework:

Operator SDK

Assists developers in bootstrapping and building an Operator based on their expertise without requiring knowledge of Kubernetes API complexities.

Operator Lifecycle Manager (OLM)

Helps Operator users install, update, and generally manage the lifecycle of all Operators and their associated services running across their clusters.

About the Operator SDK

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. Operators take advantage of Kubernetes’ extensibility to deliver the automation advantages of cloud services like provisioning, scaling, and backup and restore, while being able to run anywhere that Kubernetes can run.

Operators make it easy to manage complex, stateful applications on top of Kubernetes. However, writing an Operator today can be difficult because of challenges such as using low-level APIs, writing boilerplate, and a lack of modularity, which leads to duplication.

The Operator SDK is a framework designed to make writing operators easier by providing:

• High-level APIs and abstractions to write the operational logic more intuitively

• Tools for scaffolding and code generation to quickly bootstrap a new project

• Extensions to cover common operator use cases

Operator SDK Workflow

The SDK provides the following workflow to develop a new operator:

1. Create a new Operator project using the SDK command line interface (CLI).

2. Define new resource APIs by adding Custom Resource Definitions (CRDs).

3. Specify resources to watch using the SDK API.

4. Define the Operator reconciling logic in a designated handler and use the SDK API to interact with resources.

5. Use the SDK CLI to build and generate the Operator deployment manifests.

At a high level, an Operator using the SDK processes events for watched resources in a user-defined handler and takes actions to reconcile the state of the application.

Installing the Operator SDK CLI

The Operator SDK has a CLI tool that assists developers in creating, building, and deploying a new operator project. You can install the SDK CLI on your workstation so you are prepared to start authoring your own Operators.

Prerequisites

• dep v0.5.0+

• Git

• Go v1.10+

• Docker v17.03+

• kubectl v1.11.0+

• Access to a cluster based on Kubernetes v1.11.0+

• Access to a container image registry

Procedure

1. Clone and operator-sdk repository:

   $ mkdir -p $GOPATH/src/github.com/operator-framework
   $ cd $GOPATH/src/github.com/operator-framework
   $ git clone https://github.com/operator-framework/operator-sdk
   $ cd operator-sdk

2. Check out the desired release branch:

   $ git checkout master

3. Install the SDK CLI tool:

   $ make dep
   $ make install

   This installs the CLI binary operator-sdk at $GOPATH/bin.

4. Verify that the CLI tool was installed correctly:

   $ operator-sdk -h

Building a Memcached Operator using the Operator SDK

The Operator SDK makes it easier to build Kubernetes native applications, a process that can require deep, application-specific operational knowledge. The SDK not only lowers that barrier, but it also helps reduce the amount of boilerplate code needed for many common management capabilities, such as metering or monitoring.

This procedure walks through an example of building a simple Memcached Operator using tools and libraries provided by the SDK.

Prerequisites

• Operator SDK CLI installed on the development workstation

• Operator Lifecycle Manager (OLM) installed on a Kubernetes-based cluster (v1.8 or above to support the apps/v1beta2 API group), for example OKD 3.11 with Technology Preview OLM enabled

• Access to the cluster using an account with cluster-admin permissions

• kubectl v1.9.0+ (can alternatively use oc)

Procedure

1. Create a new project.

   Use the CLI to create a new memcached-operator project:

   $ cd $GOPATH/src/github.com/example-inc/
   $ operator-sdk new memcached-operator --api-version=cache.example.com/v1alpha1 --kind=Memcached
   $ cd memcached-operator

   This creates the memcached-operator project specifically for watching the Memcached resource with APIVersion cache.example.com/v1apha1 and Kind Memcached.

   See Appendices for more about the project directory structure.

2. Customize the Operator logic.

   For this example, the Memcached Operator executes the following reconciliation logic for each Memcached custom resource:

   • Create a Memcached deployment if it does not exist.

   • Ensure that the deployment size is the same as specified by the Memcached CustomResource (CR) spec.

   • Update the Memcached CR status with the names of the memcached pods.

Customize the logic using the following steps:
1.  Watch for the Memcached custom resource definition (CRD). By default, the `memcached-operator` watches `Memcached` resource events as shown in `cmd/memcached-operator/main.go`:
    ```
    func main() {
        printVersion()
        sdk.ExposeMetricsPort()
        resource := "cache.example.com/v1alpha1"
        kind := "Memcached"
        namespace, err := k8sutil.GetWatchNamespace()
        if err != nil {
            logrus.Fatalf("Failed to get watch namespace: %v", err)
        }
        resyncPeriod := 5
        logrus.Infof("Watching %s, %s, %s, %d", resource, kind, namespace, resyncPeriod)
        sdk.Watch(resource, kind, namespace, resyncPeriod)
        sdk.Handle(stub.NewHandler())
        sdk.Run(context.TODO())
    }
    ```
2.  Define the Memcached spec and status.
    1.  Modify the spec and status of the `Memcached` CRD at `pkg/apis/cache/v1alpha1/types.go`:
        ```
        type MemcachedSpec struct {
            // Size is the size of the memcached deployment
            Size int32 `json:"size"`
        }
        type MemcachedStatus struct {
            // Nodes are the names of the memcached pods
            Nodes []string `json:"nodes"`
        }
        ```
    2.  Update the generated code for the CR:
        ```
        $ cd $GOPATH/src/github.com/example-inc/
        $ cd memcached-operator
        $ operator-sdk generate k8s
        ```
3.  Define the Handler. The reconciliation loop for an event is defined in the `Handle()` function at `pkg/stub/handler.go`.
    Replace this file with the reference implementation found at [handler\_go](https://github.com/operator-framework/getting-started/blob/master/handler.go.tmpl#L7). You must update the highlighted line if you have changed the import path of this project to something other than `example-inc`.
    <table><tbody><tr><td><i title="Note"></i></td><td><div><p>The provided handler implementation is only meant to demonstrate the use of the SDK APIs and is not representative of the best practices of a reconciliation loop.</p></div></td></tr></tbody></table>

1. Build and run the Operator.

   1. Build the memcached-operator image and push it to a registry. Ensure you have an account on quay.io for the next step, or substitute your preferred container image registry. On the registry, create a new public image repository named memcached-operator.

      $ cd $GOPATH/src/github.com/example-inc/
      $ cd memcached-operator
      $ operator-sdk build quay.io/example/memcached-operator:v0.0.1
      $ docker push quay.io/example/memcached-operator:v0.0.1

      Kubernetes deployment manifests are generated in the deploy/operator.yaml file. The deployment image is set to the container image specified above.

   2. Log in to your cluster as a user with cluster-admin permissions and deploy the Memcached Operator:

      $ kubectl create -f deploy/rbac.yaml
      $ kubectl create -f deploy/crd.yaml
      $ kubectl create -f deploy/operator.yaml

   3. Verify that the memcached-operator pod is up and running:

      $ kubectl get pods
      NAME                                  READY     STATUS    RESTARTS   AGE
      memcached-operator-75c4b4c665-8jnj5   1/1       Running   0          20s

1. To verify that the Operator can deploy a Memcached application, create a Memcached custom resource (CR).

   1. Modify the deploy/cr.yaml file as shown:

      $ cat deploy/cr.yaml
      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 3

   2. Create the Memcached CR:

      $ kubectl apply -f deploy/cr.yaml

   3. Verify that the Memcached Operator creates the deployment for the CR:

      $ kubectl get deployment
      NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator       1         1         1            1           2m
      example-memcached        3         3         3            3           1m

   4. Check the pods and CR status to confirm the status is updated with the memcached pod names:

      $ kubectl get pods
      NAME                                  READY     STATUS    RESTARTS   AGE
      example-memcached-6fd7c98d8-7dqdr     1/1       Running   0          1m
      example-memcached-6fd7c98d8-g5k7v     1/1       Running   0          1m
      example-memcached-6fd7c98d8-m7vn7     1/1       Running   0          1m
      memcached-operator-7cc7cfdf86-vvjqk   1/1       Running   0          2m

      $ kubectl get memcached/example-memcached -o yaml
      apiVersion: cache.example.com/v1alpha1
      kind: Memcached
      metadata:
        clusterName: ""
        creationTimestamp: 2018-03-31T22:51:08Z
        generation: 0
        name: example-memcached
        namespace: default
        resourceVersion: "245453"
        selfLink: /apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/example-memcached
        uid: 0026cc97-3536-11e8-bd83-0800274106a1
      spec:
        size: 3
      status:
        nodes:
        - example-memcached-6fd7c98d8-7dqdr
        - example-memcached-6fd7c98d8-g5k7v
        - example-memcached-6fd7c98d8-m7vn7

1. To verify that the Operator can manage a deployed Memcached application, update the size of the deployment.

   1. Change the spec.size field in the memcached CR from 3 to 4:

      $ cat deploy/cr.yaml
      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 4

   2. Apply the change:

      $ kubectl apply -f deploy/cr.yaml

   3. Confirm that the Memcache Operator changes the deployment size:

      $ kubectl get deployment
      NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      example-memcached    4         4         4            4           5m

1. Finally, clean everything up:

   $ kubectl delete -f deploy/operator.yaml
   $ kubectl delete -f deploy/rbac.yaml
   $ kubectl delete -f deploy/cr.yaml
   $ kubectl delete -f deploy/crd.yaml

Managing a Memcached Operator using the Operator Lifecycle Manager

The previous section has covered manually running an Operator. In the next sections, we will explore using the Operator Lifecycle Manager (OLM), which is what enables a more robust deployment model for Operators being run in production environments.

The OLM helps you to install, update, and generally manage the lifecycle of all of the Operators (and their associated services) on a Kubernetes cluster. It runs as an Kubernetes extension and lets you use kubectl for all the lifecycle management functions without any additional tools.

Prerequisites

• OLM installed on a Kubernetes-based cluster (v1.8 or above to support the apps/v1beta2 API group), for example OKD 3.11 with Technology Preview OLM enabled

• Memcached Operator built

Procedure

1. Generate an Operator manifest.

   An Operator manifest describes how to display, create, and manage the application, in this case Memcached, as a whole. It is defined by a CustomServiceVersion (CSV) object and is required for the OLM to function.

   For the purpose of this guide, we will continue with this predefined manifest file for the next steps. You can alter the image field within this manifest to reflect the image you built in previous steps, but it is unnecessary. In the future, the Operator SDK CLI will generate an Operator manifest for you, a feature that is planned for the next release of the Operator SDK.

   See Building a CSV for the Operator Framework for more information on manually defining a manifest file.

2. Deploy the Operator.

   1. Deploying an Operator is as simple as applying the Operator’s manifest to the desired namespace in the cluster.

      $ curl -Lo memcachedoperator.0.0.1.csv.yaml https://raw.githubusercontent.com/operator-framework/getting-started/master/memcachedoperator.0.0.1.csv.yaml
      $ kubectl apply -f memcachedoperator.0.0.1.csv.yaml
      $ kubectl get ClusterServiceVersion memcachedoperator.v0.0.1 -o json | jq '.status'

   2. After applying this manifest, nothing has happened yet, because the cluster does not meet the requirements specified in our manifest. Create the RBAC rules and CustomResourceDefinition for the Memcached type managed by the Operator:

      $ kubectl apply -f deploy/rbac.yaml
      $ kubectl apply -f deploy/crd.yaml

      Because the OLM creates Operators in a particular namespace when a manifest is applied, administrators can leverage the native Kubernetes RBAC permission model to restrict which users are allowed to install Operators.

1. Create an application instance.

   The Memcached Operator is now running in the memcached namespace. Users interact with Operators via instances of CustomResources; in this case, the resource has the kind Memcached. Native Kubernetes RBAC also applies to CustomResources, providing administrators control over who can interact with each Operator.

   Creating instances of Memcached in this namespace will now trigger the Memcached Operator to instantiate pods running the memcached server that are managed by the Operator. The more CustomResources you create, the more unique instances of Memcached are managed by the Memcached Operator running in this namespace.

   $ cat <<EOF | kubectl apply -f -
   apiVersion: "cache.example.com/v1alpha1"
   kind: "Memcached"
   metadata:
     name: "memcached-for-wordpress"
   spec:
     size: 1
   EOF
   $ cat <<EOF | kubectl apply -f -
   apiVersion: "cache.example.com/v1alpha1"
   kind: "Memcached"
   metadata:
     name: "memcached-for-drupal"
   spec:
     size: 1
   EOF
   $ kubectl get Memcached
   NAME                      AGE
   memcached-for-drupal      22s
   memcached-for-wordpress   27s
   $ kubectl get pods
   NAME                                       READY     STATUS    RESTARTS   AGE
   memcached-app-operator-66b5777b79-pnsfj    1/1       Running   0          14m
   memcached-for-drupal-5476487c46-qbd66      1/1       Running   0          3s
   memcached-for-wordpress-65b75fd8c9-7b9x7   1/1       Running   0          8s

2. Update an application.

   Manually applying an update to the Operator is as simple as creating a new Operator manifest with a replaces field that references the old Operator manifest. The OLM ensures that all resources being managed by the old Operator have their ownership moved to the new Operator without fear of any programs stopping execution. It is up to the Operators themselves to execute any data migrations required to upgrade resources to run under a new version of the Operator.

   The following command demonstrates applying a new Operator manifest file using a new version of the Operator and shows that the pods remain executing:

   $ curl -Lo memcachedoperator.0.0.2.csv.yaml https://raw.githubusercontent.com/operator-framework/getting-started/master/memcachedoperator.0.0.2.csv.yaml
   $ kubectl apply -f memcachedoperator.0.0.2.csv.yaml
   $ kubectl get pods
   NAME                                       READY     STATUS    RESTARTS   AGE
   memcached-app-operator-66b5777b79-pnsfj    1/1       Running   0          3s
   memcached-for-drupal-5476487c46-qbd66      1/1       Running   0          14m
   memcached-for-wordpress-65b75fd8c9-7b9x7   1/1       Running   0          14m

Getting involved

This guide provides an effective demonstration of the value of the Operator Framework for building and managing Operators, but this is much more left out in the interest of brevity. The Operator Framework and its components are open source, so visit each project individually and learn what else you can do:

github.com/operator-framework

If you want to discuss your experience, have questions, or want to get involved, join the Operator Framework mailing list.

Appendices

Operator project scaffolding layout

After creating a new Operator project using the operator-sdk new command, the project directory has numerous generated folders and files. The following table provides a basic overview of each generated file and directory.