Running your Operator with OLM

The Operator Lifecycle Manager (OLM) is a set of cluster resources that manage the lifecycle of an Operator. The Operator SDK provides an entrypoint for deploying and deleting your Operator using an OLM-enabled Kubernetes cluster through operator-sdk run --olm, added in v0.15.0

This document assumes you are familiar with OLM and related terminology, and have read the SDK-OLM integration design proposal.

Note: before continuing, please read the caveats section below.

Setup

The SDK assumes OLM is already installed and running on your cluster. If not, you can install OLM by running operator-sdk olm install. You can check the status of OLM by running operator-sdk olm status.

The SDK also assumes you have a valid Operator bundle available on disk. If not, you can create one using operator-sdk generate csv --update-crds, which will create a ClusterServiceVersion (CSV) in a versioned bundle directory and a package manifest defining your Operator package, and copy your CustomResourceDefinition‘s to the new bundle directory.

operator-sdk run --olm command overview

Let’s look at the anatomy of the command:

  1. $ operator-sdk run --help
  2. Run an Operator in a variety of environments
  4. Usage:
  5.   operator-sdk run [flags]
  7. Flags:
  8.       --kubeconfig string           The file path to kubernetes configuration file. Defaults to location specified by $KUBECONFIG, or to default file rules if not set
  9.       --olm                         The operator to be run will be managed by OLM in a cluster. Cannot be set with another run-type flag
  10.       --olm-namespace string        [olm only] The namespace where OLM is installed (default "olm")
  11.       --operator-namespace string   [olm only] The namespace where operator resources are created in --olm mode. It must already exist in the cluster or be defined in a manifest passed to IncludePaths.
  12.       --manifests string            [olm only] Directory containing package manifest and operator bundles.
  13.       --operator-version string     [olm only] Version of operator to deploy
  14.       --install-mode string         [olm only] InstallMode to create OperatorGroup with. Format: InstallModeType=[ns1,ns2[, ...]]
  15.       --include strings             [olm only] Path to Kubernetes resource manifests, ex. Role, Subscription. These supplement or override defaults generated by run/cleanup
  16.       --timeout duration            [olm only] Time to wait for the command to complete before failing (default 2m0s)
  17.       ...

These flags correspond to options in the OLMCmd configuration model (names are similar if not identical):

  1. type OLMCmd struct {
  2.   // KubeconfigPath is the local path to a kubeconfig. This uses well-defined
  3.   // default loading rules to load the config if empty.
  4.   KubeconfigPath string
  5.   // OLMNamespace is the namespace in which OLM is installed.
  6.   OLMNamespace string
  7.   // OperatorNamespace is the cluster namespace in which operator resources
  8.   // are created.
  9.   // OperatorNamespace must already exist in the cluster or be defined in
  10.   // a manifest passed to IncludePaths.
  11.   OperatorNamespace string
  12.     // ManifestsDir is a directory containing a package manifest and N bundles
  13.     // of the operator's CSV and CRD's. OperatorVersion can be set to the
  14.     // version of the desired operator version's subdir and Run()/Cleanup() will
  15.     // deploy the operator version in that subdir.
  16.     ManifestsDir string
  17.     // OperatorVersion is the version of the operator to deploy. It must be
  18.     // a semantic version, ex. 0.1.0.
  19.     OperatorVersion string
  20.     // InstallMode specifies which supported installMode should be used to
  21.     // create an OperatorGroup. The format for this field is as follows:
  22.     //
  23.     // "InstallModeType=[ns1,ns2[, ...]]"
  24.     //
  25.     // The InstallModeType string passed must be marked as "supported" in the
  26.     // CSV being installed. The namespaces passed must exist or be created by
  27.     // passing a Namespace manifest to IncludePaths. An empty set of namespaces
  28.     // can be used for AllNamespaces.
  29.     // The default mode is OwnNamespace, which uses OperatorNamespace or the
  30.     // kubeconfig default.
  31.     InstallMode string
  32.   // IncludePaths are path to manifests of Kubernetes resources that either
  33.     // supplement or override defaults generated by methods of OLMCmd. These
  34.     // manifests can be but are not limited to: RBAC, Subscriptions,
  35.     // CatalogSources, OperatorGroups.
  36.     //
  37.     // Kinds that are overridden if supplied:
  38.     // - CatalogSource
  39.     // - Subscription
  40.     // - OperatorGroup
  41.     IncludePaths []string
  42.     // Timeout dictates how long to wait for a REST call to complete. A call
  43.     // exceeding Timeout will generate an error.
  44.     Timeout time.Duration
  45. }

Most of the above configuration options do not need explanation beyond their descriptions. The following options require more clarification:

  • --operator-version, on top of being a semantic version string corresponding to a CSV version, should point to the directory containing your Operator bundle. This path is deploy/olm-catalog/<operator-name>/<operator-version> in an SDK Operator project.
  • --include can be used if you have an existing set of manifests outside your bundle (ex. catalog manifests like a Subscription, CatalogSource, and/or OperatorGroup) you wish to facilitate Operator deployment with.
    • Currently any manifests supplied to this command will be created with the same behavior of kubectl create -f <manifest-path>.
    • If a Subscription or CatalogSource are supplied, the other must be supplied since they are linked by field references.
  • --install-mode configures the spec.targetNamespaces field of an OperatorGroup, and supports the following modes (assuming your CSV does as well):
    • OwnNamespace: the Operator will watch its own namespace. This is the default.
    • SingleNamespace: the Operator will watch a namespace, not necessarily its own.
    • MultiNamespace: the Operator will watch multiple namespaces. Example: --install-mode=MultiNamespace=my-ns-1,my-ns-2 will watch resources in namespaces my-ns-1 and my-ns-2.
    • AllNamespaces: the Operator will watch all namespaces (cluster-scoped Operators). Example: --install-mode=AllNamespaces="" will watch resources in all namespaces given

Examples

Setup

Lets walk through creating a memcached-operator and enabling OLM on your cluster.

Follow the user guides for Go, Ansible, or Helm, depending on which Operator type you are interested in.

Now that we have a working, simple Operator, we can generate a new v0.1.0 bundle:

Note: operator-sdk generate csv only officially supports Go Operators, although it will generate a barebones CSV for Ansible and Helm Operators that will require manual modification.

  1. $ operator-sdk generate csv --csv-version 0.1.0 --update-crds
  2. INFO[0000] Generating CSV manifest version 0.1.0
  3. INFO[0004] Required csv fields not filled in file deploy/olm-catalog/memcached-operator/0.1.0/memcached-operator.v0.1.0.clusterserviceversion.yaml:
  4.     spec.keywords
  5.     spec.provider
  6. INFO[0004] Created deploy/olm-catalog/memcached-operator/0.1.0/memcached-operator.v0.1.0.clusterserviceversion.yaml

A bundle directory containing a CSV and all CRDs in deploy/crds and package manifest have been created at deploy/olm-catalog/memcached-operator/0.1.0 and deploy/olm-catalog/memcached-operator/memcached-operator.package.yaml, respectively.

Ensure OLM is enabled on your cluster:

  1. # If OLM is not already installed, go ahead and install the latest version.
  2. $ operator-sdk olm install
  3. INFO[0000] Fetching CRDs for version "latest"           
  4. INFO[0001] Fetching resources for version "latest"      
  5. INFO[0007] Creating CRDs and resources                  
  6. INFO[0007]   Creating CustomResourceDefinition "clusterserviceversions.operators.coreos.com"
  7. INFO[0007]   Creating CustomResourceDefinition "installplans.operators.coreos.com"
  8. INFO[0007]   Creating CustomResourceDefinition "subscriptions.operators.coreos.com"
  9. INFO[0007]   Creating CustomResourceDefinition "catalogsources.operators.coreos.com"
  10. INFO[0007]   Creating CustomResourceDefinition "operatorgroups.operators.coreos.com"
  11. INFO[0007]   Creating Namespace "olm"                   
  12. INFO[0007]   Creating Namespace "operators"             
  13. INFO[0007]   Creating ServiceAccount "olm/olm-operator-serviceaccount"
  14. INFO[0007]   Creating ClusterRole "system:controller:operator-lifecycle-manager"
  15. INFO[0007]   Creating ClusterRoleBinding "olm-operator-binding-olm"
  16. INFO[0007]   Creating Deployment "olm/olm-operator"     
  17. INFO[0007]   Creating Deployment "olm/catalog-operator"
  18. INFO[0007]   Creating ClusterRole "aggregate-olm-edit"  
  19. INFO[0007]   Creating ClusterRole "aggregate-olm-view"  
  20. INFO[0007]   Creating OperatorGroup "operators/global-operators"
  21. INFO[0011]   Creating OperatorGroup "olm/olm-operators"
  22. INFO[0011]   Creating ClusterServiceVersion "olm/packageserver"
  23. INFO[0011]   Creating CatalogSource "olm/operatorhubio-catalog"
  24. INFO[0011] Waiting for deployment/olm-operator rollout to complete
  25. INFO[0011]   Waiting for Deployment "olm/olm-operator" to rollout: 0 of 1 updated replicas are available
  26. INFO[0013]   Deployment "olm/olm-operator" successfully rolled out
  27. INFO[0013] Waiting for deployment/catalog-operator rollout to complete
  28. INFO[0013]   Waiting for Deployment "olm/catalog-operator" to rollout: 0 of 1 updated replicas are available
  29. INFO[0018]   Deployment "olm/catalog-operator" successfully rolled out
  30. INFO[0018] Waiting for deployment/packageserver rollout to complete
  31. INFO[0018]   Waiting for Deployment "olm/packageserver" to rollout: 1 out of 2 new replicas have been updated
  32. INFO[0023]   Waiting for Deployment "olm/packageserver" to rollout: 1 old replicas are pending termination
  33. INFO[0030]   Deployment "olm/packageserver" successfully rolled out
  34. INFO[0030] Successfully installed OLM version "latest"  
  36. NAME                                            NAMESPACE    KIND                        STATUS
  37. clusterserviceversions.operators.coreos.com                  CustomResourceDefinition    Installed
  38. installplans.operators.coreos.com                            CustomResourceDefinition    Installed
  39. subscriptions.operators.coreos.com                           CustomResourceDefinition    Installed
  40. catalogsources.operators.coreos.com                          CustomResourceDefinition    Installed
  41. operatorgroups.operators.coreos.com                          CustomResourceDefinition    Installed
  42. olm                                                          Namespace                   Installed
  43. operators                                                    Namespace                   Installed
  44. olm-operator-serviceaccount                     olm          ServiceAccount              Installed
  45. system:controller:operator-lifecycle-manager                 ClusterRole                 Installed
  46. olm-operator-binding-olm                                     ClusterRoleBinding          Installed
  47. olm-operator                                    olm          Deployment                  Installed
  48. catalog-operator                                olm          Deployment                  Installed
  49. aggregate-olm-edit                                           ClusterRole                 Installed
  50. aggregate-olm-view                                           ClusterRole                 Installed
  51. global-operators                                operators    OperatorGroup               Installed
  52. olm-operators                                   olm          OperatorGroup               Installed
  53. packageserver                                   olm          ClusterServiceVersion       Installed
  54. operatorhubio-catalog                           olm          CatalogSource               Installed
  56. # Check OLM's status.
  57. $ operator-sdk olm status
  58. INFO[0000] Fetching CRDs for version "0.14.1"           
  59. INFO[0002] Fetching resources for version "0.14.1"      
  60. INFO[0002] Successfully got OLM status for version "0.14.1"
  62. NAME                                            NAMESPACE    KIND                        STATUS
  63. olm                                                          Namespace                   Installed
  64. operatorgroups.operators.coreos.com                          CustomResourceDefinition    Installed
  65. catalogsources.operators.coreos.com                          CustomResourceDefinition    Installed
  66. subscriptions.operators.coreos.com                           CustomResourceDefinition    Installed
  67. installplans.operators.coreos.com                            CustomResourceDefinition    Installed
  68. aggregate-olm-edit                                           ClusterRole                 Installed
  69. catalog-operator                                olm          Deployment                  Installed
  70. olm-operator                                    olm          Deployment                  Installed
  71. operatorhubio-catalog                           olm          CatalogSource               Installed
  72. olm-operators                                   olm          OperatorGroup               Installed
  73. aggregate-olm-view                                           ClusterRole                 Installed
  74. operators                                                    Namespace                   Installed
  75. global-operators                                operators    OperatorGroup               Installed
  76. olm-operator-serviceaccount                     olm          ServiceAccount              Installed
  77. packageserver                                   olm          ClusterServiceVersion       Installed
  78. system:controller:operator-lifecycle-manager                 ClusterRole                 Installed
  79. clusterserviceversions.operators.coreos.com                  CustomResourceDefinition    Installed
  80. olm-operator-binding-olm                                     ClusterRoleBinding          Installed

Note: certain cluster types may already have OLM enabld, but under a non-default ("olm") namespace. Set --olm-namespac=[non-default-olm-namespace] for both operator-sdk olm and operator-sdk run commands if this is the case.

Simple Operator

Assuming everything your Operator needs to successfully deploy and run exists in your bundle directory (no Secret‘s, etc.), you can deploy your Operator using the following command invocation:

  1. $ operator-sdk run --olm --manifests deploy/olm-catalog/memcached-operator --operator-version 0.1.0
  2. INFO[0000] loading Bundles                               dir=deploy/olm-catalog/memcached-operator
  3. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=memcached-operator load=bundles
  4. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=0.1.0 load=bundles
  5. INFO[0000] found csv, loading bundle                     dir=deploy/olm-catalog/memcached-operator file=memcached-operator.v0.1.0.clusterserviceversion.yaml load=bundles
  6. INFO[0000] loading bundle file                           dir=deploy/olm-catalog/memcached-operator/0.1.0 file=example.com_memcacheds_crd.yaml load=bundle
  7. INFO[0000] loading bundle file                           dir=deploy/olm-catalog/memcached-operator/0.1.0 file=memcached-operator.v0.1.0.clusterserviceversion.yaml load=bundle
  8. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=metadata load=bundles
  9. INFO[0000] loading Packages and Entries                  dir=deploy/olm-catalog/memcached-operator
  10. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=memcached-operator load=package
  11. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=0.1.0 load=package
  12. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=metadata load=package
  13. INFO[0000] Creating registry                            
  14. INFO[0000]   Creating ConfigMap "olm/memcached-operator-registry-bundles"
  15. INFO[0000]   Creating Deployment "olm/memcached-operator-registry-server"
  16. INFO[0000]   Creating Service "olm/memcached-operator-registry-server"
  17. INFO[0000] Waiting for Deployment "olm/memcached-operator-registry-server" rollout to complete
  18. INFO[0000]   Waiting for Deployment "olm/memcached-operator-registry-server" to rollout: 0 out of 1 new replicas have been updated
  19. INFO[0001]   Waiting for Deployment "olm/memcached-operator-registry-server" to rollout: 0 of 1 updated replicas are available
  20. INFO[0002]   Deployment "olm/memcached-operator-registry-server" successfully rolled out
  21. INFO[0002] Creating resources                           
  22. INFO[0002]   Creating CatalogSource "default/memcached-operator-ocs"
  23. INFO[0002]   Creating Subscription "default/memcached-operator-v0-0-1-sub"
  24. INFO[0002]   Creating OperatorGroup "default/operator-sdk-og"
  25. INFO[0002] Waiting for ClusterServiceVersion "default/memcached-operator.v0.1.0" to reach 'Succeeded' phase
  26. INFO[0002]   Waiting for ClusterServiceVersion "default/memcached-operator.v0.1.0" to appear
  27. INFO[0034]   Found ClusterServiceVersion "default/memcached-operator.v0.1.0" phase: Pending
  28. INFO[0035]   Found ClusterServiceVersion "default/memcached-operator.v0.1.0" phase: InstallReady
  29. INFO[0036]   Found ClusterServiceVersion "default/memcached-operator.v0.1.0" phase: Installing
  30. INFO[0036]   Found ClusterServiceVersion "default/memcached-operator.v0.1.0" phase: Succeeded
  31. INFO[0037] Successfully installed "memcached-operator.v0.1.0" on OLM version "0.14.1"
  33. NAME                            NAMESPACE    KIND                        STATUS
  34. memcacheds.cache.example.com    default      CustomResourceDefinition    Installed
  35. memcached-operator.v0.1.0       default      ClusterServiceVersion       Installed

This command can be further simplified by passing only --operator-version=0.1.0 and letting the command infer the bundle path.

Multi-namespaced/cluster-wide deployment

Lets say you modify your CSV’s installModes to support MultiNamespace. Now you can deploy your Operator such that it watches multiple namespaces ns-1, ns-2, and ns-3:

  1. $ operator-sdk run --olm --operator-version 0.1.0 --install-mode-MultiNamespace=ns-1,ns-2,ns-3

Supplying your own catalog manifests

I now have a reasonably complicated set of catalog requirements, ex. multiple OperatorGroup‘s in my cluster, and want full control over how the SDK tells OLM to deploy my operator.

These manifests are in deploy/catalog-manifests:

  1. $ ls -l deploy/catalog-manifests
  2. -rw-rw-r--. 1 user user  2368 Mar 24 19:21 special-operatorgroup.yaml
  3. -rw-rw-r--. 1 user user  3264 Mar 24 19:23 special-subscription.yaml
  4. -rw-rw-r--. 1 user user  5117 Mar 24 18:22 special-catalogsource.yaml

Instead of operator-sdk run --olm creating these manifests for my Operator, I can direct the command to create the above manifests:

  1. $ operator-sdk run --olm --operator-version 0.1.0 --include special-operatorgroup.yaml,special-subscription.yaml,special-catalogsource.yaml
  2. INFO[0000] loading Bundles                               dir=deploy/olm-catalog/memcached-operator
  3. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=memcached-operator load=bundles
  4. ...
  5. INFO[0002] Creating resources                           
  6. INFO[0002]   Creating OperatorGroup "default/special-operatorgroup"
  7. INFO[0002]   Creating CatalogSource "default/special-subscription"
  8. INFO[0002]   Creating Subscription "default/special-catalogsource"
  9. ...

Cleaning up

I was testing my Operator, and now I would like to remove it from your cluster in a given namespace. The operator-sdk cleanup --olm command will do this for you:

  1. $ operator-sdk cleanup --olm --operator-version 0.1.0
  2. INFO[0000] loading Bundles                               dir=deploy/olm-catalog/memcached-operator
  3. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=memcached-operator load=bundles
  4. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=0.1.0 load=bundles
  5. INFO[0000] found csv, loading bundle                     dir=deploy/olm-catalog/memcached-operator file=memcached-operator.v0.1.0.clusterserviceversion.yaml load=bundles
  6. INFO[0000] loading bundle file                           dir=deploy/olm-catalog/memcached-operator/0.1.0 file=example.com_memcacheds_crd.yaml load=bundle
  7. INFO[0000] loading bundle file                           dir=deploy/olm-catalog/memcached-operator/0.1.0 file=memcached-operator.v0.1.0.clusterserviceversion.yaml load=bundle
  8. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=metadata load=bundles
  9. INFO[0000] loading Packages and Entries                  dir=deploy/olm-catalog/memcached-operator
  10. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=memcached-operator load=package
  11. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=0.1.0 load=package
  12. INFO[0000] directory                                     dir=deploy/olm-catalog/memcached-operator file=metadata load=package
  13. INFO[0000] Deleting resources                           
  14. INFO[0000]   Deleting CatalogSource "default/memcached-operator-ocs"
  15. INFO[0000]   Deleting Subscription "default/memcached-operator-v0-0-1-sub"
  16. INFO[0000]   Deleting OperatorGroup "default/operator-sdk-og"
  17. INFO[0000]   Deleting CustomResourceDefinition "default/memcacheds.example.com"
  18. INFO[0000]   Deleting ClusterServiceVersion "default/memcached-operator.v0.1.0"
  19. INFO[0000]   Waiting for deleted resources to disappear
  20. INFO[0001] Successfully uninstalled "memcached-operator.v0.1.0" on OLM version "0.14.1"

Caveats

  • operator-sdk run (and operator-sdk cleanup) are intended to be used for testing purposes only as of now, since this command creates a transient image registry that should not be used in production. Typically a registry is deployed eparately and a set of catalog manifests are created in the cluster to inform OLM of that registry and which Operator versions it can deploy and where to deploy the Operator.
  • operator-sdk run can only deploy one Operator and one version of that Operator at a time, hence its intended purpose being testing only. Production OLM usage requires the typical setup described above.

Last modified January 1, 0001