Instructions for migrating a Ansible-based project built prior to v1.0.0 to use the new Kubebuilder-style layout.

Overview

The motivations for the new layout are related to bringing more flexibility to users and part of the process to Integrating Kubebuilder and Operator SDK.

What was changed

  • The deploy directory was replaced with the config directory including a new layout of Kubernetes manifests files:

    • CRD manifests in deploy/crds/ are now in config/crd/bases
    • CR manifests in deploy/crds/ are now in config/samples
    • Controller manifest deploy/operator.yaml is now in config/manager/manager.yaml
    • RBAC manifests in deploy are now in config/rbac/

  • build/Dockerfile is moved to Dockerfile in the project root directory

  • The molecule/ directory is now more aligned to Ansible and the new Layout

What is new

Scaffolded projects now use:

  • kustomize to manage Kubernetes resources needed to deploy your operator
  • A Makefile with helpful targets for build, test, and deployment, and to give you flexibility to tailor things to your project’s needs
  • Updated metrics configuration using kube-auth-proxy, a --metrics-addr flag, and kustomize-based deployment of a Kubernetes Service and prometheus operator ServiceMonitor

How to migrate

The easy migration path is to a project from the scratch and let the tool scaffold the new layout. Then, add your customizations and implementations. See below for an example.

Creating a new project

In Kubebuilder-style projects, CRD groups are defined using two different flags (--group and --domain).

When we initialize a new project, we need to specify the domain that all APIs in our project will share, so before creating the new project, we need to determine which domain we’re using for the APIs in our existing project.

To determine the domain, look at the spec.group field in your CRDs in the deploy/crds directory.

The domain is everything after the first DNS segment. Using cache.example.com as an example, the --domain would be example.com.

So let’s create a new project with the same domain (example.com):

  1. mkdir memcached-operator
  2. cd memcached-operator
  3. operator-sdk init --plugins=ansible --domain=example.com

Now that we have our new project initialized, we need to recreate each of our APIs. Using our API example from earlier (cache.example.com), we’ll use cache for the --group flag.

For --version and --kind, we use spec.versions[0].name and spec.names.kind, respectively.

For each API in the existing project, run:

  1. operator-sdk create api \
  2.     --group=cache \
  3.     --version=v1 \
  4.     --kind=Memcached

Running the above command creates an empty roles/<kind>. We can copy over the content of our old roles/<kind> to the new one.

Migrating your Custom Resource samples

Update the CR manifests in config/samples with the values of the CRs in your existing project which are in deploy/crds/<group>_<version>_<kind>_cr.yaml In our example the config/samples/cache_v1alpha1_memcached.yaml will look like:

  1. apiVersion: cache.example.com/v1alpha1
  2. kind: Memcached
  3. metadata:
  4.   name: memcached-sample
  5. spec:
  6.   # Add fields here
  7.   size: 3

Migrating watches.yaml

Update the watches.yaml file with your roles/playbooks and check if you have custom options in the watches.yaml file of your existing project. If so, update the new watches.yaml file to match.

In our example, we will replace # FIXME: Specify the role or playbook for this resource. with our previous role and it will look like:

  1. ---
  2. # Use the 'create api' subcommand to add watches to this file.
  3. - version: v1alpha1
  4.   group: cache.example.com
  5.   kind: Memcached
  6.   role: memcached
  7. #+kubebuilder:scaffold:watch

NOTE: Do not remove the +kubebuilder:scaffold:watch marker. It allows the tool to update the watches file when new APIs are created.

Migrating your Molecule tests

If you are using Molecule in your project will be required to port the tests for the new layout.

See that default structure changed from:

  1. ├── cluster
  2. │   ├── converge.yml
  3. │   ├── create.yml
  4. │   ├── destroy.yml
  5. │   ├── molecule.yml
  6. │   ├── prepare.yml
  7. │   └── verify.yml
  8. ├── default
  9. │   ├── converge.yml
  10. │   ├── molecule.yml
  11. │   ├── prepare.yml
  12. │   └── verify.yml
  13. ├── templates
  14. │   └── operator.yaml.j2
  15. └── test-local
  16.     ├── converge.yml
  17.     ├── molecule.yml
  18.     ├── prepare.yml
  19.     └── verify.yml

To:

  1. ├── default
  2. │   ├── converge.yml
  3. │   ├── create.yml
  4. │   ├── destroy.yml
  5. │   ├── kustomize.yml
  6. │   ├── molecule.yml
  7. │   ├── prepare.yml
  8. │   ├── tasks
  9. │   │   └── foo_test.yml
  10. │   └── verify.yml
  11. └── kind
  12.     ├── converge.yml
  13.     ├── create.yml
  14.     ├── destroy.yml
  15.     └── molecule.yml

Ensure that the provisioner.host_vars.localhost has the following host_vars:

  1. ....
  2.     host_vars:
  3.       localhost:
  4.         ansible_python_interpreter: '{{ ansible_playbook_python }}'
  5.         config_dir: ${MOLECULE_PROJECT_DIRECTORY}/config
  6.         samples_dir: ${MOLECULE_PROJECT_DIRECTORY}/config/samples
  7.         operator_image: ${OPERATOR_IMAGE:-""}
  8.         operator_pull_policy: ${OPERATOR_PULL_POLICY:-"Always"}
  9.         kustomize: ${KUSTOMIZE_PATH:-kustomize}
  10. ...

For more information read the Testing with Molecule.

Checking the Permissions (RBAC)

In your new project, roles are automatically generated in config/rbac/role.yaml. If you modified these permissions manually in deploy/role.yaml in your existing project, you need to re-apply them in config/rbac/role.yaml.

New projects are configured to watch all namespaces by default, so they need a ClusterRole to have the necessary permissions. Ensure that config/rbac/role.yaml remains a ClusterRole if you want to retain the default behavior of the new project conventions.

The following rules were used in earlier versions of ansible-operator to automatically create and manage services and servicemonitors for metrics collection. If your operator’s don’t require these rules, they can safely be left out of the new config/rbac/role.yaml file:

  1.   - apiGroups:
  2.     - monitoring.coreos.com
  3.     resources:
  4.     - servicemonitors
  5.     verbs:
  6.     - get
  7.     - create
  8.   - apiGroups:
  9.     - apps
  10.     resourceNames:
  11.     - memcached-operator
  12.     resources:
  13.     - deployments/finalizers
  14.     verbs:
  15.     - update

Configuring your Operator

If your existing project has customizations in deploy/operator.yaml, they need to be ported to config/manager/manager.yaml. If you are passing custom arguments in your deployment, make sure to also update config/default/auth_proxy_patch.yaml.

Note that the following environment variables are no longer used.

  • OPERATOR_NAME is deprecated. It is used to define the name for a leader election config map. Operator authors should begin using --leader-election-id instead.
  • POD_NAME has been removed. It was used to enable a particular pod to hold the leader election lock when the Ansible operator used the leader for life mechanism. Ansible operator now uses controller-runtime’s leader with lease mechanism.

Exporting metrics

If you are using metrics and would like to keep them exported you will need to configure it in the config/default/kustomization.yaml. Please see the metrics doc to know how you can perform this setup.

The default port used by the metric endpoint binds to was changed from :8383 to :8080. To continue using port 8383, specify --metrics-addr=:8383 when you start the operator.

Checking the changes

Finally, follow the steps in the “run the Operator” section to verify your project is running.

Note that, you also can troubleshooting by checking the container logs. E.g $ kubectl logs deployment.apps/memcached-operator-controller-manager -n memcached-operator-system -c manager

Last modified February 11, 2021: align the sdk with kb (#4402) (4fc8a17c)