Kubernetes V2 Provider

The Kubernetes V2 Provider is the standard Kubernetes provider for Spinnaker. You can use it to deploy applications to a Kubernetes cluster.

Spinnaker’s Kubernetes provider fully supports Kubernetes-native, manifest-based deployments and is the recommended provider for deploying to Kubernetes with Spinnaker.

Accounts

A Spinnaker Account maps to a credential that can authenticate against your Kubernetes Cluster.

Prerequisites

The Kubernetes provider has two requirements:

  • A kubeconfig file

    The kubeconfig file allows Spinnaker to authenticate against your cluster and to have read/write access to any resources you expect it to manage. You can think of it as private key file to let Spinnaker connect to your cluster. You can request this from your Kubernetes cluster administrator.

  • kubectl CLI tool

    Spinnaker relies on kubectl to manage all API access. It’s installed along with Spinnaker.

    Spinnaker also relies on kubectl to access your Kubernetes cluster; only kubectl fully supports many aspects of the Kubernetes API, such as 3-way merges on kubectl apply, and API discovery. Though this creates a dependency on a binary, the good news is that any authentication method or API resource that kubectl supports is also supported by Spinnaker. This is an improvement over the original Kubernetes provider in Spinnaker.

Optional: Create a Kubernetes Service Account

If you want, you can associate Spinnaker with a Kubernetes Service Account , even when managing multiple Kubernetes clusters. This can be useful if you need to grant Spinnaker certain roles in the cluster later on, or you typically depend on an authentication mechanism that doesn’t work in all environments.

Given that you want to create a Service Account in existing context $CONTEXT, the following commands will create spinnaker-service-account, and add its token under a new user called ${CONTEXT}-token-user in context $CONTEXT.

  1. CONTEXT=$(kubectl config current-context)
  3. # This service account uses the ClusterAdmin role -- this is not necessary,
  4. # more restrictive roles can by applied.
  5. kubectl apply --context $CONTEXT \
  6.     -f /downloads/kubernetes/service-account.yml
  9. TOKEN=$(kubectl get secret --context $CONTEXT \
  10.    $(kubectl get serviceaccount spinnaker-service-account \
  11.        --context $CONTEXT \
  12.        -n spinnaker \
  13.        -o jsonpath='{.secrets[0].name}') \
  14.    -n spinnaker \
  15.    -o jsonpath='{.data.token}' | base64 --decode)
  17. kubectl config set-credentials ${CONTEXT}-token-user --token $TOKEN
  19. kubectl config set-context $CONTEXT --user ${CONTEXT}-token-user

Optional: Configure Kubernetes roles (RBAC)

If your Kubernetes cluster supports RBAC and you want to restrict permissions granted to your Spinnaker account, you will need to follow the below instructions.

The following YAML creates the correct ClusterRole, ClusterRoleBinding, and ServiceAccount. If you limit Spinnaker to operating on an explicit list of namespaces (using the namespaces option), you need to use Role and RoleBinding instead of ClusterRole and ClusterRoleBinding, and apply the Role and RoleBinding to each namespace Spinnaker manages. You can read about the difference between ClusterRole and Role in the Kubernetes docs . If you’re using RBAC to restrict the Spinnaker service account to a particular namespace, you must specify that namespace when you add the account to Spinnaker. If you don’t specify any namespaces, then Spinnaker will attempt to list all namespaces, which requires a cluster-wide role.

  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: ClusterRole
  3. metadata:
  4.   name: spinnaker-role
  5. rules:
  6.   - apiGroups: ['']
  7.     resources:
  8.       [
  9.         'namespaces',
  10.         'configmaps',
  11.         'events',
  12.         'replicationcontrollers',
  13.         'serviceaccounts',
  14.         'pods/log',
  15.       ]
  16.     verbs: ['get', 'list']
  17.   - apiGroups: ['']
  18.     resources: ['pods', 'services', 'secrets']
  19.     verbs:
  20.       [
  21.         'create',
  22.         'delete',
  23.         'deletecollection',
  24.         'get',
  25.         'list',
  26.         'patch',
  27.         'update',
  28.         'watch',
  29.       ]
  30.   - apiGroups: ['autoscaling']
  31.     resources: ['horizontalpodautoscalers']
  32.     verbs: ['list', 'get']
  33.   - apiGroups: ['apps']
  34.     resources: ['controllerrevisions']
  35.     verbs: ['list']
  36.   - apiGroups: ['extensions', 'apps']
  37.     resources: ['daemonsets', 'deployments', 'deployments/scale', 'ingresses', 'replicasets', 'statefulsets']
  38.     verbs:
  39.       [
  40.         'create',
  41.         'delete',
  42.         'deletecollection',
  43.         'get',
  44.         'list',
  45.         'patch',
  46.         'update',
  47.         'watch',
  48.       ]
  49.   # These permissions are necessary for halyard to operate. We use this role also to deploy Spinnaker itself.
  50.   - apiGroups: ['']
  51.     resources: ['services/proxy', 'pods/portforward']
  52.     verbs:
  53.       [
  54.         'create',
  55.         'delete',
  56.         'deletecollection',
  57.         'get',
  58.         'list',
  59.         'patch',
  60.         'update',
  61.         'watch',
  62.       ]
  63. ---
  64. apiVersion: rbac.authorization.k8s.io/v1
  65. kind: ClusterRoleBinding
  66. metadata:
  67.   name: spinnaker-role-binding
  68. roleRef:
  69.   apiGroup: rbac.authorization.k8s.io
  70.   kind: ClusterRole
  71.   name: spinnaker-role
  72. subjects:
  73.   - namespace: spinnaker
  74.     kind: ServiceAccount
  75.     name: spinnaker-service-account
  76. ---
  77. apiVersion: v1
  78. kind: ServiceAccount
  79. metadata:
  80.   name: spinnaker-service-account
  81.   namespace: spinnaker

Migrating from Spinnaker’s legacy Kubernetes provider

Prior to the deprecation of Spinnaker’s legacy (V1) Kubernetes provider, the standard provider was often referred to as the V2 provider. For clarity, this section refers to the providers as the V1 and V2 providers.

There is no automatic pipeline migration from the V1 provider to V2, for a few reasons:

  • Unlike the V1 provider, the V2 provider encourages you to store your Kubernetes Manifests outside of Spinnaker in some versioned, backing storage, such as Git or GCS.

  • The V2 provider encourages you to leverage the Kubernetes native deployment orchestration (e.g. Deployments ) instead of the Spinnaker red/black, where possible.

  • The initial operations available on Kubernetes manifests (e.g. scale, pause rollout, delete) in the V2 provider don’t map nicely to the operations in the V1 provider unless you contort Spinnaker abstractions to match those of Kubernetes. To avoid building dense and brittle mappings between Spinnaker’s logical resources and Kubernetes’s infrastructure resources, we chose to adopt the Kubernetes resources and operations more natively.

  • The V2 provider does not use the Docker Registry Provider .

    You may still need Docker Registry accounts to trigger pipelines, but otherwise we encourage you to stop using Docker Registry accounts in Spinnaker. The V2 provider requires that you manage your private registry configuration and authentication yourself.

However, you can easily migrate your infrastructure into the V2 provider. For any V1 account you have running, you can add a V2 account following the steps below . This will surface your infrastructure twice (once per account) helping your pipeline & operation migration.

A V1 and V2 provider surfacing the same infrastructure

A V1 and V2 provider surfacing the same infrastructure

Adding an account

First, make sure that the provider is enabled:

  1. hal config provider kubernetes enable

Then add the account:

  1. CONTEXT=$(kubectl config current-context)
  3. hal config provider kubernetes account add my-k8s-account \
  4.     --context $CONTEXT

Finally, enable artifact support .

Advanced account settings

If you’re looking for more configurability, please see the other options listed in the Halyard Reference .

Alibaba Cloud Container Service for Kubernetes (ACK) Setup

Deploy to ACK using the Kubernetes provider for Spinnaker.

Set up the Kubernetes provider for Amazon EKS

Set up Spinnaker on AWS EKS using the Kubernetes-V2 provider

Google Kubernetes Engine (GKE) Setup

Deploy to GKE using the Kubernetes provider for Spinnaker.

Kubernetes Provider

Spinnaker’s Kubernetes provider fully supports Kubernetes-native, manifest-based deployments and is the recommended provider for deploying to Kubernetes with Spinnaker.

Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) Setup

Deploy to OKE using the Kubernetes provider for Spinnaker.

Last modified November 10, 2021: Extend Kubernetes role to cover statefulsets (#153) (f458978)