Authenticating Kubeflow to Google Cloud

Authentication and authorization to Google Cloud

This page describes in-cluster and local authentication for Kubeflow Google Cloud deployments.

In-cluster authentication

Starting from Kubeflow v0.6, you consume Kubeflow from custom namespaces (that is, namespaces other than kubeflow). The kubeflow namespace is only for running Kubeflow system components. Individual jobs and model deployments run in separate namespaces.

Google Kubernetes Engine (GKE) workload identity

Starting in v0.7, Kubeflow uses the new GKE feature: workload identity. This is the recommended way to access Google Cloud APIs from your GKE cluster. You can configure a Kubernetes service account (KSA) to act as a Google Cloud service account (GSA).

If you deployed Kubeflow following the Google Cloud instructions, then the profiler controller automatically binds the “default-editor” service account for every profile namespace to a default Google Cloud service account created during kubeflow deployment. The Kubeflow deployment process also creates a default profile for the cluster admin.

For more info about profiles see the Multi-user isolation page.

Here is an example profile spec:

  1. apiVersion: kubeflow.org/v1beta1
  2. kind: Profile
  3. spec:
  4.   plugins:
  5.   - kind: WorkloadIdentity
  6.     spec:
  7.       gcpServiceAccount: ${SANAME}@${PROJECT}.iam.gserviceaccount.com
  8. ...

You can verify that there is a KSA called default-editor and that it has an annotation of the corresponding GSA:

  1. kubectl -n ${PROFILE_NAME} describe serviceaccount default-editor
  3. ...
  4. Name:        default-editor
  5. Annotations: iam.gke.io/gcp-service-account: ${KFNAME}-user@${PROJECT}.iam.gserviceaccount.com
  6. ...

You can double check that GSA is also properly set up:

  1. gcloud --project=${PROJECT} iam service-accounts get-iam-policy ${KFNAME}-user@${PROJECT}.iam.gserviceaccount.com

When a pod uses KSA default-editor, it can access Google Cloud APIs with the role granted to the GSA.

Provisioning custom Google service accounts in namespaces: When creating a profile, you can specify a custom Google Cloud service account for the namespace to control which Google Cloud resources are accessible.

Prerequisite: you must have permission to edit your Google Cloud project’s IAM policy and to create a profile custom resource (CR) in your Kubeflow cluster.

  1. if you don’t already have a Google Cloud service account you want to use, create a new one. For example: user1-gcp@<project-id>.iam.gserviceaccount.com:
  1. gcloud iam service-accounts create user1-gcp@<project-id>.iam.gserviceaccount.com
  1. You can bind roles to the Google Cloud service account to allow access to the desired Google Cloud resources. For example to run BigQuery job, you can grant access like so:
  1. gcloud projects add-iam-policy-binding <project-id> \
  2.       --member='serviceAccount:user1-gcp@<project-id>.iam.gserviceaccount.com' \
  3.       --role='roles/bigquery.jobUser'
  1. Grant owner permission of service account user1-gcp@<project-id>.iam.gserviceaccount.com to cluster account <cluster-name>-admin@<project-id>.iam.gserviceaccount.com:
  1. gcloud iam service-accounts add-iam-policy-binding \
  2.       user1-gcp@<project-id>.iam.gserviceaccount.com \
  3.       --member='serviceAccount:<cluster-name>-admin@<project-id>.iam.gserviceaccount.com' --role='roles/owner'
  1. Manually create a profile for user1 and specify the Google Cloud service account to bind in plugins field:
  1. apiVersion: kubeflow.org/v1beta1
  2. kind: Profile
  3. metadata:
  4.   name: profileName   # replace with the name of the profile (the user's namespace name)
  5. spec:
  6.   owner:
  7.     kind: User
  8.     name: user1@email.com   # replace with the email of the user
  9.   plugins:
  10.   - kind: WorkloadIdentity
  11.     spec:
  12.       gcpServiceAccount: user1-gcp@project-id.iam.gserviceaccount.com

Note: The profile controller currently doesn’t perform any access control checks to see whether the user creating the profile should be able to use the Google Cloud service account. As a result, any user who can create a profile can get access to any service account for which the admin controller has owner permissions. We will improve this in subsequent releases.

You can find more details on workload identity in the GKE documentation.

Authentication from Kubeflow Pipelines

Starting from Kubeflow v1.1, Kubeflow Pipelines supports multi-user isolation. Therefore, pipeline runs are executed in user namespaces also using the default-editor KSA.

Additionally, the Kubeflow Pipelines UI, visualization, and TensorBoard server instances are deployed in your user namespace using the default-editor KSA. Therefore, to visualize results in the Pipelines UI, they can fetch artifacts in Google Cloud Storage using permissions of the same GSA you configured for this namespace.

For more details, refer to Authenticating Pipelines to Google Cloud.

Local authentication

gcloud

Use the gcloud tool to interact with Google Cloud on the command line. You can use the gcloud command to set up Google Kubernetes Engine (GKE) clusters, and interact with other Google services.

Logging in

You have two options for authenticating the gcloud command:

  • You can use a user account to authenticate using a Google account (typically Gmail). You can register a user account using gcloud auth login, which brings up a browser window to start the familiar Google authentication flow.

  • You can create a service account within your Google Cloud project. You can then download a .json key file associated with the account, and run the gcloud auth activate-service-account command to authenticate your gcloud session.

You can find more information in the Google Cloud docs.

Listing active accounts

You can run the following command to verify you are authenticating with the expected account:

  1. gcloud auth list

In the output of the command, an asterisk denotes your active account.

Viewing IAM roles

Permissions are handled in Google Cloud using IAM Roles. These roles define which resources your account can read or write to. Provided you have the necessary permissions, you can check which roles were assigned to your account using the following gcloud command:

  1. PROJECT_ID=your-gcp-project-id-here
  3. gcloud projects get-iam-policy $PROJECT_ID --flatten="bindings[].members" \
  4.     --format='table(bindings.role)' \
  5.     --filter="bindings.members:$(gcloud config list account --format 'value(core.account)')"

You can view and modify roles through the Google Cloud IAM console.

You can find more information about IAM in the Google Cloud docs.

kubectl

The kubectl tool is used for interacting with a Kubernetes cluster through the command line.

Connecting to a cluster using a Google Cloud account

If you set up your Kubernetes cluster using GKE, you can authenticate with the cluster using a Google Cloud account. The following commands fetch the credentials for your cluster and save them to your local kubeconfig file:

  1. CLUSTER_NAME=your-gke-cluster
  2. ZONE=your-gcp-zone
  4. gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE

You can find more information in the Google Cloud docs.

Changing active clusters

If you work with multiple Kubernetes clusters, you may have multiple contexts saved in your local kubeconfig file. You can view the clusters you have saved by run the following command:

  1. kubectl config get-contexts

You can view which cluster is currently being controlled by kubectl with the following command:

  1. CONTEXT_NAME=your-new-context
  3. kubectl config set-context $CONTEXT_NAME

You can find more information in the Kubernetes docs.

Checking RBAC permissions

Like GKE IAM, Kubernetes permissions are typically handled with a “role-based authorization control” (RBAC) system. Each Kubernetes service account has a set of authorized roles associated with it. If your account doesn’t have the right roles assigned to it, certain tasks fail.

You can check if an account has the proper permissions to run a command by building a query structured as kubectl auth can-i [VERB] [RESOURCE] --namespace [NAMESPACE]. For example, the following command verifies that your account has permissions to create deployments in the kubeflow namespace:

  1. kubectl auth can-i create deployments --namespace kubeflow

You can find more information in the Kubernetes docs.

Adding RBAC permissions

If you find you are missing a permission you need, you can grant the missing roles to your service account using Kubernetes resources.

  • Roles describe the permissions you want to assign. For example, verbs: ["create"], resources:["deployments"]
  • RoleBindings define a mapping between the Role, and a specific service account

By default, Roles and RoleBindings apply only to resources in a specific namespace, but there are also ClusterRoles and ClusterRoleBindings that can grant access to resources cluster-wide

You can find more information in the Kubernetes docs.

Next steps

See the troubleshooting guide for help with diagnosing and fixing issues you may encounter with Kubeflow on Google Cloud

Last modified 13.05.2021: (GKE) Update GKE related outdated docs (#2696) (0cb01ff8)