Authenticating Kubeflow to GCP

Authentication and authorization to GCP

In-cluster authentication

When you set up Kubeflow for GCP, it will automaticallyprovision three service accountswith different privileges in the kubeflow namespace. In particular, the ${KFAPP}-user service account ismeant to grant your user services access to GCP. The credentials to this service account can be accessed withinthe cluster as a Kubernetes secret called user-gcp-sa.

The secret will have basic access to a limited set of GCP services by default, but more roles can be granted through theGCP IAM console.

Distributing Secrets

Starting from Kubeflow version 0.6, you can consume Kubeflow from custom namespaces (i.e., namespaces other than kubeflow).The kubeflow namespace is intended for running Kubeflow system components while individual jobs and model deploymentsare expected to run in separate namespaces. In order to do this you will need to install GCP credentials into the newnamespace and create a PodDefault object to attach the credentials to certain pods.

Credentials

You can add credentials to the new namespace by either copying them from an existing Kubeflow namespace or bycreating a new service account.

To copy credentials from one namespace to another namespace use the following CLI commands (Note: there is anissue filed to automate these commands):

  1. NAMESPACE=<new kubeflow namespace>
  2. SOURCE=kubeflow
  3. NAME=user-gcp-sa
  4. SECRET=$(kubectl -n ${SOURCE} get secrets ${NAME} -o jsonpath="{.data.${NAME}\.json}" | base64 -d)
  5. kubectl create -n ${NAMESPACE} secret generic ${NAME} --from-literal="${NAME}.json=${SECRET}"

To create a new service account instead of copying credentials, use the following steps:

  • Create a service account with the desired roles
  1. export PROJECT_ID=<GCP project id>
  2. export NAMESPACE=<new kubeflow namespace>
  3. export SA_NAME=<service account name>
  4. export GCPROLES=roles/editor
  5. gcloud --project=${PROJECT_ID} iam service-accounts create $SA_NAME
  6. gcloud projects add-iam-policy-binding $PROJECT_ID \
  7. --member serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
  8. --role $GCPROLES
  • Download the JSON service account key, set KEYPATH to the correct path, and create the key
  1. export KEYPATH=some/path/${SA_NAME}.gcp.json
  2. gcloud --project=${PROJECT_ID} iam service-accounts keys create ${KEYPATH} \
  3. --iam-account $SA_NAME@$PROJECT_ID.iam.gserviceaccount.com
  • Upload the JSON service account key to cluster as a secret
  1. kubectl create secret generic user-gcp-sa -n $NAMESPACE \ --from-file=user-gcp-sa.json=${KEYPATH}
PodDefault Object

The PodDefault object is a way to centrally manage configurations that should be added to all pods.

The PodDefault will match all pods with the specified selector and modify the pods to inject the volumes,secrets, and environment variables listed in the pod manifest.

Create a pod default in a file called add-gcp-secret.yaml and apply it using: kubectl apply -f add-gcp-secret.yaml -n $NAMESPACE.

  1. apiVersion: "kubeflow.org/v1alpha1"
  2. kind: PodDefault
  3. metadata:
  4.   name: add-gcp-secret
  5. spec:
  6.  selector:
  7.   matchLabels:
  8.     addgcpsecret: "true"
  9.  desc: "add gcp credential"
  10.  env:
  11.  - name: GOOGLE_APPLICATION_CREDENTIALS
  12.    value: /secret/gcp/user-gcp-sa.json
  13.  volumeMounts:
  14.  - name: secret-volume
  15.    mountPath: /secret/gcp
  16.  volumes:
  17.  - name: secret-volume
  18.    secret:
  19.     secretName: user-gcp-sa

Authentication from a Pod

You must do two things to access a GCP service account from a Pod:

  • Mount the secret as a file. This will give your Pod access to your GCP account,so be careful which Pods you grant access to.
  • Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the service account.GCP libraries will use this environment variable to find the service account and authenticate with GCP. The following YAML describes a Pod that has access to the ${KFAPP}-user service account:
  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: mypod
  5. spec:
  6.   containers:
  7.   - name: mypod
  8.     image: myimage
  9.     env:
  10.     - name: GOOGLE_APPLICATION_CREDENTIALS
  11.       value: "/var/secrets/user-sa.json"
  12.     volumeMounts:
  13.     - name: gcp-secret
  14.       mountPath: "/var/secrets/user-sa.json"
  15.       readOnly: true
  16.   volumes:
  17.   - name: gcp-secret
  18.     secret:
  19.       secretName: myappname-user

Authentication from Kubeflow Pipelines

In Kubeflow Pipelines, each step describes acontainer that is run independently. If you want to grant access for a single step to use one of your service accounts, you can usekfp.gcp.use_gcp_secret().Examples for how to use this function can be found in theKubeflow examples repo.

Local Authentication

gcloud

The gcloud tool is used to interact with Google Cloud Platform (GCP) over the command line.You can use the gcloud command to set up Google Kubernetes Engine (GKE) clusters,and interact with other Google services.

Logging in

You have two options for authenticating the gcloud command:

  • You can use a user account to authenticate using a Google account (typically Gmail).You can register a user account using gcloud auth login,which brings up a browser window to start the familiar Google authentication flow.

  • You can create a service account within your GCP project. You can thendownload a .json key fileassociated with the account, and run thegcloud auth activate-service-accountcommand to authenticate your gcloud session.

You can find more information in the GCP docs.

Listing active accounts

You can run the following command to verify you are authenticating with the expected account.In the output of the command, an asterisk denotes your active account.

  1. gcloud auth list
Viewing IAM roles

Permissions are handled in GCP using IAM Roles.These roles define which resources your account can read or write to. Provided you have thenecessary permissions,you can check which roles were assigned to your account using the following gcloud command:

  1. PROJECT_ID=your-gcp-project-id-here
  3. gcloud projects get-iam-policy $PROJECT_ID --flatten="bindings[].members" \
  4.     --format='table(bindings.role)' \
  5.     --filter="bindings.members:$(gcloud config list account --format 'value(core.account)')"

You can view and modify roles through theGCP IAM console.

You can find more information about IAM in theGCP docs.

kubectl

The kubectl tool is used for interacting with a Kubernetes cluster through the command line.

Connecting to a cluster using a GCP account

If you set up your Kubernetes cluster using GKE, you can authenticate with the cluster using a GCP account.The following commands fetch the credentials for your cluster and save them to your localkubeconfig file:

  1. CLUSTER_NAME=your-gke-cluster
  2. ZONE=your-gcp-zone
  4. gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE

You can find more information in theGCP docs.

Changing active clusters

If you work with multiple Kubernetes clusters, you may have multiple contexts saved in your localkubeconfig file.You can view the clusters you have saved by run the following command:

  1. kubectl config get-contexts

You can view which cluster is currently being controlled by kubectl with the following command:

  1. CONTEXT_NAME=your-new-context
  3. kubectl config set-context $CONTEXT_NAME

You can find more information in theKubernetes docs.

Checking RBAC permissions

Like GKE IAM, Kubernetes permissions are typically handled with a “role-based authorization control” (RBAC) system.Each Kubernetes service account has a set of authorized roles associated with it. If your account doesn’t have theright roles assigned to it, certain tasks will fail.

You can check if an account has the proper permissions to run a command by building a query structured askubectl auth can-i [VERB] [RESOURCE] —namespace [NAMESPACE]. For example, the following command will verifythat your account has permissions to create deployments in the kubeflow namespace:

  1. kubectl auth can-i create deployments --namespace kubeflow

You can find more information in theKubernetes docs.

Adding RBAC permissions

If you find you are missing a permission you need, you can grant the missing roles to your service account usingKubernetes resources.

  • Roles describe the permissions you want to assign. For example, verbs: ["create"], resources:["deployments"]
  • RoleBindings define a mapping between the Role, and a specific service account

By default, Roles and RoleBindings apply only to resources in a specific namespace, but there are alsoClusterRoles and ClusterRoleBindings that can grant access to resources cluster-wide

You can find more information in theKubernetes docs.

Next steps

See the troubleshooting guide for help with diagnosing and fixing issues you may encounter with Kubeflow on GCP