Getting Started with Multi-user Isolation

How to use multi-user isolation with profiles

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Usage overview

After Kubeflow is installed and configured, you will by default be accessing your primary profile. A profile owns a Kubernetes namespace of the same name along with a collection of Kubernetes resources. Users have view and modify access to their primary profiles. You can share access to your profile with another user in the system. When sharing the access to a profile with another user, you can choose to whether to provide only read access or read/modify access. For all practical purposes when working through the Kubeflow central dashboard, the active namespace is directly tied with the active profile.

Example of usage

You can select your active profile from the top bar on the Kubeflow central dashboard. Note that you can only see the profiles to which you have view or modify access.

Select active profile

This guide illustrates the user isolation functionality using the Jupyter notebooks service which is the first service in the system to have full integration with the multi-user isolation functionality.

After you select an active profile, the Notebooks Servers UI displays only the active notebook servers in the currently selected profile. All other notebook servers remain hidden from you. If you switch the active profile, the view switches the list of active notebooks appropriately. You can connect to any of the listed notebook servers and view and modify the existing Jupyter notebooks available in the server.

For example, the following image shows the list of notebook servers available in a user’s primary profile:

List of notebooks in active profile

When an unauthorized user accesses the notebooks in this profile, they see an error:

Error listing notebooks in inacessible profile

When you create Jupyter notebook servers from the Notebooks Servers UI, the notebook pods are created in your active profile. If you don’t have modify access to the active profile, you can only browse currently active notebook servers and access the existing notebooks but cannot create new notebook servers in that profile. You can create notebook servers in your primary profile which you have view and modify access to.

Onboarding a new user

Kubeflow v1.2.0 provides automatic profile creation for authenticated users on first login. Additionally, an administrator can create a profile for any user in the Kubeflow cluster. Here an administrator is a person who has cluster-admin role binding in the Kubernetes cluster. This person has permissions to create and modify Kubernetes resources in the cluster. For example, the person who deployed Kubeflow will have administration privileges in the cluster.

Pre-requisites: grant user minimal Kubernetes cluster access

You must grant each user the minimal permission scope that allows them to connect to the Kubernetes cluster.

For example, for Google Cloud users, you should grant the following Cloud Identity and Access Management (IAM) roles. In the following commands, replace [PROJECT] with your Google Cloud project and replace [EMAIL] with the user’s email address:

  • To access the Kubernetes cluster, the user needs the Kubernetes Engine Cluster Viewer role:

    1. gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/container.clusterViewer

  • To access the Kubeflow UI through IAP, the user needs the IAP-secured Web App User role:

    1. gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/iap.httpsResourceAccessor

    Note: you need to grant the user IAP-secured Web App User role even if the user is already an owner or editor of the project. IAP-secured Web App User role is not implied by just Project Owner or Project Editor roles.

  • To be able to run gcloud get credentials and see logs in Cloud Logging (formerly Stackdriver), the user needs viewer access on the project:

    1. gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/viewer

Automatic creation of profiles

Kubeflow v1.2.0 provides automatic profile creation:

  • The Kubeflow deployment process automatically creates a profile for the user performing the deployment. When the user access the Kubeflow central dashboard they see their profile in the dropdown list.
  • The automatic profile creation can be disabled as part of the deployment by setting the registration-flow env variable to false. And an admin can manually create profiles per user or per project and add collaborators through YAML files. Modify the kustomize/centraldashboard/base/parama.env to set the registration variable to false
  1. clusterDomain=cluster.local
  2. userid-header=kubeflow-userid
  3. userid-prefix=
  4. registration-flow=false
  • When an authenticated user logs into the system and visits the central dashboard for the first time, they trigger a profile creation automatically.
    • A brief message introduces profiles: Automatic profile creation step 1
    • The user can name their profile and click Finish: Automatic profile creation step 2
    • This redirects the user to the dashboard where they can view and select their profile in the dropdown list.

Manual profile creation

An administrator can manually create profiles for users as described below.

Create a profile.yaml file with the following content on your local machine:

  1. apiVersion: kubeflow.org/v1beta1
  2. kind: Profile
  3. metadata:
  4.   name: profileName   # replace with the name of profile you want, this will be user's namespace name
  5. spec:
  6.   owner:
  7.     kind: User
  8.     name: userid@email.com   # replace with the email of the user
  10.   resourceQuotaSpec:    # resource quota can be set optionally
  11.    hard:
  12.      cpu: "2"
  13.      memory: 2Gi
  14.      requests.nvidia.com/gpu: "1"
  15.      persistentvolumeclaims: "1"
  16.      requests.storage: "5Gi"

Run the following command to create the corresponding profile resource:

  1. kubectl create -f profile.yaml
  3. kubectl apply -f profile.yaml  #if you are modifying the profile

The above command creates a profile named profileName. The profile owner is userid@email.com and has view and modify access to that profile. The following resources are created as part of the profile creation:

  • A Kubernetes namespace that shares the same name with the corresponding profile.
  • Kubernetes RBAC (Role-based access control) role binding role binding for the namespace: Admin. This makes the profile owner the namespace administrator, thus giving them access to the namespace using kubectl (via the Kubernetes API).
  • Istio namespace-scoped ServiceRole: ns-access-istio. This allows access to all services in the target namespace via Istio routing.
  • Istio namespace-scoped ServiceRoleBinding: owner-binding-istio. This binds the ServiceRole ns-access-istio to the profile owner. The profile owner can therefore access services in the namespace.
  • Namespace-scoped service-accounts default-editor and default-viewer to be used by user-created pods in the namespace.
  • Namespace scoped resource quota limits will be placed.

Note: Due to a one-to-one correspondence of profiles with Kubernetes namespaces, the terms profile and namespace are sometimes used interchangably in the documentation.

Batch creation of user profiles

Administrators might want to create profiles for multiple users as a batch. You can do this by creating a profile.yaml on the local machine with multiple sections of profile descriptions as shown below:

  1. apiVersion: kubeflow.org/v1beta1
  2. kind: Profile
  3. metadata:
  4.   name: profileName1   # replace with the name of profile you want
  5. spec:
  6.   owner:
  7.     kind: User
  8.     name: userid1@email.com   # replace with the email of the user
  9. ---
  10. apiVersion: kubeflow.org/v1beta1
  11. kind: Profile
  12. metadata:
  13.   name: profileName2   # replace with the name of profile you want
  14. spec:
  15.   owner:
  16.     kind: User
  17.     name: userid2@email.com   # replace with the email of the user

Run the following command to apply the namespaces to the Kubernetes cluster:

  1. kubectl create -f profile.yaml
  3. kubectl apply -f profile.yaml  #if you are modifying the profiles

This will create multiple profiles, one for each individual listed in the sections in profile.yaml.

Listing and describing profiles

An administrator can list the existing profiles in the system:

  1. $ kubectl get profiles

and describe a specific profile using:

  1. $ kubectl describe profile profileName

Deleting an existing profile

An administrator can delete an existing profile using:

  1. $ kubectl delete profile profileName

This will delete the profile, the corresponding namespace and any Kubernetes resources associated with the profile. The profile’s owner or other users with access to the profile will no longer have access to the profile and will not see it in the dropdown list on the central dashboard.

Managing contributors through the Kubeflow UI

Kubeflow v1.2.0 allows sharing of profiles with other users in the system. An owner of a profile can share access to their profile using the Manage Contributors tab available through the dashboard.

Manage Contributors in Profiles

Here is an example of the Manage Contributors tab view:

Manage Contributors in Profiles

Notice that in the above view the account associated with the profile is a cluster administrator (Cluster Admin) as this account was used to deploy Kubeflow. The view lists the profiles accessible to the user along with the role associated with that profile.

To add or remove a contributor, add/remove the email address or the user identifier in the Contributors to your namespace field.

Add Contributors

The Manage Contributors tab shows the contributors that the namespace owner has added. Note that the cluster administrator can view all the profiles in the system along with their contributors.

View Contributors

The contributors have access to all the Kubernetes resources in the namespace and can create notebook servers as well as access existing notebooks.

Managing contributors manually

An administrator can manually add contributors to an existing profile as described below.

Create a rolebinding.yaml file with the following content on your local machine:

  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: RoleBinding
  3. metadata:
  4.   annotations:
  5.     role: edit
  6.     user: userid@email.com   # replace with the email of the user from your Active Directory case sensitive
  7.   name: user-userid-email-com-clusterrole-edit
  8.   # Ex: if the user email is lalith.vaka@kp.org the name should be user-lalith-vaka-kp-org-clusterrole-edit
  9.   # Note: if the user email is Lalith.Vaka@kp.org from your Active Directory, the name should be user-lalith-vaka-kp-org-clusterrole-edit
  10.   namespace: profileName # replace with the namespace/profile name that you are adding contribbitors to
  11. roleRef:
  12.   apiGroup: rbac.authorization.k8s.io
  13.   kind: ClusterRole
  14.   name: kubeflow-edit
  15. subjects:
  16. - apiGroup: rbac.authorization.k8s.io
  17.   kind: User
  18.   name: userid@email.com   # replace with the email of the user from your Active Directory case sensitive

Create a servicerolebinding.yaml file with the following content on your local machine:

  1. apiVersion: rbac.istio.io/v1alpha1
  2. kind: ServiceRoleBinding
  3. metadata:
  4.   annotations:
  5.     role: edit
  6.     user: userid@email.com   # replace with the email of the user from your Active Directory case sensitive
  7.   generation: 1
  8.   name: user-userid-email-com-clusterrole-edit
  9.   # Ex: if the user email is lalith.vaka@kp.org the name should be user-lalith-vaka-kp-org-clusterrole-edit
  10.   # Note: if the user email is Lalith.Vaka@kp.org from your Active Directory, the name should be user-lalith-vaka-kp-org-clusterrole-edit
  11.   namespace: profileName # replace with the namespace/profile name that you are adding contribbitors to
  12. spec:
  13.   roleRef:
  14.     kind: ServiceRole
  15.     name: ns-access-istio
  16.   subjects:
  17.   - properties:
  18.       request.headers[kubeflow-userid]: accounts.google.com:userid@email.com   # replace with the email of the user from your Active Directory case sensitive
  19.       # for GCP, use x-goog-authenticated-user-email instead of kubeflow-userid for authentication purpose
  20. status: {}

Run the following command to create the corresponding contributor resources:

  1. kubectl create -f rolebinding.yaml
  2. kubectl create -f servicerolebinding.yaml

The above command adds a contributor userid@email.com to the profile named profileName. The contributor userid@email.com has view and modify access to that profile.

Last modified 21.02.2021: Fix resourceQuotaSpec with GPU (#2460) (30f46633)