Multi-user Isolation

Isolation of user-created resources for convenience and organization

As of v0.6, Kubeflow supports for multi-user isolation of user-createdresources in a Kubeflow deployment. The primary purpose of this functionalityis to enable multiple users to operate on a shared Kubeflow deployment withoutstepping on each others’ jobs and resources. It provides the users with theconvenience of clutter free browsing of notebooks, training jobs, servingdeployments and other resources. The isolation mechanisms also preventaccidental deletion/modification of resources of other users in the deployment.Note that the isolation support in Kubeflow doesn’t provide any hard securityguarantees against malicious attempts by users to infiltrate other user’sprofiles.

Usage overview

An administrator needs to deploy Kubeflow and configure the authentication.service for the deployment. A user can log into the system and will by defaultbe accessing their primary profile. A profile is a collection of Kubernetesresources along with a Kubernetes namespace of the same name. Users have viewand modify access to their primary profile. The owner of a profile can shareaccess to the profile with another user in the system. When sharing the accessto a profile, the owner can choose to share it as a view access only or viewand modify access with the user. For all practical purposes when workingthrough the Kubeflow central dashboard, the active namespace is directly tiedwith the active profile.

Example of usage

A user can select their active profile from the top bar on the Kubeflow centraldashboard. Note that the user can only view the profiles in the dropdown listthat they have view or modify access to.

This guide illustrates the user isolation functionality using the Jupyternotebooks service which is the first service in the system to have fullintegration with the multi-user isolation functionality.

Once an active profile has been selected by the user, the Notebooks Servers UIwill display only the active notebook servers in the currently selectedprofile. All other notebook servers remain hidden from the user. If they switchthe active profile, the view will switch the list of active notebooksappropriately. The user can connect to any of the listed notebook servers andview and modify the existing Jupyter notebooks available in the server.

For example, the following image shows the list of Notebook servers availablein the user’s primary profile:

And when an unauthorized user accesses the Notebooks in this profile, they arepresented with an error.

When the users create Jupyter notebook servers from the Notebooks Servers UI,the notebook pods are created in the active profile. If the user doesn’t havemodify access to the active profile, the user can only browse currently activenotebook servers and access the existing jupyter notebooks but cannot createnew notebook servers in that profile. The users can create notebookservers in their primary profile which they have view and modify access to.

Onboard new user

Kubeflow v0.6.2 provides automatic profile creation for authenticated users onfirst login. Additionally, an administrator can create a profile for anyuser in the kubeflow cluster. Here an administrator is a person who hascluster-adminrole binding in the Kubernetes cluster. This person has permissions to createand modify Kubernetes resources in the cluster. For example, the person whodeployed Kubeflow will have administration privileges in the cluster.

Pre-requisite: Grant user minimal Kubernetes cluster view access

User should be given minimal permission scope that allows them to connect to the Kubernetes cluster.For example for GCP users, they can be granted IAM roles: Kubernetes Engine Cluster Viewer and IAP-secured Web App User

Automatic creation of Profiles

Kubeflow v0.6.2 onwards provides automatic profile creations as a convenienceto the users:

• Kubeflow deployment process automatically creates a profile for the userperforming the deployment. When the user access the Central Dashboardthey’ll already see their profile in the dropdown list.
• When an authenticated user logs into the system and visits the CentralDashboard for the first time, they trigger a profile creation automaticlly.
  • A brief message introduces profiles
  • The user can name their profile and click Finish.
  • This redirects the user to the dashboard where they can view and selectthier profile in the dropdown list.

Manual profile creation

An administrator can manually create profiles for users as described below.

Create aprofile.yaml file with the following contents on your local machine:

$ cat << EOF > profile.yaml
apiVersion: kubeflow.org/v1alpha1
kind: Profile
metadata:
  name: profileName   # replace with the name of profile you want
spec:
  owner:
    kind: User
    name: userid@email.com   # replace with the email of the user
EOF

Create the corresponding profile resource using:

$ kubectl create -f profile.yaml

The above creates a profile named profileName whose owner isuserid@email.com and has view and modify access to that profile.The following resources are created as part of the profile creation:

• A Kubernetes namespace that shares the same name with the correspondingprofile.
• Kubernetes RBAC rolebinding for the namespace Admin. This makes theprofile owner the namespace admin, allowing access to the namespace viaKubernetes API (using kubectl).
• Istio namespace-scoped ServiceRole ns-access-istio. This allows access toall services in the target namespace via Istio routing.
• Istio namespace-scoped ServiceRoleBinding owner-binding-istio. This bindsthe ServiceRole ns-access-istio to the profile owner. The profile owner cantherefore access services in the namespace created.
• Namespace-scoped service-accounts editor and viewer to be used byuser-created pods in above namespace.

Note: Due to a 1-to-1 correspondence of Profiles with KubernetesNamespaces, Profiles and Namespaces are sometimes used interchangably in thedocumentation.

Batch creation of user profiles

Administrators might want to create profiles for multiple users as a batch. This canbe done by creating a profile.yaml on the local machine with multiple sections ofprofile descriptions as shown below:

$ cat << EOF > profile.yaml
apiVersion: kubeflow.org/v1alpha1
kind: Profile
metadata:
  name: profileName1   # replace with the name of profile you want
spec:
  owner:
    kind: User
    name: userid1@email.com   # replace with the email of the user
---
apiVersion: kubeflow.org/v1alpha1
kind: Profile
metadata:
  name: profileName2   # replace with the name of profile you want
spec:
  owner:
    kind: User
    name: userid2@email.com   # replace with the email of the user
EOF

Apply to the Kubernetes cluster using:

kubectl create -f profile.yaml

This will create multiple profiles for each individual listed in the sectionsin profile.yaml.

Listing and describing profiles

An administrator can list the existing profiles in the system:

$ kubectl get profiles

and describe a specific profile using:

$ kubectl describe profile profileName

Deleting an existing profile

An administrator can delete an existing profile using:

$ kubectl delete profile profileName

This will delete the profile, the corresponding namespace and any Kubernetesresources associated with the profile. The profile’s owner or other users withaccess to the profile will no longer have access to the profile and will not seeit in the dropdown list on the central dashboard.

Managing Contributors through UI

Kubeflow v0.6.2 onwards allows sharing of profiles with other users in thesystem. An owner of a profile can share access to their profile using theManage Contributors tab available through the dashboard.

Here is an example of the Manage Contributors tab view:

Notice that in the above view the account associated with the profile isCluster Admin as this account was used to deploy Kubeflow. It lists theprofiles accessible to the user along with the role associated with thatprofile.

Adding and removing contributors is easily possible by simply adding/removing theemail address or the user identifier in the Contributors to your namespace field.

Once added, the Manage Contributors tab will show the profiles with thiercorresponding contributors listed. Note that Cluster Admin can view all theprofiles in the system along with their contributors.

The contributors will have access to all the Kubernetes resources in thenamespace and will be able to created notebook servers as well as accessexisting notebooks. The contributor’s access can be removed by the owner of aprofile by visiting the Manage Contributors tab and removing the useremail/id from the list of contributors.

Current Integration and Limitations

The Jupyter notebooks service is the first application to be fully integrated withmulti-user isolation. Access to the notebooks and the creation of notebooks iscontrolled by the profile access policies set by the Administrator or the ownersof the profiles. Resources created by the notebooks (eg. Training jobs anddeployments) will also inherit the same access.

Metadata and Pipelines or any other applications currently don’t have fullfledged integration with isolation, though they will have access to the useridentity through the headers of the incoming requests. It’s upto the individualapplications to leverage the available identity and create isolation storiesthat make sense for them.

On GCP, the authentication and identify token is generated by GCP IAM and carriedthrough the requests as a JWT Token in header. Other cloud providers can have asimilar header to provide identity information.

For on-premise deployments, Kubeflow leverages Dex as a federated OpenID connectionprovider and can be integrated with LDAP or Active Directory to provide authenticationand identity services.