Install Kubeflow

Instructions for deploying Kubeflow on Amazon EKS

This guide describes how to use the kfctl CLI to deploy Kubeflow on Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon Web Services (AWS).

Kubernetes versions 1.15+ on Amazon EKS are compatible with Kubeflow version 1.2. Please see the compatibility matrix for more information.

Understanding the deployment process

kfctl is used for deploying and managing Kubeflow. The kfctl deployment process consists of the following commands:

  • kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You may optionally run kfctl build to edit the resources before running kfctl apply.
  • kfctl apply - Creates or updates the resources in your cluster
  • kfctl delete - Deletes previously created resources

App layout

When working with kfctl on AWS, the Kubeflow app directory contains the following files and directories:

  • A configuration YAML file that defines configuration related to your Kubeflow deployment.

    • In the walkthrough below, a copy of the GitHub-based configuration YAML file is used to deploy Kubeflow (as CONFIG_URI below).
    • When you run kfctl apply or kfctl build, it works with this local version of the configuration file, which you can customize as needed.

  • aws_config is a directory that contains a sample eksctl cluster configuration file as well as JSON files defining IAM policies.

  • kustomize is a directory that contains the kustomize packages for Kubeflow applications.

    • The directory is created when you run kfctl build or kfctl apply.
    • You can customize the Kubernetes resources (modify the manifests and run kfctl apply again).

The provisioning scripts can be used with a new cluster, or you can install Kubeflow on an existing cluster. We recommend that you create a new cluster for better isolation.

For information on customizing your deployment on AWS and Amazon EKS, please see Customizing Kubeflow on AWS for more information.

If you experience any issues with installation, see the troubleshooting guidance for more information.

Prerequisites

EKS cluster

Before moving forward with Kubeflow installation, you will need a Kubernetes cluster in Amazon EKS. If you already have a cluster, ensure that your current kubectl context is set and move on to the next step.

There are several ways to provision a cluster in EKS, including with the aws CLI, in the EKS Console, or via AWS CloudFormation, Terraform, or the AWS Cloud Development Kit (CDK). A simple way to get started is by using eksctl, which we recommend here.

First, set a few environment variables to specify your desired cluster name, AWS region, Kubernetes version, and Amazon EC2 instance type to use for cluster nodes.

For example:

  1. export AWS_CLUSTER_NAME=kubeflow-demo
  2. export AWS_REGION=us-west-2
  3. export K8S_VERSION=1.18
  4. export EC2_INSTANCE_TYPE=m5.large

Now, create a cluster configuration file for use with eksctl.

  1. cat << EOF > cluster.yaml
  2. ---
  3. apiVersion: eksctl.io/v1alpha5
  4. kind: ClusterConfig
  6. metadata:
  7.   name: ${AWS_CLUSTER_NAME}
  8.   version: "${K8S_VERSION}"
  9.   region: ${AWS_REGION}
  11. managedNodeGroups:
  12. - name: kubeflow-mng
  13.   desiredCapacity: 3
  14.   instanceType: ${EC2_INSTANCE_TYPE}
  15. EOF

Finally, create the cluster using eksctl.

  1. eksctl create cluster -f cluster.yaml

Prepare your environment

Note: kfctl is currently available for Linux and macOS users only. If you use Windows, you can install kfctl on Windows Subsystem for Linux (WSL). Refer to the official instructions for setting up WSL.

Follow the steps below to download the kfctl binary and set some handy environment variables.

  1. Download the kfctl v1.2.0 release from the Kubeflow releases page.

  2. Unpack the tar ball and add the current working directory to your shell’s path to simplify use of kfctl.

    1. tar -xvf kfctl_v1.2.0_<platform>.tar.gz
    2. export PATH=$PATH:$PWD

  3. Set an environment variable for the configuration file.

    Option 1: Use the default configuration file for authentication using Dex:

    1. export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_aws.v1.2.0.yaml"

    Option 2: Alternatively, use this configuration file to enable multi-user authentication with AWS Cognito. For more information on this configuration, see Authentication and Authorization before moving forward.

    1. export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_aws_cognito.v1.2.0.yaml"

  4. Set an environment variable with the name of your Amazon EKS cluster.

    1. export AWS_CLUSTER_NAME=<YOUR EKS CLUSTER NAME>

  5. Finally, create a deployment directory for your cluster, change to it, and download the kfctl configuration file.

    1. mkdir ${AWS_CLUSTER_NAME} && cd ${AWS_CLUSTER_NAME}
    2. wget -O kfctl_aws.yaml $CONFIG_URI

Note your EKS cluster name must be set correctly. This is used during the deployment process and will cause issues if not set.

Configure Kubeflow

Modifications can be made to the local configuration file prior to deployment. Edit the file as follows with your favorite editor.

By default, the username is set to admin@kubeflow.org and the password is 12341234. To secure your Kubeflow deployment, change this configuration.

Since v1.0.1, Kubeflow supports use of AWS IAM Roles for Service Accounts (IRSA). This allows for fine-grained policy configuration bound to a specific service account in your Kubernetes cluster, as opposed to attaching the required policies to the node instance role.

kfctl will create two roles, kf-admin-{aws-region}-{cluster-name} and kf-user-{aws-region}-{cluster-name}, and two service accounts in the kubeflow namespace, kf-admin and kf-user. The kf-admin role will be assumed by components like alb-ingress-controller, profile-controller or any Kubeflow control plane components which need to talk to AWS services, while the kf-user role can be used by user applications.

This is only available on clusters managed by Amazon EKS. For DIY Kubernetes on AWS, check out aws/amazon-eks-pod-identity-webhook for configuration options.

The method of attaching required IAM policies to the EKS node instance role is still supported (Option 2 below), but using IAM Roles for Service Accounts is recommended.

Option 1: Use AWS IAM Roles For Service Accounts (default and recommended)

kfctl will create or reuse your cluster’s IAM OIDC Identity Provider, will create the required IAM roles, and configure the trust relationship binding the roles with your Kubernetes Service Accounts.

This is the default configuration, just update this configuration file section with your AWS Region.

  1. region: ${AWS_REGION} (e.g. us-west-2)
  2. enablePodIamPolicy: true

Note: By default, no policies are attached to the kf-user-{aws-region}-{cluster-name} IAM Role, you can configure this as needed.

Option 2: Use Node Group Role

If you would prefer to not use IRSA, follow these steps to modify the configuration file as needed.

  1. Retrieve your cluster’s node instance role. This is the IAM Role that’s used to provide permissions to your Kubernetes nodes. For example:

    1. aws iam list-roles \
    2.     | jq -r ".Roles[] \
    3.     | select(.RoleName \
    4.     | startswith(\"eksctl-$AWS_CLUSTER_NAME\") and contains(\"NodeInstanceRole\")) \
    5.     .RoleName"
    7. eksctl-kubeflow-example-nodegroup-ng-123-NodeInstanceRole-4567

    Note, this example assumes that you used eksctl to create your cluster. If you use other provisioning tools to create your worker node groups, find the role used by your Kubernetes nodes via AWS CLI or Console.

  2. Update this configuration file section with your AWS Region and one or more IAM role names.

  1. region: ${AWS_REGION} (e.g. us-west-2)
  2. roles:
  3. - ${NODE_INSTANCE_ROLENAME} (e.g. eksctl-kubeflow-example-nodegroup-ng-123-NodeInstanceRole-4567)

Deploy Kubeflow

  1. Run the following commands to initialize the Kubeflow cluster:

    1. kfctl apply -V -f kfctl_aws.yaml

  2. Wait for all the resources to become ready in the kubeflow namespace.

    1. kubectl -n kubeflow get all

Access Kubeflow central dashboard

Run the following command to get your Kubeflow service’s endpoint host name and copy link in browser.

  1. kubectl get ingress -n istio-system
  3. NAMESPACE      NAME            HOSTS   ADDRESS                                                       PORTS   AGE
  4. istio-system   istio-ingress   *       123-istiosystem-istio-2af2-4567.us-west-2.elb.amazonaws.com   80      1h

This deployment may take 3-5 minutes to become ready. Once complete, you can verify the installation by opening the ingress address in your preferred browser.

Add static users for basic authentication

To add users to basic auth, edit the Dex ConfigMap under the key staticPasswords.

  1. # Edit the dex config with extra users.
  2. kubectl edit configmap dex -n auth
  4. # The original example of configmap as below
  5. staticPasswords:
  6. - email: admin@kubeflow.org
  7.   hash: JDJhJDEwJEU4SGhqTnpBRzc2eWJJM1RHSDk5Ly4xcWxIckx6UGlJbzMzdW9BWHZ4VU5hTWxjZXAzVTBp
  8.   username: admin
  9.   userID: 08a8684b-db88-4b73-90a9-3cd1661f5466
  11. # If you want to add a static user (test@kubeflow.org: 123456789)
  12. # The password (123456789) must be hashed with bcrypt with an at least 10 difficulty level.
  13. # You can use an online tool like: https://passwordhashing.com/BCrypt
  14. # After change, the example of configmap:
  15. staticPasswords:
  16. - email: admin@kubeflow.org
  17.   hash: JDJhJDEwJEU4SGhqTnpBRzc2eWJJM1RHSDk5Ly4xcWxIckx6UGlJbzMzdW9BWHZ4VU5hTWxjZXAzVTBp
  18.   username: admin
  19.   userID: 08a8684b-db88-4b73-90a9-3cd1661f5466
  20. - email: test@kubeflow.org
  21.   hash: $2b$10$ow6fWbPojHUg56hInYmYXe.B7u3frcSR.kuUkQp2EzXs5t0xfMRtS
  22.   username: test
  23.   userID: 08a8684b-db88-4b73-90a9-3cd1661f5466
  25. # After editing the config, restart Dex to pick up the changes in the ConfigMap
  26. kubectl rollout restart deployment dex -n auth

Post Installation

Kubeflow provides multi-tenancy support and users are not able to create notebooks in either the kubeflow or default namespaces.

During the first user login, the system will create the anonymous namespace. To create additional users, you can create profiles.

Create a manifest file profile.yaml with the following contents.

  1. apiVersion: kubeflow.org/v1beta1
  2. kind: Profile
  3. metadata:
  4.   name: test
  5. spec:
  6.   owner:
  7.     kind: User
  8.     name: test@amazon.com

Note: spec.owner.name is the user’s email.

Now create the profile via kubectl create -f profile.yaml. The Profile controller will create a new namespace name and related service account to allow for notebook creation in that namespace.

Check Multi-Tenancy in Kubeflow for more details.

Last modified 04.05.2021: refactor and refresh aws docs (#2688) (ef4cda60)