Install Kubeflow

Instructions for deploying Kubeflow on AWS with the shell

This guide describes how to use the kfctl CLI to deploy Kubeflow on Amazon Web Services (AWS).

Compatibility

EKS enabled E2E tests between EKS versions and Kubeflow versions since 1.2.

EKS VersionsKubeflow 1.2
1.15compatible
1.16compatible
1.17compatible
1.18compatible
  • incompatible: the combination does not work at all
  • compatible: all Kubeflow features have been tested and verified for the EKS version
  • no known issues: the combination has not been fully tested but there are no reported issues

Prerequisites

EKS cluster

There’re many ways to provision EKS cluster, using AWS EKS CLI, CloudFormation or Terraform, AWS CDK or eksctl. Here, we highly recommend you to create an EKS cluster using eksctl.

You are required to have an existing Amazon Elastic Kubernetes Service (Amazon EKS) cluster before moving the next step.

The installation tool uses the eksctl command and doesn’t support the --profile option in that command. If you need to switch role, use the aws sts assume-role commands. See the AWS guide to using temporary security credentials to request access to AWS resources.

Prepare your environment

Note: kfctl is currently available for Linux and macOS users only. If you use Windows, you can install kfctl on Windows Subsystem for Linux (WSL). Refer to the official instructions for setting up WSL.

To deploy Kubeflow on your existing Amazon EKS cluster, you need to provide AWS_CLUSTER_NAME, cluster region and worker roles.

Follow these steps to download the kfctl binary for the Kubeflow CLI and set some handy environment variables:

  1. Download the kfctl v1.2.0 release from the Kubeflow releases page.

  2. Unpack the tar ball:

    1. tar -xvf kfctl_v1.2.0_<platform>.tar.gz

  3. Create environment variables to make the deployment process easier:

    1. # 1. Add kfctl to PATH, to make the kfctl binary easier to use.
    2. export PATH=$PATH:"<path to kfctl>"
    4. # 2. Use the following kfctl configuration file for the AWS setup without authentication:
    5. export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_aws.v1.2.0.yaml"
    7. # Alternatively, use the following kfctl configuration if you want to enable
    8. # authentication, authorization and multi-user:
    9. export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_aws_cognito.v1.2.0.yaml"
    11. # 3. Set an environment variable for your AWS cluster name.
    12. export AWS_CLUSTER_NAME=<YOUR EKS CLUSTER NAME>
    14. # 4. Create the directory you want to store deployment, this has to be ${AWS_CLUSTER_NAME}
    15. mkdir ${AWS_CLUSTER_NAME} && cd ${AWS_CLUSTER_NAME}
    17. # 5. Download your configuration files, so that you can customize the configuration before deploying Kubeflow.
    18. wget -O kfctl_aws.yaml $CONFIG_URI

Notes:

  • ${CONFIG_URI} - The GitHub address of the configuration YAML file that you want to use to deploy Kubeflow. For AWS deployments, the following configurations are available:

    • https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_aws.v1.2.0.yaml

    • https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_aws_cognito.v1.2.0.yaml

      When you run kfctl apply or kfctl build (see the next step), kfctl creates a local version of the configuration YAML file which you can further customize if necessary.

  • ${AWS_CLUSTER_NAME} - The name of your eks cluster. This will be picked by kfctl and set value to metadata.name. alb-ingress-controller requires correct value to provision application load balancers. Alb will be only created with correct cluster name.

Configure Kubeflow

Since v1.0.1, Kubeflow supports to use AWS IAM Roles for Service Account to fine grain control AWS service access. kfctl will create two roles kf-admin-${region}-${cluster_name} and kf-user-${region}-${cluster_name} and Kubernetes service account kf-admin and kf-user under kubeflow namespace. kf-admin-${region}-${cluster_name} will be assumed by components like alb-ingress-controller, profile-controller or any Kubeflow control plane components which need to talk to AWS services. kf-user-${region}-${cluster_name} can be used by user’s application.

This is only available on EKS, for DIY Kubernetes on AWS, check out aws/amazon-eks-pod-identity-webhook to setup webhook.

Traditional way to attach IAM policies to node group role is still working, feel free choose the way you like to use.

Option 1: Use IAM For Service Account

kfctl will help create or reuse IAM OIDC Identity Provider, create role and handle trust relationship binding with Kubernetes Service Accounts.

Note: By default, we don’t attach any policies to kf-user-${region}-${cluster_name}, you can attach policies based on your need.

Add enablePodIamPolicy: true in your ${CONFIG_URI} file:

  1. region: us-west-2
  2. enablePodIamPolicy: true
  4. # you can delete following roles settings.
  5. #roles:
  6. #- eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6

Check IAM Role For Service Account for more usage.

Option 2: Use Node Group Role

  1. Retrieve the AWS Region and IAM role name for your worker nodes. To get the IAM role name for your Amazon EKS worker node, run the following command:

    1. aws iam list-roles \
    2.     | jq -r ".Roles[] \
    3.     | select(.RoleName \
    4.     | startswith(\"eksctl-$AWS_CLUSTER_NAME\") and contains(\"NodeInstanceRole\")) \
    5.     .RoleName"
    7. eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6

    Note: The above command assumes that you used eksctl to create your cluster. If you use other provisioning tools to create your worker node groups, find the role that is associated with your worker nodes in the Amazon EC2 console.

  2. Change cluster region and worker role names in your kfctl_aws.yaml file:

  1. region: us-west-2
  2. roles:
  3. - eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6

If you have multiple node groups, you will see corresponding number of node group roles. In that case, please provide the role names as an array.

By default, the username is set to admin@kubeflow.org and the password is 12341234. To secure your Kubeflow deployment, change this configuration.

Deploy Kubeflow

  1. Run the following commands to initialize the Kubeflow cluster:

    1. kfctl apply -V -f kfctl_aws.yaml

  2. Wait for all the resources to become ready in the kubeflow namespace.

    1. kubectl -n kubeflow get all

Access Kubeflow central dashboard

Run the following command to get your Kubeflow service’s endpoint host name and copy link in browser.

  1. kubectl get ingress -n istio-system
  3. NAMESPACE      NAME            HOSTS   ADDRESS                                                             PORTS   AGE
  4. istio-system   istio-ingress   *       a743484b-istiosystem-istio-2af2-xxxxxx.us-west-2.elb.amazonaws.com   80      1h

This deployment may take 3-5 minutes to become ready. Verify that the address works by opening it in your preferred Internet browser.

Add static users for basic authentication

To add users to basic auth, you just have to edit the Dex ConfigMap under the key staticPasswords.

  1. # Edit the dex config with extra users.
  2. kubectl edit configmap dex -n auth
  4. # The original example of configmap as below
  5. staticPasswords:
  6. - email: admin@kubeflow.org
  7.   hash: JDJhJDEwJEU4SGhqTnpBRzc2eWJJM1RHSDk5Ly4xcWxIckx6UGlJbzMzdW9BWHZ4VU5hTWxjZXAzVTBp
  8.   username: admin
  9.   userID: 08a8684b-db88-4b73-90a9-3cd1661f5466
  11. # If you want to add a static user (test@kubeflow.org: 123456789)
  12. # The password (123456789) must be hashed with bcrypt with an at least 10 difficulty level.
  13. # You can use an online tool like: https://passwordhashing.com/BCrypt
  14. # After change, the example of configmap:
  15. staticPasswords:
  16. - email: admin@kubeflow.org
  17.   hash: JDJhJDEwJEU4SGhqTnpBRzc2eWJJM1RHSDk5Ly4xcWxIckx6UGlJbzMzdW9BWHZ4VU5hTWxjZXAzVTBp
  18.   username: admin
  19.   userID: 08a8684b-db88-4b73-90a9-3cd1661f5466
  20. - email: test@kubeflow.org
  21.   hash: $2b$10$ow6fWbPojHUg56hInYmYXe.B7u3frcSR.kuUkQp2EzXs5t0xfMRtS
  22.   username: test
  23.   userID: 08a8684b-db88-4b73-90a9-3cd1661f5466
  25. # After editing the config, restart Dex to pick up the changes in the ConfigMap
  26. kubectl rollout restart deployment dex -n auth

Post Installation

Kubeflow provides multi-tenancy support and users are not able to create notebooks in kubeflow, default namespace.

The first time you visit the cluster, you can create a namespace anonymous to use. If you want to create different users, you can create Profile and then kubectl apply -f profile.yaml. Profile controller will create new namespace and service account which is allowed to create notebook in that namespace.

  1. apiVersion: kubeflow.org/v1beta1
  2. kind: Profile
  3. metadata:
  4.   name: test
  5. spec:
  6.   owner:
  7.     kind: User
  8.     name: test@amazon.com

Note: spec.owner.name has to match your IDP user’s email.

Check Multi-Tenancy in Kubeflow for more details.

Understanding the deployment process

The kfctl deployment process is controlled by the following commands:

  • kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
  • kfctl apply - Creates or updates the resources.
  • kfctl delete - Deletes the resources.

App layout

Your Kubeflow app directory ${KF_DIR} contains the following files and directories:

  • ${CONFIG_URI} is a YAML file that defines configurations related to your Kubeflow deployment.

    • This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow.
    • When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_URI}, which you can further customize if necessary.

  • aws_config is a directory that contains a sample eksctl cluster configuration file that defines the AWS cluster and policy files to attach to your node group roles.

    • You can modify the cluster_config.yaml and cluster_features.yaml files to customize your AWS infrastructure.

  • kustomize is a directory that contains the kustomize packages for Kubeflow applications.

    • The directory is created when you run kfctl build or kfctl apply.
    • You can customize the Kubernetes resources (modify the manifests and run kfctl apply again).

The provisioning scripts can either bring up a new cluster and install Kubeflow on it, or you can install Kubeflow on your existing cluster. We recommend that you create a new cluster for better isolation.

If you experience any issues running these scripts, see the troubleshooting guidance for more information.

Last modified 20.04.2021: Apply Docs Restructure to `v1.2-branch` = update `v1.2-branch` to current `master` v2 (#2612) (4e2602bd)