Configuring Kubeflow with kfctl and kustomize

The basics of Kubeflow configuration with kfctl and kustomize

Kfctl is the Kubeflow command-line interface (CLI) that you can use toinstall and configure Kubeflow.

Kubeflow makes use of kustomize to help customize YAMLconfigurations. With kustomize, you can traverse a Kubernetes manifest to add,remove, or update configuration options without forking the manifest. Amanifest is a YAML file containing a description of the applications that youwant to include in your Kubeflow deployment.

Overview of kfctl and kustomize

This section describes how kfctl works with kustomize to set up yourKubeflow deployment.

The Kubeflow deployment process

Kfctl is the Kubeflow CLI that you can use to set up a Kubernetes cluster withKubeflow installed, or to deploy Kubeflow to an existing Kubernetes cluster.See the Kubeflow getting-started guide forinstallation instructions based on your deployment scenario.

The kfctl deployment process includes the following commands:

  • kfctl build - (Optional) Creates configuration files defining the variousresources in your deployment but does not deploy Kubeflow.You only need to run kfctl build if you wantto edit the resources before running kfctl apply.
  • kfctl apply - Creates or updates the resources.
  • kfctl delete - Deletes the resources.

Specifying a configuration file when initializing your deployment

When you install Kubeflow, the deployment process uses one of a few possibleYAML configuration files to bootstrap the configuration. You can see all theconfiguration files onGitHub.

As an example, this guide uses thekfctl_k8s_istio.yamlconfiguration. For more details about this configuration, see thekfctl_k8s_istio deployment guide.

Typically, you specify the configuration file with a -f <config-file>parameter when you run kfctl build or kfctl apply. The following exampleuses kfctl build:

  1. # Set KF_NAME to the name of your Kubeflow deployment. You also use this
  2. # value as directory name when creating your configuration directory.
  3. # For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
  4. export KF_NAME=<your choice of name for the Kubeflow deployment>
  6. # Set the path to the base directory where you want to store one or more 
  7. # Kubeflow deployments. For example, /opt/.
  8. # Then set the Kubeflow application directory for this deployment.
  9. export BASE_DIR=<path to a base directory>
  10. export KF_DIR=${BASE_DIR}/${KF_NAME}
  12. # Set the URI of the configuration file to use when deploying Kubeflow. 
  13. # For example:
  14. export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml"
  16. # Create your Kubeflow configurations:
  17. mkdir -p ${KF_DIR}
  18. cd ${KF_DIR}
  19. kfctl build -V -f ${CONFIG_URI}

Kfctl has now built the configuration files in your Kubeflow applicationdirectory (see below) but has not yet deployed Kubeflow.To complete the deployment, run kfctl apply. See the next section onapplying the configuration.

Applying the configuration to your Kubeflow cluster

When you first run kfctl build or kfctl apply, kfctl createsa local version of the YAML configuration file,which you can further customize if necessary.

Follow these steps to apply the configurations to your Kubeflow cluster:

  • Set an environment variable pointing to your local configuration file.For example, this guide uses the kfctl_k8s_istio.0.7.1.yamlconfiguration. If you chose a different configuration in the previous step,you must change the file name to reflect your configuration:
  1. export CONFIG_FILE=${KF_DIR}/kfctl_k8s_istio.0.7.1.yaml
  • Apply the configurations:
  1. kfctl apply -V -f ${CONFIG_FILE}

Your Kubeflow directory layout

Your Kubeflow application directory is the directory where you choose to storeyour Kubeflow configurations during deployment. This guide refers to thedirectory as ${KF_DIR}. The directory contains the following files anddirectories:

  • ${CONFIG_FILE} is a YAML file that stores your primary Kubeflowconfiguration in the form of a KfDef Kubernetes object.

    • This file is a copy of the GitHub-based configuration YAMLfile thatyou used when deploying Kubeflow.
    • When you first run kfctl build or kfctl apply, kfctl createsa local version of the configuration file at ${CONFIG_FILE},which you can further customize if necessary.
    • The YAML defines each Kubeflow application as a kustomize package.

  • _config is a directory that containsconfigurations specific to your chosen platform or cloud provider.For example, gcp_config or aws_config. Thisdirectory may or may not be present, depending on your setup.

    • The directory is created when you run kfctl build or kfctl apply.
    • To customize these configurations, you can modify parametersin your ${CONFIG_FILE}, and then run kfctl apply to applythe configuration to your Kubeflow cluster.

  • kustomize is a directory that contains Kubeflow application manifests.That is, the directory contains the kustomize packages for the Kubeflowapplications that are included in your deployment.

    • The directory is created when you run kfctl build or kfctl apply.
    • To customize these configurations, you can modify parametersin your ${CONFIG_FILE}, and then run kfctl apply to applythe configuration to your Kubeflow cluster.

How your configuration is generated

The content of your ${CONFIG_FILE} is the result of running kustomizeon the base and overlay kustomization.yaml files in theKubeflow manifests.The overlays reflect the configuration file that you specify when runningkfctl build or kfctl apply.

Below are some examples of configuration files:

  • kfctl_k8s_istio.yamlto install Kubeflow on an existing Kubernetes cluster.
  • kfctl_existing_arrikto.yamlto install Kubeflow on an existing Kubernetes cluster with Dex and Istio forauthentication.
  • kfctl_gcp_iap.yamlto create a Google Kubernetes Engine (GKE) cluster with Kubeflow usingCloud Identity-Aware Proxy (Cloud IAP) for access control.

The kustomize package manager in kfctl uses the information in your${CONFIG_FILE} to traverse the directories under theKubeflow manifests and tocreate kustomize build targets based on the manifests.

Installing kustomize

Make sure that you have the minimum required version of kustomize:2.0.3 or later.

  • Follow the kustomize installationguide,choosing the relevant options for your operating system. For example, ifyou’re on Linux:

    • Set some variables for the operating system:
  1. export opsys=linux
  • Download the kustomize binary:
  1. curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases |\
  2. grep browser_download |\
  3. grep download/kustomize |\
  4. grep -m 1 $opsys |\
  5. cut -d '"' -f 4 |\
  6. xargs curl -O -L
  • Move the binary:
  1. mkdir -p ${HOME}/bin
  2. mv kustomize_*_${opsys}_amd64 ${HOME}/bin/kustomize
  3. chmod u+x ${HOME}/bin/kustomize
  • Include the kustomize command in your path:
  1.   export PATH=$PATH:${HOME}/bin/kustomize

Modifying configuration before deployment

Kustomize lets you customize raw, template-free YAML files for multiplepurposes, leaving the original YAML untouched and usable as is.

You can use the following command to build and apply kustomize directories:

  1. kustomize build <kustomization_directory> | kubectl apply -f -

The Kubeflow manifests repo containskustomize build targets, each with a base directory. You can use kustomize togenerate YAML output and pass it to kfctl. You can also makechanges to the kustomize targets in the manifests repo as needed.

Modifying the configuration of an existing deployment

To customize the Kubeflow resources running within the cluster, you can modifyparameters in your ${CONFIG_FILE} file. Then re-run kfctl apply.

For example, to modify settings for the Spartakus usage reporting tool withinyour Kubeflow deployment:

  • Edit the configuration file at ${CONFIG_FILE}.

  • Find and replace the parameter values for spartakus to suit yourrequirements:

  1. - kustomizeConfig:
  2.     parameters:
  3.     - initRequired: true
  4.         name: usageId
  5.         value: <randomly-generated-id>
  6.     - initRequired: true
  7.         name: reportUsage
  8.         value: "true"
  9.     repoRef:
  10.         name: manifests
  11.         path: common/spartakus
  12.     name: spartakus
  • Regenerate and deploy your Kubeflow resources:
  1. cd ${KF_DIR}
  2. kfctl apply -V -f ${CONFIG_FILE}

More examples

For examples of customizing your deployment, see the guide to customizingKubeflow on GKE.

For information about how Kubeflow uses Spartakus, see the guide tousage reporting.

More about kustomize

Below are some useful kustomize terms, from thekustomize glossary:

  • base: A combination of a kustomization and resource(s). Bases can bereferred to by other kustomizations.

  • kustomization: Refers to a kustomization.yaml file, or more generally toa directory containing the kustomization.yaml file and all the relative filepaths that the YAML file references.

  • overlay: A combination of a kustomization that refers to a base, and apatch. An overlay may have multiple bases.

  • patch: General instructions to modify a resource.

  • resource: Any valid YAML file that defines an object with a kind and ametadata/name field.

  • target: The argument to kustomize build. For example,kustomize build $TARGET. A target must be a path or a URL to akustomization. A target can be a base or an overlay.

  • variant: The outcome of applying an overlay to a base.

Read more about kustomize in thekustomize documentation.