Configuring Kubeflow with kfctl and kustomize

Configuring Kubeflow with kfctl and kustomize

The basics of Kubeflow configuration with kfctl and kustomize

Kfctl is the Kubeflow command-line interface (CLI) that you can use toinstall and configure Kubeflow.

Kubeflow makes use of kustomize to help customize YAMLconfigurations. With kustomize, you can traverse a Kubernetes manifest to add,remove, or update configuration options without forking the manifest. Amanifest is a YAML file containing a description of the applications that youwant to include in your Kubeflow deployment.

Overview of kfctl and kustomize

This section describes how kfctl works with kustomize to set up yourKubeflow deployment.

The Kubeflow installation process

Kfctl is the Kubeflow CLI that you can use to set up a Kubernetes cluster withKubeflow installed, or to deploy Kubeflow to an existing Kubernetes cluster.See the Kubeflow getting-started guide forinstallation instructions based on your deployment scenario.

The deployment process consists of three steps, init, generate, andapply, so that you can modify your configuration before actually deployingKubeflow.

kfctl init - one time set up.
kfctl generate - creates config files defining the various resources.
kfctl apply - creates or updates the resources.

Specifying a configuration file when initializing your deployment

When you install Kubeflow, the deployment process uses one of a few possibleYAML configuration files to bootstrap the configuration.

Typically, you specify the configuration file with a —config parameterwhen you run kfctl init. For example:

export KFAPP="<your choice of application directory name>"
export CONFIG="https://raw.githubusercontent.com/kubeflow/kubeflow/v0.6-branch/bootstrap/config/kfctl_existing_arrikto.0.6.2.yaml"
kfctl init ${KFAPP} --config=${CONFIG} -V

For details of the above deployment, see the guide to deployment using thekfctl_existing_arrikto configuration.

As another example, theGoogle Cloud Platform (GCP) initialization command looks like this:

export KFAPP="<your choice of application directory name>"
export PROJECT="<your GCP project ID>"
export CONFIG="https://raw.githubusercontent.com/kubeflow/kubeflow/c54401e/bootstrap/config/kfctl_gcp_iap.0.6.2.yaml"
kfctl init ${KFAPP} --project=${PROJECT} --config=${CONFIG} -V

For details of the above deployment, see the guide to deploymenton GCP using the CLI.

Your Kubeflow directory layout

Your Kubeflow app directory is the directory where you choose to storeyour Kubeflow configurations during deployment. This guide refers to thedirectory as ${KFAPP}. The directory contains the following files anddirectories:

app.yaml stores your primary Kubeflow configuration in the form of aKfDef Kubernetes object.
- The values are set when you run kfctl init.
- The values are snapshotted inside app.yaml to make your appself contained.
- The YAML defines each Kubeflow application as a kustomize package.
[platform-name]_config is a directory that containsconfigurations specific to your chosen platform or cloud provider. Thisdirectory may or may not be present, depending on your setup.
- The directory is created when you run kfctl generate platform.
- To customize these configurations, you can modify parameters in the KfDefobject in your app.yaml file, and then re-run kfctl generate andkfctl apply.
kustomize is a directory that contains Kubeflow application manifests.That is, the directory contains the kustomize packages for the Kubeflowapplications that are included in your deployment.
- The directory is created when you run kfctl generate.
- To customize these configurations, you can modify parameters in the KfDefobject in your app.yaml file, and then re-run kfctl generate andkfctl apply.

How your configuration is generated

The content of your app.yaml is the result of running kustomizeon the base and overlay kustomization.yaml files in theKubeflow configdirectory. The overlays reflect the options that you choose when callingkfctl init.

Below are some examples of KfDef configuration files:

kfctl_k8s_istio.yamlto install Kubeflow on an existing Kubernetes cluster.
kfctl_gcp_basic_auth.yamlto set up a Google Kubernetes Engine (GKE) cluster with Kubeflow using basicauthentication.

The kustomize package manager in kfctl uses the information in yourapp.yaml to traverse the directories under theKubeflow manifests and tocreate kustomize build targets based on the manifests.

Installing kustomize

Make sure that you have the minimum required version of kustomize:2.0.3 or later.

Follow the kustomize installationguide,choosing the relevant options for your operating system. For example, ifyou’re on Linux:
- Set some variables for the operating system:

export opsys=linux

Download the kustomize binary:

curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases/latest |\
grep browser_download |\
grep $opsys |\
cut -d '"' -f 4 |\
xargs curl -O -L

Move the binary:

mkdir -p ${HOME}/bin
mv kustomize_*_${opsys}_amd64 ${HOME}/bin/kustomize
chmod u+x ${HOME}/bin/kustomize

Include the kustomize command in your path:

  export PATH=$PATH:${HOME}/bin/kustomize

Modifying configuration before deployment

Kustomize lets you customize raw, template-free YAML files for multiplepurposes, leaving the original YAML untouched and usable as is.

You can use the following command to build and apply kustomize directories:

kustomize build <kustomization_directory> | kubectl apply -f -

The Kubeflow manifests repo containskustomize build targets, each with a base directory. You can use kustomize togenerate YAML output and pass it to kfctl. You can also makechanges to the kustomize targets in the manifests repo as needed.

Modifying the configuration of an existing deployment

To customize the Kubeflow resources running within the cluster, you can modifyparameters in the KfDef object in your ${KFAPP}/app.yaml file, whereKFAPP is the directory where you stored your Kubeflow configurations duringdeployment. Then re-run kfctl generate and kfctl apply.

For example, to modify settings for the Spartakus usage reporting tool withinyour Kubeflow deployment:

Edit the configuration file at ${KFAPP}/app.yaml.
Find and replace the parameter values for spartakus to suit yourrequirements:

- kustomizeConfig:
    parameters:
    - initRequired: true
        name: usageId
        value: <randomly-generated-id>
    - initRequired: true
        name: reportUsage
        value: "true"
    repoRef:
        name: manifests
        path: common/spartakus
    name: spartakus

Due toKubeflow issue #3810,you currently need to tear down your Kubeflow cluster before applying theconfiguration change. This command deletes all your Kubeflow resources:

cd ${KFAPP}
kfctl delete all -V

Regenerate and deploy your Kubeflow resources:

cd ${KFAPP}
kfctl generate all -V
kfctl apply all -V

More examples

For examples of customizing your deployment, see the guide to customizingKubeflow on GKE.

For information about how Kubeflow uses Spartakus, see the guide tousage reporting.

More about kustomize

Below are some useful kustomize terms, from thekustomize glossary:

base: A combination of a kustomization and resource(s). Bases can bereferred to by other kustomizations.
kustomization: Refers to a kustomization.yaml file, or more generally toa directory containing the kustomization.yaml file and all the relative filepaths that the YAML file references.
overlay: A combination of a kustomization that refers to a base, and apatch. An overlay may have multiple bases.
patch: General instructions to modify a resource.
resource: Any valid YAML file that defines an object with a kind and ametadata/name field.
target: The argument to kustomize build. For example,kustomize build $TARGET. A target must be a path or a URL to akustomization. A target can be a base or an overlay.
variant: The outcome of applying an overlay to a base.

Read more about kustomize in thekustomize documentation.