Using environment variables in pipelines

How to set and use environment variables in Kubeflow pipelines

This page describes how to pass environment variables to Kubeflow pipelinecomponents.

Before you start

Set up your environment:

Using environment variables

In this example, you pass an environment variable to a lightweight Pythoncomponent, which writes the variable’s value to the log.

Learn more about lightweight Python components

To build a component, define a stand-alone Python function and then callkfp.components.func_to_container_op(func) to convert thefunction to a component that can be used in a pipeline. The following function gets anenvironment variable and writes it to the log.

  1. def logg_env_function():
  2.   import os
  3.   import logging
  4.   logging.basicConfig(level=logging.INFO)
  5.   env_variable = os.getenv('example_env')
  6.   logging.info('The environment variable is: {}'.format(env_variable))

Transform the function into a component usingkfp.components.func_to_container_op(func).

  1. image_name = 'tensorflow/tensorflow:1.11.0-py3'
  2. logg_env_function_op = comp.func_to_container_op(logg_env_function,
  3.                                                  base_image=image_name)

Add this component to a pipeline. Use add_env_variable to pass anenvironment variable into the component. This code is the same no matter if yourusing python lightweight components or a container operation.

  1. import kfp.dsl as dsl
  2. from kubernetes.client.models import V1EnvVar
  4. @dsl.pipeline(
  5.   name='Env example',
  6.   description='A pipline showing how to use environment variables'
  7. )
  8. def environment_pipline():
  9.   env_var = V1EnvVar(name='example_env', value='env_variable')
  10.   #Returns a dsl.ContainerOp class instance. 
  11.   container_op = logg_env_function_op().add_env_variable(env_var)

To pass more environment variables into a component, add more instances ofadd_env_variable(). Use the following command to run this pipeline using theKubeflow Pipelines SDK.

  1. #Specify pipeline argument values
  2. arguments = {}
  4. #Submit a pipeline run
  5. kfp.Client().create_run_from_pipeline_func(environment_pipline,
  6.                                            arguments=arguments)

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified 24.02.2020: Formats code samples and updates link text (#1733) (97d4f0f2)