Build Lightweight Python Components

Building your own lightweight pipelines components from Python

Lightweight Python components do not require you to build a new container imagefor every code change. They’re intended for fast iteration in a notebookenvironment.

Advantages over container components:

  • Faster iteration: No need to build new container image after every change(building images takes some time).
  • Easier authoring: Components can be created in a local environment. Docker andKubernetes are not required.

Building a lightweight Python component

To build a component, define a stand-alone Python function and then callkfp.components.func_to_container_op(func) to convert the function to acomponent that can be used in a pipeline.

There are several requirements for the component function:

  • The function must be stand-alone.

    • It should not use any code declared outside the function definition.
    • Any imports should be added inside the main component function.
    • Any helper functions should also be defined inside the main componentfunction.

  • The function can only import packages that are available in the base image.

    • If you need to import a package that’s not available in the default baseimage you can try to find a container image that already includes therequired packages. (As a workaround you can use the subprocess moduleto run pip install for the required package.)

  • If the function operates on numbers, the parameters must have type hints.Supported types are int, float, bool. All other arguments are passed asstrings.

  • To build a component with multiple output values, use Python’styping.NamedTupletype hint syntax:

  1. NamedTuple('MyFunctionOutputs', [('output_name_1', type), ('output_name_2', float)])

The NamedTuple class is already imported, so that it can be used in thefunction declaration.

Tutorials

See the notebook onlightweight Python component basicsfor an example of creating lightweight Python components and using them in apipeline.

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified 10.09.2019: Fix links (#1141) (e5ec21ba)