Pipeline Metrics

Export and visualize pipeline metrics

This page shows you how to export metrics from a Kubeflow Pipelines component. For details about how to build a component, see the guide to building your own component.

Overview of metrics

Kubeflow Pipelines supports the export of scalar metrics. You can write a list of metrics to a local file to describe the performance of the model. The pipeline agent uploads the local file as your run-time metrics. You can view the uploaded metrics as a visualization in the Runs page for a particular experiment in the Kubeflow Pipelines UI.

Export the metrics dictionary

To enable metrics, your component must have an output called MLPipeline Metrics and return a JSON-serialized metrics dictionary. Otherwise the Kubeflow Pipelines UI will not render the visualization. In other words, the .outputs.artifacts setting for the generated pipeline template should show: - {name: mlpipeline-metrics, path: /tmp/outputs/mlpipeline_metrics/data}. (The file path does not matter.)

An example Lightweight python component that outputs metrics dictionary by writing it to an output file:

  1. from kfp.components import InputPath, OutputPath, create_component_from_func
  3. def produce_metrics(
  4.   # Note when the `create_component_from_func` method converts the function to a component, the function parameter "mlpipeline_metrics_path" becomes an output with name "mlpipeline_metrics" which is the correct name for metrics output.
  5.   mlpipeline_metrics_path: OutputPath('Metrics'),
  6. ):
  7.   import json
  9.   accuracy = 0.9
  10.   metrics = {
  11.     'metrics': [{
  12.       'name': 'accuracy-score', # The name of the metric. Visualized as the column name in the runs table.
  13.       'numberValue':  accuracy, # The value of the metric. Must be a numeric value.
  14.       'format': "PERCENTAGE",   # The optional format of the metric. Supported values are "RAW" (displayed in raw format) and "PERCENTAGE" (displayed in percentage format).
  15.     }]
  16.   }
  17.   with open(mlpipeline_metrics_path, 'w') as f:
  18.     json.dump(metrics, f)
  20. produce_metrics_op = create_component_from_func(
  21.     produce_metrics,
  22.     base_image='python:3.7',
  23.     packages_to_install=[],
  24.     output_component_file='component.yaml',
  25. )

Here’s an example of a lightweight Python component that outputs a metrics dictionary by returning it from the function:

  1. from typing import NamedTuple
  2. from kfp.components import InputPath, OutputPath, create_component_from_func
  4. def produce_metrics() -> NamedTuple('Outputs', [
  5.   ('mlpipeline_metrics', 'Metrics'),
  6. ]):
  7.   import json
  9.   accuracy = 0.9
  10.   metrics = {
  11.     'metrics': [{
  12.       'name': 'accuracy-score', # The name of the metric. Visualized as the column name in the runs table.
  13.       'numberValue':  accuracy, # The value of the metric. Must be a numeric value.
  14.       'format': "PERCENTAGE",   # The optional format of the metric. Supported values are "RAW" (displayed in raw format) and "PERCENTAGE" (displayed in percentage format).
  15.     }]
  16.   }
  17.   return [json.dumps(metrics)]
  19. produce_metrics_op = create_component_from_func(
  20.     produce_metrics,
  21.     base_image='python:3.7',
  22.     packages_to_install=[],
  23.     output_component_file='component.yaml',
  24. )

An example script-based component.yaml component:

  1. name: Produce metrics
  2. outputs:
  3. - {name: MLPipeline Metrics, type: Metrics}
  4. implementation:
  5.   container:
  6.     image: alpine
  7.     command:
  8.     - sh
  9.     - -exc
  10.     - |
  11.       output_metrics_path=$0
  12.       mkdir -p "$(dirname "$output_metrics_path")"
  13.       echo '{
  14.         "metrics": [{
  15.           "name": "accuracy-score",
  16.           "numberValue": 0.8,
  17.           "format": "PERCENTAGE"
  18.         }]
  19.       }' > "$output_metrics_path"
  20.     - {outputPath: MLPipeline Metrics}

Refer to the full example of a component that generates a confusion matrix data from prediction results.

  • The output name must be MLPipeline Metrics or MLPipeline_Metrics (case does not matter).

  • The name of each metric must match the following pattern: ^[a-zA-Z]([-_a-zA-Z0-9]{0,62}[a-zA-Z0-9])?$.

    For Kubeflow Pipelines version 0.5.1 or earlier, name must match the following pattern ^[a-z]([-a-z0-9]{0,62}[a-z0-9])?$

  • numberValue must be a numeric value.

  • format can only be PERCENTAGE, RAW, or not set.

View the metrics

To see a visualization of the metrics:

  1. Open the Experiments page in the Kubeflow Pipelines UI.
  2. Click one of your experiments. The Runs page opens showing the top two metrics, where top is determined by prevalence (that is, the metrics with the highest count) and then by metric name. The metrics appear as columns for each run.

The following example shows the accuracy-score and roc-auc-score metrics for two runs within an experiment:

Metrics from a pipeline run

Next step

Visualize the output of your component by writing out metadata for an output viewer.

Last modified 03.03.2021: Move Kubeflow Pipelines under /components (#2505) (c34470b8)