Pipelines

This document defines Pipelines and their capabilities.

Syntax

To define a configuration file for a Pipeline resource, you can specify thefollowing fields:

  • Required:
    • apiVersion - Specifies the API version, for exampletekton.dev/v1beta1.
    • kind - Specify the Pipeline resource object.
    • metadata - Specifies data to uniquely identify thePipeline resource object, for example a name.
    • spec - Specifies the configuration information foryour Pipeline resource object. In order for a Pipeline to do anything,the spec must include:
    • tasks - Specifies which Tasks to run and how to runthem
  • Optional:
    • description - Description of the Pipeline.
    • resources - Specifies whichPipelineResources of which types the Pipeline will beusing in its Tasks
    • tasks
      • resources.inputs / resource.outputs
      • runAfter - Used when the Pipeline Taskshould be executed after another Pipeline Task, but there is nooutput linking required
      • retries - Used when the task is wanted to be executed ifit fails. Could be a network error or a missing dependency. It does notapply to cancellations.
      • conditions - Used when a task is to be executed only if the specifiedconditions are evaluated to be true.
      • timeout - Specifies timeout after which the TaskRun for a Pipeline Task willfail. There is no default timeout for a Pipeline Task timeout. If no timeout is specified forthe Pipeline Task, the only timeout taken into account for running a Pipeline will be atimeout for the PipelineRun.

Description

The description field is an optional field and can be used to provide description of the Pipeline.

Declared resources

In order for a Pipeline to interact with the outside world, it will probablyneed PipelineResources which will be given toTasks as inputs and outputs.

Your Pipeline must declare the PipelineResources it needs in a resourcessection in the spec, giving each a name which will be used to refer to thesePipelineResources in the Tasks.

For example:

  1. spec:
  2.   resources:
  3.     - name: my-repo
  4.       type: git
  5.     - name: my-image
  6.       type: image

Workspaces

workspaces are a way of declaring volumes you expect to be made available to yourexecuting Pipeline and its Tasks.

Here’s a short example of a Pipeline Spec with workspaces:

  1. spec:
  2.   workspaces:
  3.     - name: pipeline-ws1 # The name of the workspace in the Pipeline
  4.   tasks:
  5.     - name: use-ws-from-pipeline
  6.       taskRef:
  7.         name: gen-code # gen-code expects a workspace with name "output"
  8.       workspaces:
  9.         - name: output
  10.           workspace: pipeline-ws1
  11.     - name: use-ws-again
  12.       taskRef:
  13.         name: commit # commit expects a workspace with name "src"
  14.       runAfter:
  15.         - use-ws-from-pipeline # important: use-ws-from-pipeline writes to the workspace first
  16.       workspaces:
  17.         - name: src
  18.           workspace: pipeline-ws1

For complete documentation on using workspaces in Pipelines, seeworkspaces.md.

For a complete example see the Workspaces PipelineRunin the examples directory.

Parameters

Pipelines can declare input parameters that must be supplied to the Pipelineduring a PipelineRun. Pipeline parameters can be used to replace templatevalues in PipelineTask parameters’ values.

Parameter names are limited to alpha-numeric characters, - and and canonly start with alpha characters and . For example, fooIs-Bar_ is a validparameter name, barIsBa$ or 0banana are not.

Each declared parameter has a type field, assumed to be string if not provided by the user. The other possible type is array — useful, for instance, when a dynamic number of string arguments need to be supplied to a task. When the actual parameter value is supplied, its parsed type is validated against the type field.

Usage

The following example shows how Pipelines can be parameterized, and theseparameters can be passed to the Pipeline from a PipelineRun.

Input parameters in the form of $(params.foo) are replaced inside of thePipelineTask parameters’ values (see alsovariable substitution).

The following Pipeline declares an input parameter called ‘context’, and usesit in the PipelineTask’s parameter. The description and default fields fora parameter are optional, and if the default field is specified and thisPipeline is used by a PipelineRun without specifying a value for ‘context’,the default value will be used.

  1. apiVersion: tekton.dev/v1beta1
  2. kind: Pipeline
  3. metadata:
  4.   name: pipeline-with-parameters
  5. spec:
  6.   params:
  7.     - name: context
  8.       type: string
  9.       description: Path to context
  10.       default: /some/where/or/other
  11.   tasks:
  12.     - name: build-skaffold-web
  13.       taskRef:
  14.         name: build-push
  15.       params:
  16.         - name: pathToDockerFile
  17.           value: Dockerfile
  18.         - name: pathToContext
  19.           value: "$(params.context)"

The following PipelineRun supplies a value for context:

  1. apiVersion: tekton.dev/v1beta1
  2. kind: PipelineRun
  3. metadata:
  4.   name: pipelinerun-with-parameters
  5. spec:
  6.   pipelineRef:
  7.     name: pipeline-with-parameters
  8.   params:
  9.     - name: "context"
  10.       value: "/workspace/examples/microservices/leeroy-web"

Pipeline Tasks

A Pipeline will execute a graph of Tasks (seeordering for how to express this graph). A valid Pipelinedeclaration must include a reference to at least one Task. EachTask within a Pipeline must have avalidname and task reference, for example:

  1. tasks:
  2.   - name: build-the-image
  3.     taskRef:
  4.       name: build-push

Declared PipelineResources can be given to Tasks inthe Pipeline as inputs and outputs, for example:

  1. spec:
  2.   tasks:
  3.     - name: build-the-image
  4.       taskRef:
  5.         name: build-push
  6.       resources:
  7.         inputs:
  8.           - name: workspace
  9.             resource: my-repo
  10.         outputs:
  11.           - name: image
  12.             resource: my-image

Parameters can also be provided:

  1. spec:
  2.   tasks:
  3.     - name: build-skaffold-web
  4.       taskRef:
  5.         name: build-push
  6.       params:
  7.         - name: pathToDockerFile
  8.           value: Dockerfile
  9.         - name: pathToContext
  10.           value: /workspace/examples/microservices/leeroy-web

from

Sometimes you will have Pipeline Tasks that need to take asinput the output of a previous Task, for example, an image built by a previousTask.

Express this dependency by adding from on PipelineResourcesthat your Tasks need.

  • The (optional) from key on an input source defines a set of previousPipelineTasks (i.e. the named instance of a Task) in the Pipeline
  • When the from key is specified on an input source, the version of theresource that is from the defined list of tasks is used
  • from can support fan in and fan out
  • The from clause expresses ordering, i.e. thePipeline Task which provides the PipelineResource must runbefore the Pipeline Task which needs that PipelineResource as an input
    • The name of the PipelineResource must correspond to a PipelineResourcefrom the Task that the referenced PipelineTask gives as an output

For example see this Pipeline spec:

  1. - name: build-app
  2.   taskRef:
  3.     name: build-push
  4.   resources:
  5.     outputs:
  6.       - name: image
  7.         resource: my-image
  8. - name: deploy-app
  9.   taskRef:
  10.     name: deploy-kubectl
  11.   resources:
  12.     inputs:
  13.       - name: image
  14.         resource: my-image
  15.         from:
  16.           - build-app

The resource my-image is expected to be given to the deploy-app Task fromthe build-app Task. This means that the PipelineResource my-image mustalso be declared as an output of build-app.

This also means that the build-app Pipeline Task will run before deploy-app,regardless of the order they appear in the spec.

runAfter

Sometimes you will need to have Pipeline Tasks that need torun in a certain order, but they do not have an explicitoutput to input dependency (which isexpressed via from). In this case you can use runAfter to indicatethat a Pipeline Task should be run after one or more previous Pipeline Tasks.

For example see this Pipeline spec:

  1. - name: test-app
  2.   taskRef:
  3.     name: make-test
  4.   resources:
  5.     inputs:
  6.       - name: workspace
  7.         resource: my-repo
  8. - name: build-app
  9.   taskRef:
  10.     name: kaniko-build
  11.   runAfter:
  12.     - test-app
  13.   resources:
  14.     inputs:
  15.       - name: workspace
  16.         resource: my-repo

In this Pipeline, we want to test the code before we build from it, but thereis no output from test-app, so build-app uses runAfter to indicate thattest-app should run before it, regardless of the order they appear in thespec.

retries

Sometimes you need a policy for retrying tasks which have problems such asnetwork errors, missing dependencies or upload problems. Any of those issues mustbe reflected as False (corev1.ConditionFalse) within the TaskRun StatusSucceeded Condition. For that reason there is an optional attribute calledretries which declares how many times that task should be retried in case offailure.

By default and in its absence there are no retries; its value is 0.

  1. tasks:
  2.   - name: build-the-image
  3.     retries: 1
  4.     taskRef:
  5.       name: build-push

In this example, the task “build-the-image” will be executed and if the firstrun fails a second one would triggered. But, if that fails no more wouldtriggered: a max of two executions.

conditions

Sometimes you will need to run tasks only when some conditions are true. The conditions fieldallows you to list a series of references to Conditions that are run before the taskis run. If all of the conditions evaluate to true, the task is run. If any of the conditions are false,the Task is not run. Its status.ConditionSucceeded is set to False with the reason set to ConditionCheckFailed.However, unlike regular task failures, condition failures do not automatically fail the entire pipelinerun – other tasks that are not dependent on the task (via from or runAfter) are still run.

  1. tasks:
  2.   - name: conditional-task
  3.     taskRef:
  4.       name: build-push
  5.     conditions:
  6.       - conditionRef: my-condition
  7.         params:
  8.           - name: my-param
  9.             value: my-value
  10.         resources:
  11.           - name: workspace
  12.             resource: source-repo

In this example, my-condition refers to a Condition custom resource. The build-pushtask will only be executed if the condition evaluates to true.

Resources in conditions can also use the from field to indicate that theyexpect the output of a previous task as input. As with regular Pipeline Tasks, using fromimplies ordering – if task has a condition that takes in an output resource fromanother task, the task producing the output resource will run first:

  1. tasks:
  2.   - name: first-create-file
  3.     taskRef:
  4.       name: create-file
  5.     resources:
  6.       outputs:
  7.         - name: workspace
  8.           resource: source-repo
  9.   - name: then-check
  10.     conditions:
  11.       - conditionRef: "file-exists"
  12.         resources:
  13.           - name: workspace
  14.             resource: source-repo
  15.             from: [first-create-file]
  16.     taskRef:
  17.       name: echo-hello

Timeout

The Timeout property of a Pipeline Task allows a timeout to be defined for a TaskRun thatis part of a PipelineRun. If the TaskRun exceeds the amount of time specified, the TaskRunwill fail and the PipelineRun associated with a Pipeline will fail as well.

There is no default timeout for Pipeline Tasks, so a timeout must be specified with a Pipeline Taskwhen defining a Pipeline if one is needed. An example of a Pipeline Task with a Timeout is shown below:

  1. spec:
  2.   tasks:
  3.     - name: build-the-image
  4.       taskRef:
  5.         name: build-push
  6.       Timeout: "0h1m30s"

The Timeout property is specified as part of the Pipeline Task on the Pipeline spec. The aboveexample has a timeout of one minute and 30 seconds.

Results

Tasks can declare results that they will emit during their execution. These results can be used as values for params in subsequent tasks of a Pipeline. Tekton will infer the ordering of these Tasks to ensure that the Task emitting the results runs before the Task consuming those results in its parameters.

Using a Task result as a value for another Task’s parameter is done with variable substitution. Here is what a Pipeline Task’s param looks like with a result wired into it:

  1. params:
  2.   - name: foo
  3.     value: "$(tasks.previous-task-name.results.bar-result)"

In this example the previous pipeline task has name “previous-task-name” and its result is declared in the Task definition as having name “bar-result”.

For a complete example demonstrating Task Results in a Pipeline see the pipelinerun example.

Ordering

The Pipeline Tasks in a Pipeline can be connected and runin a graph, specifically a Directed Acyclic Graph or DAG. Each of the PipelineTasks is a node, which can be connected with an edge (i.e. a Graph) such that one will runbefore another (i.e. Directed), and the execution will eventually complete(i.e. Acyclic, it will not get caught in infinite loops).

This is done using:

For example see this Pipeline spec:

  1. - name: lint-repo
  2.   taskRef:
  3.     name: pylint
  4.   resources:
  5.     inputs:
  6.       - name: workspace
  7.         resource: my-repo
  8. - name: test-app
  9.   taskRef:
  10.     name: make-test
  11.   resources:
  12.     inputs:
  13.       - name: workspace
  14.         resource: my-repo
  15. - name: build-app
  16.   taskRef:
  17.     name: kaniko-build-app
  18.   runAfter:
  19.     - test-app
  20.   resources:
  21.     inputs:
  22.       - name: workspace
  23.         resource: my-repo
  24.     outputs:
  25.       - name: image
  26.         resource: my-app-image
  27. - name: build-frontend
  28.   taskRef:
  29.     name: kaniko-build-frontend
  30.   runAfter:
  31.     - test-app
  32.   resources:
  33.     inputs:
  34.       - name: workspace
  35.         resource: my-repo
  36.     outputs:
  37.       - name: image
  38.         resource: my-frontend-image
  39. - name: deploy-all
  40.   taskRef:
  41.     name: deploy-kubectl
  42.   resources:
  43.     inputs:
  44.       - name: my-app-image
  45.         resource: my-app-image
  46.         from:
  47.           - build-app
  48.       - name: my-frontend-image
  49.         resource: my-frontend-image
  50.         from:
  51.           - build-frontend

This will result in the following execution graph:

  1.         |            |
  2.         v            v
  3.      test-app    lint-repo
  4.     /        \
  5.    v          v
  6. build-app  build-frontend
  7.    \          /
  8.     v        v
  9.     deploy-all
  • The lint-repo and test-app Pipeline Tasks will begin executingsimultaneously. (They have no from or runAfter clauses.)
  • Once test-app completes, both build-app and build-frontend will beginexecuting simultaneously (both runAfter test-app).
  • When both build-app and build-frontend have completed, deploy-all willexecute (it requires PipelineResources from both Pipeline Tasks).
  • The entire Pipeline will be finished executing after lint-repo anddeploy-all have completed.

Examples

For complete examples, seethe examples folder.