3 - Configuring Pipelines
- Basic Configuration
- Advanced Configuration

3 - Configuring Pipelines

Configuring a pipeline automates the process of triggering and publishing builds. This section describes how to set up a pipeline in a production environment.

The Basic Configuration section provides sequential instruction on how to configure a functional pipeline.
The Advanced Configuration section provides instructions for configuring pipeline options.

Basic Configuration

To configure a functional pipeline for your project, begin by completing the mandatory basic configuration steps.

Pipeline Configuration Outline

Initial configuration of a pipeline in a production environment involves completion of several mandatory procedures.

Note: Before setting up a pipeline for a production environment, we recommend trying the Pipeline Quick Start Guide.

1. Configuring Version Control Providers

Begin configuration of your pipeline by enabling authentication with your version control provider. Rancher Pipeline supports integration with GitHub and GitLab.

Select your provider’s tab below and follow the directions.

From the context menu, open the project for which you’re configuring a pipeline.
From the main menu, select Resources > Pipelines.
Follow the directions displayed to setup an OAuth application in GitHub.

From GitHub, copy the Client ID and Client Secret. Paste them into Rancher.
If you’re using GitHub for enterprise, select Use a private github enterprise installation. Enter the host address of your GitHub installation.
Click Authenticate.
Enable the repository for which you want to run a pipeline. Then click Done.

From the context menu, open the project for which you’re configuring a pipeline.
From the main menu, select Resources > Pipelines.
Follow the directions displayed to setup a GitLab application.

From GitLab, copy the Application ID and Secret. Paste them into Rancher.
If you’re using GitLab for enterprise setup, select Use a private gitlab enterprise installation. Enter the host address of your GitLab installation.
Click Authenticate.
Enable the repository for which you want to run a pipeline. Then click Done.

Note:1. Pipeline uses Gitlab v4 API and the supported Gitlab version is 9.0+.2. If you use GitLab 10.7+ and your Rancher setup is in a local network, enable the Allow requests to the local network from hooks and services option in GitLab admin settings.

Result: A pipeline is added to the project.

2. Configuring Pipeline Stages and Steps

Now that the pipeline is added to your project, you need to configure its automated stages and steps. For your convenience, there are multiple built-in step types for dedicated tasks.

From your project’s Pipeline tab, find your new pipeline, and select Ellipsis (…) > Edit Config.

Note: When configuring a pipeline, it takes a few moments for Rancher to check for an existing pipeline configuration.

Click Configure pipeline for this branch.
Add stages to your pipeline execution by clicking Add Stage.
Add steps to each stage by clicking Add a Step. You can add multiple steps to each stage.

Note: As you build out each stage and step, click Show advanced options to make Advanced Configurations, such as rules to trigger or skip pipeline actions, add environment variables, or inject environment variables using secrets. Advanced options are available the pipeline, each stage, and each individual step.

Step types available include:

Clone

The first stage is preserved to be a cloning step that checks out source code from your repo. Rancher handles the cloning of the git repository. This action is equivalent to git clone <repository_link> <workspace_dir>.

Run Script

The Run Script step executes arbitrary commands in the workspace inside a specified container. You can use it to build, test and do more, given whatever utilities the base image provides. For your convenience you can use variables to refer to metadata of a pipeline execution. Please go to the Pipeline Variable Reference for the list of available variables.

From the Step Type drop-down, choose Run Script and fill in the form.
Click Add.

# example
stages:
- name: Build something
  steps:
  - runScriptConfig:
      image: golang
      shellScript: go build

Build and Publish Images

The Build and Publish Image step builds and publishes a Docker image. This process requires a Dockerfile in your source code’s repository to complete successfully.

From the Step Type drop-down, choose Build and Publish.
Fill in the rest of the form. Descriptions for each field are listed below. When you’re done, click Add.

FieldDescriptionDockerfile PathThe relative path to the Dockerfile in the source code repo. By default, this path is ./Dockerfile, which assumes the Dockerfile is in the root directory. You can set it to other paths in different use cases (./path/to/myDockerfile for example).Image NameThe image name in name:tag format. The registry address is not required. For example, to build example.com/repo/my-image:dev, enter repo/my-image:dev.Push image to remote repositoryAn option to set the registry that publishes the image that’s built. To use this option, enable it and choose a registry from the drop-down. If this option is disabled, the image is pushed to the internal registry.Build Context (Show advanced options)By default, the root directory of the source code (.). For more details, see the Docker build command documentation.

# example
stages:
- name: Publish Image
  steps:
  - publishImageConfig:
      dockerfilePath: ./Dockerfile
      buildContext: .
      tag: repo/app:v1
      pushRemote: true
      registry: example.com

You can use specific arguments for Docker daemon and the build. They are not exposed in the UI, but they are available in pipeline YAML format, as indicated in the example above. Available variables includes:

Variable NameDescriptionPLUGIN_DRY_RUNDisable docker pushPLUGIN_DEBUGDocker daemon executes in debug modePLUGIN_MIRRORDocker daemon registry mirrorPLUGIN_INSECUREDocker daemon allows insecure registriesPLUGIN_BUILD_ARGSDocker build args, a comma separated list

Deploy YAML

This step deploys arbitrary Kubernetes resources to the project. This deployment requires a Kubernetes manifest file to be present in the source code repository. Pipeline variable substitution is supported in the manifest file. You can view an example file at GitHub. For available variables, refer to Pipeline Variable Reference.

From the Step Type drop-down, choose Deploy YAML and fill in the form.
Enter the YAML Path, which is the path to the manifest file in the source code.
Click Add.

# example
stages:
- name: Deploy
  steps:
  - applyYamlConfig:
      path: ./deployment.yaml

When you’re finished adding stages and steps, click Done.

3. Running the Pipeline

Run your pipeline for the first time. From the Pipeline tab, find your pipeline and select Ellipsis (…) > Run.

During this initial run, your pipeline is tested, and the following pipeline components are deployed to your project as workloads in a new namespace dedicated to the pipeline:

docker-registry
jenkins
minioThis process takes several minutes. When it completes, you can view each pipeline component from the project Workloads tab.

4. Configuring Persistent Data for Pipeline Components

The internal Docker registry and the Minio workloads use ephemeral volumes by default. This default storage works out-of-the-box and makes testing easy, but you lose the build images and build logs if the node running the Docker Registry or Minio fails. In most cases this is fine. If you want build images and logs to survive node failures, you can configure the Docker Registry and Minio to use persistent volumes.

Complete both A—Configuring Persistent Data for Docker RegistryandB—Configuring Persistent Data for Minio.

Prerequisites (for both parts A and B):
Persistent volumes must be available for the cluster.

A. Configuring Persistent Data for Docker Registry

From the project that you’re configuring a pipeline for, select the Workloads tab.
Find the docker-registry workload and select Ellipsis (…) > Edit.
Scroll to the Volumes section and expand it. Make one of the following selections from the Add Volume menu, which is near the bottom of the section:
- Add Volume > Add a new persistent volume (claim)
- Add Volume > Use an existing persistent volume (claim)
Complete the form that displays to choose a persistent volume for the internal Docker registry.

Enter a Name for the volume claim.
- Select a volume claim Source:
  - If you select Use a Storage Class to provision a new persistent volume, select a Storage Class and enter a Capacity.
  - If you select Use an existing persistent volume, choose a Persistent Volume from the drop-down.
- From the Customize section, choose the read/write access for the volume.
- Click Define.

Enter a Name for the volume claim.
- Choose a Persistent Volume Claim from the drop-down.
- From the Customize section, choose the read/write access for the volume.
- Click Define.

From the Mount Point field, enter /var/lib/registry, which is the data storage path inside the Docker registry container.
Click Upgrade.

B. Configuring Persistent Data for Minio

From the Workloads tab, find the minio workload and select Ellipsis (…) > Edit.
Scroll to the Volumes section and expand it. Make one of the following selections from the Add Volume menu, which is near the bottom of the section:
- Add Volume > Add a new persistent volume (claim)
- Add Volume > Use an existing persistent volume (claim)
Complete the form that displays to choose a persistent volume for the internal Docker registry.

Enter a Name for the volume claim.
- Select a volume claim Source:
  - If you select Use a Storage Class to provision a new persistent volume, select a Storage Class and enter a Capacity.
  - If you select Use an existing persistent volume, choose a Persistent Volume from the drop-down.
- From the Customize section, choose the read/write access for the volume.
- Click Define.

Enter a Name for the volume claim.
- Choose a Persistent Volume Claim from the drop-down.
- From the Customize section, choose the read/write access for the volume.
- Click Define.

From the Mount Point field, enter /data, which is the data storage path inside the Minio container.
Click Upgrade.

Result: Persistent storage is configured for your pipeline components.

Advanced Configuration

During the process of configuring a pipeline, you can configure advanced options for triggering the pipeline or configuring environment variables.

Configuring Pipeline Trigger Rules

When a repository is enabled, a webhook for it is automatically set in the version control system. By default, the project pipeline is triggered by a push event to a specific repository, but you can add (or change) events that trigger a build, such as a pull request or a tagging.

Trigger rules come in two types:

Run this when:

This type of rule starts the pipeline, stage, or step when a trigger explicitly occurs.

Do Not Run this when:

If all conditions evaluate to true, then the pipeline/stage/step is executed. Otherwise it is skipped. When a stage/step is skipped, it is considered successful and follow-up stages/steps continue to run. Wildcard character (*) expansion is supported in conditions.

You can configure trigger rules for the entire pipeline in two different contexts:

During Initial Pipeline Configuration

From the context menu, open the project for which you’ve configured a pipeline. Then select the Pipelines tab.
From the pipeline for which you want to edit build triggers, select Ellipsis (…) > Edit Config.
Click Show advanced options.
From Trigger Rules, configure rules to run or skip the pipeline.
- Click Add Rule. In the Value field, enter the name of the branch that triggers the pipeline.
- Optional: Add more branches that trigger a build.

While Editing Pipeline Settings

After you’ve configured a pipeline, you can go back and choose the events that trigger a pipeline execution.

Note: This option is not available for example repositories.

From the context menu, open the project for which you’ve configured a pipeline. Then select the Pipelines tab.
From the pipeline for which you want to edit build triggers, select Ellipsis (…) > Setting.
Select (or clear) the events that you want to trigger a pipeline execution.
Click Save.

From the context menu, open the project for which you’ve configured a pipeline. Then select the Pipelines tab.
From the pipeline for which you want to edit triggers, select Ellipsis (…) > Edit Config.
From the pipeline stage that you want to configure a trigger for, click the Edit icon.
Click Show advanced options.
Add one or more trigger rules.
- Click Add Rule.
- Choose the Type that triggers the stage.

TypeValueBranchThe name of the branch that triggers the stage.EventThe type of event that triggers the stage (Push, Pull Request, Tag).

Click Save.

From the context menu, open the project for which you’ve configured a pipeline. Then select the Pipelines tab.
From the pipeline for which you want to edit triggers, select Ellipsis (…) > Edit Config.
From the pipeline step that you want to configure a trigger for, click the Edit icon.
Click Show advanced options.
Add one or more trigger rules.
- Click Add Rule.
- Choose the Type that triggers the step.

TypeValueBranchThe name of the branch that triggers the stage.EventThe type of event that triggers the stage (Push, Pull Request, Tag).

Click Save.

# example
stages:
  - name: Build something
    # Conditions for stages
    when:
      branch: master
      event: [ push, pull_request ]
    # Multiple steps run concurrently
    steps:
    - runScriptConfig:
        image: busybox
        shellScript: date -R
      # Conditions for steps
      when:
        branch: [ master, dev ]
        event: push
# branch conditions for the pipeline
branch:
  include: [ master, feature/*]
  exlclude: [ dev ]

Configuring Timeouts

Each pipeline execution has a default timeout of 60 minutes. If the pipeline execution cannot complete within its timeout period, the pipeline is aborted. If a pipeline has more executions than can be completed in 60 minutes,

From the context menu, open the project for which you’ve configured a pipeline. Then select the Pipelines tab.
From the pipeline for which you want to edit the timeout, select Ellipsis (…) > Edit Config.
Click Show advanced options.
Enter a new value in the Timeout field.

# example
stages:
  - name: Build something
    steps:
    - runScriptConfig:
        image: busybox
        shellScript: ls
timeout: 30

Configuring Environment Variables

When configuring a pipeline, you can use environment variables to configure the step’s script.

From the context menu, open the project for which you’ve configured a pipeline. Then select the Pipelines tab.
From the pipeline in which you want to use environment variables, select Ellipsis (…) > Edit Config.
Click the Edit icon for the step in which you want to use environment variables.
Click Show advanced options.
Click Add Variable, and then enter a key and value in the fields that appear. Add more variables if needed.
Edit the script, adding your environment variable(s).
Click Save.

# example
stages:
  - name: Build something
    steps:
    - runScriptConfig:
        image: busybox
        shellScript: echo ${FIRST_KEY} && echo ${SECOND_KEY}
      env:
        FIRST_KEY: VALUE
        SECOND_KEY: VALUE2

Configuring Pipeline Secrets

If you need to use security-sensitive information in your pipeline scripts (like a password), you can pass them in using Kubernetes secrets.

Prerequisite: Create a secret for your project for use in pipelines.
Note: Secret injection is disabled on pull request events.

From the context menu, open the project for which you’ve configured a pipeline. Then select the Pipelines tab.
From the pipeline in which you want to use environment variables, select Ellipsis (…) > Edit Config.
Click the Edit icon for the step in which you want to use environment variables.
Click Show advanced options.
Click Add From Secret. Select the secret file that you want to use. Then choose a key. Optionally, you can enter an alias for the key.
Click Save.

# example
stages:
  - name: Build something
    steps:
    - runScriptConfig:
        image: busybox
        shellScript: echo ${ALIAS_ENV}
      # environment variables from project secrets
      envFrom:
      - sourceName: my-secret
        sourceKey: secret-key
        targetKey: ALIAS_ENV

Configuring the Executor Quota

The executor quota decides how many builds can run simultaneously in the project. If the number of triggered builds exceeds the quota, subsequent builds will queue until a vacancy opens. By default, the quota is 2, but you can change it.

From the context menu, open the project for which you’ve configured a pipeline.
From the main menu, select Resources > Pipelines.
From The maximum number of pipeline executors increment the Scale up or down to change the quota. A value of 0 or less removes the quota limit.