Pipelines

As of Rancher v2.5, Git-based deployment pipelines are now deprecated. We recommend handling pipelines with Rancher Continuous Delivery powered by Fleet. To get to Fleet in Rancher, click ☰ > Continuous Delivery. Note that pipelines in Kubernetes 1.21+ are no longer supported.

Rancher’s pipeline provides a simple CI/CD experience. Use it to automatically checkout code, run builds or scripts, publish Docker images or catalog applications, and deploy the updated software to users.

Setting up a pipeline can help developers deliver new software as quickly and efficiently as possible. Using Rancher, you can integrate with a GitHub repository to setup a continuous integration (CI) pipeline.

After configuring Rancher and GitHub, you can deploy containers running Jenkins to automate a pipeline execution:

  • Build your application from code to image.
  • Validate your builds.
  • Deploy your build images to your cluster.

  • Run unit tests.

  • Run regression tests.

Note: Rancher’s pipeline provides a simple CI/CD experience, but it does not offer the full power and flexibility of and is not a replacement of enterprise-grade Jenkins or other CI tools your team uses.

This section covers the following topics:

Concepts

For an explanation of concepts and terminology used in this section, refer to this page.

How Pipelines Work

After enabling the ability to use pipelines in a project, you can configure multiple pipelines in each project. Each pipeline is unique and can be configured independently.

A pipeline is configured off of a group of files that are checked into source code repositories. Users can configure their pipelines either through the Rancher UI or by adding a .rancher-pipeline.yml into the repository.

Before pipelines can be configured, you will need to configure authentication to your version control provider, e.g. GitHub, GitLab, Bitbucket. If you haven’t configured a version control provider, you can always use Rancher’s example repositories to view some common pipeline deployments.

When you configure a pipeline in one of your projects, a namespace specifically for the pipeline is automatically created. The following components are deployed to it:

  • Jenkins:

    The pipeline’s build engine. Because project users do not directly interact with Jenkins, it’s managed and locked.

    Note: There is no option to use existing Jenkins deployments as the pipeline engine.

  • Docker Registry:

    Out-of-the-box, the default target for your build-publish step is an internal Docker Registry. However, you can make configurations to push to a remote registry instead. The internal Docker Registry is only accessible from cluster nodes and cannot be directly accessed by users. Images are not persisted beyond the lifetime of the pipeline and should only be used in pipeline runs. If you need to access your images outside of pipeline runs, please push to an external registry.

  • Minio:

    Minio storage is used to store the logs for pipeline executions.

Note: The managed Jenkins instance works statelessly, so don’t worry about its data persistency. The Docker Registry and Minio instances use ephemeral volumes by default, which is fine for most use cases. If you want to make sure pipeline logs can survive node failures, you can configure persistent volumes for them, as described in data persistency for pipeline components.

Roles-based Access Control for Pipelines

If you can access a project, you can enable repositories to start building pipelines.

Only administrators, cluster owners or members, or project owners can configure version control providers and manage global pipeline execution settings.

Project members can only configure repositories and pipelines.

Setting up Pipelines

Prerequisite

Prerequisite: Because the pipelines app was deprecated in favor of Fleet, you will need to turn on the feature flag for legacy features before using pipelines. Note that pipelines in Kubernetes 1.21+ are no longer supported.

  1. In the upper left corner, click ☰ > Global Settings.
  2. Click Feature Flags.
  3. Go to the legacy feature flag and click ⋮ > Activate.
  1. Configure version control providers
  2. Configure repositories
  3. Configure the pipeline

1. Configure Version Control Providers

Before you can start configuring a pipeline for your repository, you must configure and authorize a version control provider:

  • GitHub
  • GitLab
  • Bitbucket

Select your provider’s tab below and follow the directions.

  1. In the upper left corner, click ☰ > Cluster Management.
  2. Go to the cluster where you want to configure pipelines and click Explore.
  3. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
  4. In the left navigation bar, click Legacy > Project > Pipelines.
  5. Click the Configuration tab.
  6. Follow the directions displayed to Setup a Github application. Rancher redirects you to Github to set up an OAuth App in Github.
  7. From GitHub, copy the Client ID and Client Secret. Paste them into Rancher.
  8. If you’re using GitHub for enterprise, select Use a private github enterprise installation. Enter the host address of your GitHub installation.

  9. Click Authenticate.

  10. In the upper left corner, click ☰ > Cluster Management.

  11. Go to the cluster where you want to configure pipelines and click Explore.
  12. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
  13. In the left navigation bar, click Legacy > Project > Pipelines.
  14. Click the Configuration tab.
  15. Click GitLab.
  16. Follow the directions displayed to Setup a GitLab application. Rancher redirects you to GitLab.
  17. From GitLab, copy the Application ID and Secret. Paste them into Rancher.
  18. If you’re using GitLab for enterprise setup, select Use a private gitlab enterprise installation. Enter the host address of your GitLab installation.
  19. Click Authenticate.

Note: 1. Pipeline uses Gitlab v4 API and the supported Gitlab version is 9.0+.
2. If you use GitLab 10.7+ and your Rancher setup is in a local network, enable the Allow requests to the local network from hooks and services option in GitLab admin settings.

  1. In the upper left corner, click ☰ > Cluster Management.
  2. Go to the cluster where you want to configure pipelines and click Explore.
  3. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
  4. In the left navigation bar, click Legacy > Project > Pipelines.
  5. Click the Configuration tab.
  6. Click Bitbucket and leave Use Bitbucket Cloud selected by default.
  7. Follow the directions displayed to Setup a Bitbucket Cloud application. Rancher redirects you to Bitbucket to setup an OAuth consumer in Bitbucket.
  8. From Bitbucket, copy the consumer Key and Secret. Paste them into Rancher.

  9. Click Authenticate.

  10. In the upper left corner, click ☰ > Cluster Management.

  11. Go to the cluster where you want to configure pipelines and click Explore.
  12. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
  13. In the left navigation bar, click Legacy > Project > Pipelines.
  14. Click the Configuration tab.
  15. Click Bitbucket and choose the Use private Bitbucket Server setup option.
  16. Follow the directions displayed to Setup a Bitbucket Server application.
  17. Enter the host address of your Bitbucket server installation.
  18. Click Authenticate.

Note: Bitbucket server needs to do SSL verification when sending webhooks to Rancher. Please ensure that Rancher server’s certificate is trusted by the Bitbucket server. There are two options:

  1. Setup Rancher server with a certificate from a trusted CA.
  2. If you’re using self-signed certificates, import Rancher server’s certificate to the Bitbucket server. For instructions, see the Bitbucket server documentation for configuring self-signed certificates.

Result: After the version control provider is authenticated, you will be automatically re-directed to start configuring which repositories you want start using with a pipeline.

2. Configure Repositories

After the version control provider is authorized, you are automatically re-directed to start configuring which repositories that you want start using pipelines with. Even if someone else has set up the version control provider, you will see their repositories and can build a pipeline.

  1. In the upper left corner, click ☰ > Cluster Management.
  2. Go to the cluster where you want to configure pipelines and click Explore.
  3. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
  4. In the left navigation bar, click Legacy > Project > Pipelines.

  5. Click on Configure Repositories.

  6. A list of repositories are displayed. If you are configuring repositories the first time, click on Authorize & Fetch Your Own Repositories to fetch your repository list.

  7. For each repository that you want to set up a pipeline, click on Enable.

  8. When you’re done enabling all your repositories, click on Done.

Results: You have a list of repositories that you can start configuring pipelines for.

3. Configure the Pipeline

Now that repositories are added to your project, you can start configuring the pipeline by adding automated stages and steps. For your convenience, there are multiple built-in step types for dedicated tasks.

  1. In the upper left corner, click ☰ > Cluster Management.
  2. Go to the cluster where you want to configure pipelines and click Explore.
  3. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
  4. In the left navigation bar, click Legacy > Project > Pipelines.
  5. Find the repository that you want to set up a pipeline for.

  6. Configure the pipeline through the UI or using a yaml file in the repository, i.e. .rancher-pipeline.yml or .rancher-pipeline.yaml. Pipeline configuration is split into stages and steps. Stages must fully complete before moving onto the next stage, but steps in a stage run concurrently. For each stage, you can add different step types. Note: As you build out each step, there are different advanced options based on the step type. Advanced options include trigger rules, environment variables, and secrets. For more information on configuring the pipeline through the UI or the YAML file, refer to the pipeline configuration reference.

    • If you are going to use the UI, select the vertical ⋮ > Edit Config to configure the pipeline using the UI. After the pipeline is configured, you must view the YAML file and push it to the repository.
    • If you are going to use the YAML file, select the vertical ⋮ > View/Edit YAML to configure the pipeline. If you choose to use a YAML file, you need to push it to the repository after any changes in order for it to be updated in the repository. When editing the pipeline configuration, it takes a few moments for Rancher to check for an existing pipeline configuration.

  7. Select which branch to use from the list of branches.

  8. Optional: Set up notifications.

  9. Set up the trigger rules for the pipeline.

  10. Enter a Timeout for the pipeline.

  11. When all the stages and steps are configured, click Done.

Results: Your pipeline is now configured and ready to be run.

Pipeline Configuration Reference

Refer to this page for details on how to configure a pipeline to:

  • Run a script
  • Build and publish images
  • Publish catalog templates
  • Deploy YAML
  • Deploy a catalog app

The configuration reference also covers how to configure:

  • Notifications
  • Timeouts
  • The rules that trigger a pipeline
  • Environment variables
  • Secrets

Running your Pipelines

Run your pipeline for the first time. Find your pipeline and select the vertical ⋮ > Run.

During this initial run, your pipeline is tested, and the following pipeline components are deployed to your project as workloads in a new namespace dedicated to the pipeline:

  • docker-registry
  • jenkins
  • minio

This process takes several minutes. When it completes, you can view each pipeline component from the project Workloads tab.

Triggering a Pipeline

When a repository is enabled, a webhook is automatically set in the version control provider. By default, the pipeline is triggered by a push event to a repository, but you can modify the event(s) that trigger running the pipeline.

Available Events:

  • Push: Whenever a commit is pushed to the branch in the repository, the pipeline is triggered.
  • Pull Request: Whenever a pull request is made to the repository, the pipeline is triggered.
  • Tag: When a tag is created in the repository, the pipeline is triggered.

Note: This option doesn’t exist for Rancher’s example repositories.

Modifying the Event Triggers for the Repository

  1. In the upper left corner, click ☰ > Cluster Management.
  2. Go to the cluster where you want to configure pipelines and click Explore.
  3. In the dropdown menu in the top navigation bar, select the project where you want to configure pipelines.
  4. In the left navigation bar, click Legacy > Project > Pipelines.
  5. Find the repository where you want to modify the event triggers. Select the vertical ⋮ > Setting.
  6. Select which event triggers (Push, Pull Request or Tag) you want for the repository.
  7. Click Save.