Configuring Jenkins images

OKD provides a container image for running Jenkins. This image provides a Jenkins server instance, which can be used to set up a basic flow for continuous testing, integration, and delivery.

The image is based on the Red Hat Universal Base Images (UBI).

OKD follows the LTS release of Jenkins. OKD provides an image that contains Jenkins 2.x.

The OKD Jenkins images are available on Quay.io or registry.redhat.io.

For example:

$ podman pull registry.redhat.io/openshift4/ose-jenkins:<v4.3.0>

To use these images, you can either access them directly from these registries or push them into your OKD container image registry. Additionally, you can create an image stream that points to the image, either in your container image registry or at the external location. Your OKD resources can then reference the image stream.

But for convenience, OKD provides image streams in the openshift namespace for the core Jenkins image as well as the example Agent images provided for OKD integration with Jenkins.

Configuration and customization

You can manage Jenkins authentication in two ways:

• OKD OAuth authentication provided by the OKD Login plug-in.

• Standard authentication provided by Jenkins.

OKD OAuth authentication

OAuth authentication is activated by configuring options on the Configure Global Security panel in the Jenkins UI, or by setting the OPENSHIFT_ENABLE_OAUTH environment variable on the Jenkins Deployment configuration to anything other than false. This activates the OKD Login plug-in, which retrieves the configuration information from pod data or by interacting with the OKD API server.

Valid credentials are controlled by the OKD identity provider.

Jenkins supports both browser and non-browser access.

Valid users are automatically added to the Jenkins authorization matrix at log in, where OKD roles dictate the specific Jenkins permissions that users have. The roles used by default are the predefined admin, edit, and view. The login plug-in executes self-SAR requests against those roles in the project or namespace that Jenkins is running in.

Users with the admin role have the traditional Jenkins administrative user permissions. Users with the edit or view role have progressively fewer permissions.

The default OKD admin, edit, and view roles and the Jenkins permissions those roles are assigned in the Jenkins instance are configurable.

When running Jenkins in an OKD pod, the login plug-in looks for a config map named openshift-jenkins-login-plugin-config in the namespace that Jenkins is running in.

If this plug-in finds and can read in that config map, you can define the role to Jenkins Permission mappings. Specifically:

• The login plug-in treats the key and value pairs in the config map as Jenkins permission to OKD role mappings.

• The key is the Jenkins permission group short ID and the Jenkins permission short ID, with those two separated by a hyphen character.

• If you want to add the Overall Jenkins Administer permission to an OKD role, the key should be Overall-Administer.

• To get a sense of which permission groups and permissions IDs are available, go to the matrix authorization page in the Jenkins console and IDs for the groups and individual permissions in the table they provide.

• The value of the key and value pair is the list of OKD roles the permission should apply to, with each role separated by a comma.

• If you want to add the Overall Jenkins Administer permission to both the default admin and edit roles, as well as a new Jenkins role you have created, the value for the key Overall-Administer would be admin,edit,jenkins.

Jenkins users’ permissions that are stored can be changed after the users are initially established. The OKD Login plug-in polls the OKD API server for permissions and updates the permissions stored in Jenkins for each user with the permissions retrieved from OKD. If the Jenkins UI is used to update permissions for a Jenkins user, the permission changes are overwritten the next time the plug-in polls OKD.

You can control how often the polling occurs with the OPENSHIFT_PERMISSIONS_POLL_INTERVAL environment variable. The default polling interval is five minutes.

The easiest way to create a new Jenkins service using OAuth authentication is to use a template.

Jenkins authentication

Jenkins authentication is used by default if the image is run directly, without using a template.

The first time Jenkins starts, the configuration is created along with the administrator user and password. The default user credentials are admin and password. Configure the default password by setting the JENKINS_PASSWORD environment variable when using, and only when using, standard Jenkins authentication.

Procedure

• Create a Jenkins application that uses standard Jenkins authentication:

  $ oc new-app -e \
      JENKINS_PASSWORD=<password> \
      openshift4/ose-jenkins

Jenkins environment variables

The Jenkins server can be configured with the following environment variables:

Providing Jenkins cross project access

If you are going to run Jenkins somewhere other than your same project, you must provide an access token to Jenkins to access your project.

Procedure

1. Identify the secret for the service account that has appropriate permissions to access the project Jenkins must access:

   $ oc describe serviceaccount jenkins

   Example output

   Name:       default
   Labels:     <none>
   Secrets:    {  jenkins-token-uyswp    }
               {  jenkins-dockercfg-xcr3d    }
   Tokens:     jenkins-token-izv1u
               jenkins-token-uyswp

   In this case the secret is named jenkins-token-uyswp.

2. Retrieve the token from the secret:

   $ oc describe secret <secret name from above>

   Example output

   Name:       jenkins-token-uyswp
   Labels:     <none>
   Annotations:    kubernetes.io/service-account.name=jenkins,kubernetes.io/service-account.uid=32f5b661-2a8f-11e5-9528-3c970e3bf0b7
   Type:   kubernetes.io/service-account-token
   Data
   ====
   ca.crt: 1066 bytes
   token:  eyJhbGc..<content cut>....wRA

   The token parameter contains the token value Jenkins requires to access the project.

Jenkins cross volume mount points

The Jenkins image can be run with mounted volumes to enable persistent storage for the configuration:

• /var/lib/jenkins is the data directory where Jenkins stores configuration files, including job definitions.

Customizing the Jenkins image through source-to-image

To customize the official OKD Jenkins image, you can use the image as a source-to-image (S2I) builder.

You can use S2I to copy your custom Jenkins jobs definitions, add additional plug-ins, or replace the provided config.xml file with your own, custom, configuration.

To include your modifications in the Jenkins image, you must have a Git repository with the following directory structure:

plugins

This directory contains those binary Jenkins plug-ins you want to copy into Jenkins.

plugins.txt

This file lists the plug-ins you want to install using the following syntax:

pluginId:pluginVersion

configuration/jobs

This directory contains the Jenkins job definitions.

configuration/config.xml

This file contains your custom Jenkins configuration.

The contents of the configuration/ directory is copied to the /var/lib/jenkins/ directory, so you can also include additional files, such as credentials.xml, there.

Sample build configuration customizes the Jenkins image in OKD

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: custom-jenkins-build
spec:
  source:                       (1)
    git:
      uri: https://github.com/custom/repository
    type: Git
  strategy:                     (2)
    sourceStrategy:
      from:
        kind: ImageStreamTag
        name: jenkins:2
        namespace: openshift
    type: Source
  output:                       (3)
    to:
      kind: ImageStreamTag
      name: custom-jenkins:latest

Configuring the Jenkins Kubernetes plug-in

The OKD Jenkins image includes the pre-installed Kubernetes plug-in that allows Jenkins agents to be dynamically provisioned on multiple container hosts using Kubernetes and OKD.

To use the Kubernetes plug-in, OKD provides images that are suitable for use as Jenkins agents, including the Base, Maven, and Node.js images.

Both the Maven and Node.js agent images are automatically configured as Kubernetes pod template images within the OKD Jenkins image configuration for the Kubernetes plug-in. That configuration includes labels for each of the images that can be applied to any of your Jenkins jobs under their Restrict where this project can be run setting. If the label is applied, jobs run under an OKD pod running the respective agent image.

The Jenkins image also provides auto-discovery and auto-configuration of additional agent images for the Kubernetes plug-in.

With the OKD sync plug-in, the Jenkins image on Jenkins startup searches for the following within the project that it is running or the projects specifically listed in the plug-in’s configuration:

• Image streams that have the label role set to jenkins-agent.

• Image stream tags that have the annotation role set to jenkins-agent.

• Config maps that have the label role set to jenkins-agent.

When it finds an image stream with the appropriate label, or image stream tag with the appropriate annotation, it generates the corresponding Kubernetes plug-in configuration so you can assign your Jenkins jobs to run in a pod that runs the container image that is provided by the image stream.

The name and image references of the image stream or image stream tag are mapped to the name and image fields in the Kubernetes plug-in pod template. You can control the label field of the Kubernetes plug-in pod template by setting an annotation on the image stream or image stream tag object with the key agent-label. Otherwise, the name is used as the label.

When it finds a config map with the appropriate label, it assumes that any values in the key-value data payload of the config map contain Extensible Markup Language (XML) that is consistent with the configuration format for Jenkins and the Kubernetes plug-in pod templates. One key benefit of using config maps, rather than image streams or image stream tags, is that you can control all the parameters of the Kubernetes plug-in pod template.

Sample config map for jenkins-agent

kind: ConfigMap
apiVersion: v1
metadata:
  name: jenkins-agent
  labels:
    role: jenkins-agent
data:
  template1: |-
    <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
      <inheritFrom></inheritFrom>
      <name>template1</name>
      <instanceCap>2147483647</instanceCap>
      <idleMinutes>0</idleMinutes>
      <label>template1</label>
      <serviceAccount>jenkins</serviceAccount>
      <nodeSelector></nodeSelector>
      <volumes/>
      <containers>
        <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
          <name>jnlp</name>
          <image>openshift/jenkins-agent-maven-35-centos7:v3.10</image>
          <privileged>false</privileged>
          <alwaysPullImage>true</alwaysPullImage>
          <workingDir>/tmp</workingDir>
          <command></command>
          <args>${computer.jnlpmac} ${computer.name}</args>
          <ttyEnabled>false</ttyEnabled>
          <resourceRequestCpu></resourceRequestCpu>
          <resourceRequestMemory></resourceRequestMemory>
          <resourceLimitCpu></resourceLimitCpu>
          <resourceLimitMemory></resourceLimitMemory>
          <envVars/>
        </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
      </containers>
      <envVars/>
      <annotations/>
      <imagePullSecrets/>
      <nodeProperties/>
    </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>

The following example shows two containers that reference image streams that are present in the openshift namespace. One container handles the JNLP contract for launching Pods as Jenkins Agents. The other container uses an image with tools for building code in a particular coding language:

kind: ConfigMap
apiVersion: v1
metadata:
  name: jenkins-agent
  labels:
    role: jenkins-agent
data:
  template2: |-
        <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
          <inheritFrom></inheritFrom>
          <name>template2</name>
          <instanceCap>2147483647</instanceCap>
          <idleMinutes>0</idleMinutes>
          <label>template2</label>
          <serviceAccount>jenkins</serviceAccount>
          <nodeSelector></nodeSelector>
          <volumes/>
          <containers>
            <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
              <name>jnlp</name>
              <image>image-registry.openshift-image-registry.svc:5000/openshift/jenkins-agent-base:latest</image>
              <privileged>false</privileged>
              <alwaysPullImage>true</alwaysPullImage>
              <workingDir>/home/jenkins/agent</workingDir>
              <command></command>
              <args>\$(JENKINS_SECRET) \$(JENKINS_NAME)</args>
              <ttyEnabled>false</ttyEnabled>
              <resourceRequestCpu></resourceRequestCpu>
              <resourceRequestMemory></resourceRequestMemory>
              <resourceLimitCpu></resourceLimitCpu>
              <resourceLimitMemory></resourceLimitMemory>
              <envVars/>
            </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
            <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
              <name>java</name>
              <image>image-registry.openshift-image-registry.svc:5000/openshift/java:latest</image>
              <privileged>false</privileged>
              <alwaysPullImage>true</alwaysPullImage>
              <workingDir>/home/jenkins/agent</workingDir>
              <command>cat</command>
              <args></args>
              <ttyEnabled>true</ttyEnabled>
              <resourceRequestCpu></resourceRequestCpu>
              <resourceRequestMemory></resourceRequestMemory>
              <resourceLimitCpu></resourceLimitCpu>
              <resourceLimitMemory></resourceLimitMemory>
              <envVars/>
            </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
          </containers>
          <envVars/>
          <annotations/>
          <imagePullSecrets/>
          <nodeProperties/>
        </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>

After it is installed, the OKD Sync plug-in monitors the API server of OKD for updates to image streams, image stream tags, and config maps and adjusts the configuration of the Kubernetes plug-in.

The following rules apply:

• Removing the label or annotation from the config map, image stream, or image stream tag results in the deletion of any existing PodTemplate from the configuration of the Kubernetes plug-in.

• If those objects are removed, the corresponding configuration is removed from the Kubernetes plug-in.

• Either creating appropriately labeled or annotated ConfigMap, ImageStream, or ImageStreamTag objects, or the adding of labels after their initial creation, leads to creating of a PodTemplate in the Kubernetes-plugin configuration.

• In the case of the PodTemplate by config map form, changes to the config map data for the PodTemplate are applied to the PodTemplate settings in the Kubernetes plug-in configuration and overrides any changes that were made to the PodTemplate through the Jenkins UI between changes to the config map.

To use a container image as a Jenkins agent, the image must run the agent as an entry point. For more details, see the official Jenkins documentation.

Jenkins permissions

If in the config map the <serviceAccount> element of the pod template XML is the OKD service account used for the resulting pod, the service account credentials are mounted into the pod. The permissions are associated with the service account and control which operations against the OKD master are allowed from the pod.

Consider the following scenario with service accounts used for the pod, which is launched by the Kubernetes Plug-in that runs in the OKD Jenkins image.

If you use the example template for Jenkins that is provided by OKD, the jenkins service account is defined with the edit role for the project Jenkins runs in, and the master Jenkins pod has that service account mounted.

The two default Maven and NodeJS pod templates that are injected into the Jenkins configuration are also set to use the same service account as the Jenkins master.

• Any pod templates that are automatically discovered by the OKD sync plug-in because their image streams or image stream tags have the required label or annotations are configured to use the Jenkins master service account as their service account.

• For the other ways you can provide a pod template definition into Jenkins and the Kubernetes plug-in, you have to explicitly specify the service account to use. Those other ways include the Jenkins console, the podTemplate pipeline DSL that is provided by the Kubernetes plug-in, or labeling a config map whose data is the XML configuration for a pod template.

• If you do not specify a value for the service account, the default service account is used.

• Ensure that whatever service account is used has the necessary permissions, roles, and so on defined within OKD to manipulate whatever projects you choose to manipulate from the within the pod.

Creating a Jenkins service from a template

Templates provide parameter fields to define all the environment variables with predefined default values. OKD provides templates to make creating a new Jenkins service easy. The Jenkins templates should be registered in the default openshift project by your cluster administrator during the initial cluster setup.

The two available templates both define deployment configuration and a service. The templates differ in their storage strategy, which affects whether the Jenkins content persists across a pod restart.

• jenkins-ephemeral uses ephemeral storage. On pod restart, all data is lost. This template is only useful for development or testing.

• jenkins-persistent uses a Persistent Volume (PV) store. Data survives a pod restart.

To use a PV store, the cluster administrator must define a PV pool in the OKD deployment.

After you select which template you want, you must instantiate the template to be able to use Jenkins.

Procedure

1. Create a new Jenkins application using one of the following methods:

   • A PV:

     $ oc new-app jenkins-persistent

   • Or an emptyDir type volume where configuration does not persist across pod restarts:

     $ oc new-app jenkins-ephemeral

With both templates, you can run oc describe on them to see all the parameters available for overriding.

For example:

$ oc describe jenkins-ephemeral

Using the Jenkins Kubernetes plug-in

In the following example, the openshift-jee-sample BuildConfig object causes a Jenkins Maven agent pod to be dynamically provisioned. The pod clones some Java source code, builds a WAR file, and causes a second BuildConfig, openshift-jee-sample-docker to run. The second BuildConfig layers the new WAR file into a container image.

Sample BuildConfig that uses the Jenkins Kubernetes plug-in

kind: List
apiVersion: v1
items:
- kind: ImageStream
  apiVersion: image.openshift.io/v1
  metadata:
    name: openshift-jee-sample
- kind: BuildConfig
  apiVersion: build.openshift.io/v1
  metadata:
    name: openshift-jee-sample-docker
  spec:
    strategy:
      type: Docker
    source:
      type: Docker
      dockerfile: |-
        FROM openshift/wildfly-101-centos7:latest
        COPY ROOT.war /wildfly/standalone/deployments/ROOT.war
        CMD $STI_SCRIPTS_PATH/run
      binary:
        asFile: ROOT.war
    output:
      to:
        kind: ImageStreamTag
        name: openshift-jee-sample:latest
- kind: BuildConfig
  apiVersion: build.openshift.io/v1
  metadata:
    name: openshift-jee-sample
  spec:
    strategy:
      type: JenkinsPipeline
      jenkinsPipelineStrategy:
        jenkinsfile: |-
          node("maven") {
            sh "git clone https://github.com/openshift/openshift-jee-sample.git ."
            sh "mvn -B -Popenshift package"
            sh "oc start-build -F openshift-jee-sample-docker --from-file=target/ROOT.war"
          }
    triggers:
    - type: ConfigChange

It is also possible to override the specification of the dynamically created Jenkins agent pod. The following is a modification to the preceding example, which overrides the container memory and specifies an environment variable.

Sample BuildConfig that the Jenkins Kubernetes Plug-in, specifying memory limit and environment variable

kind: BuildConfig
apiVersion: build.openshift.io/v1
metadata:
  name: openshift-jee-sample
spec:
  strategy:
    type: JenkinsPipeline
    jenkinsPipelineStrategy:
      jenkinsfile: |-
        podTemplate(label: "mypod", (1)
                    cloud: "openshift", (2)
                    inheritFrom: "maven", (3)
                    containers: [
            containerTemplate(name: "jnlp", (4)
                              image: "openshift/jenkins-agent-maven-35-centos7:v3.10", (5)
                              resourceRequestMemory: "512Mi", (6)
                              resourceLimitMemory: "512Mi", (7)
                              envVars: [
              envVar(key: "CONTAINER_HEAP_PERCENT", value: "0.25") (8)
            ])
          ]) {
          node("mypod") { (9)
            sh "git clone https://github.com/openshift/openshift-jee-sample.git ."
            sh "mvn -B -Popenshift package"
            sh "oc start-build -F openshift-jee-sample-docker --from-file=target/ROOT.war"
          }
        }
  triggers:
  - type: ConfigChange

By default, the pod is deleted when the build completes. This behavior can be modified with the plug-in or within a pipeline Jenkinsfile.

Upstream Jenkins has more recently introduced a YAML declarative format for defining a podTemplate pipeline DSL in-line with your pipelines. An example of this format, using the sample java-builder pod template that is defined in the OKD Jenkins image:

def nodeLabel = 'java-buidler'
pipeline {
  agent {
    kubernetes {
      cloud 'openshift'
      label nodeLabel
      yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    worker: ${nodeLabel}
spec:
  containers:
  - name: jnlp
    image: image-registry.openshift-image-registry.svc:5000/openshift/jenkins-agent-base:latest
    args: ['\$(JENKINS_SECRET)', '\$(JENKINS_NAME)']
  - name: java
    image: image-registry.openshift-image-registry.svc:5000/openshift/java:latest
    command:
    - cat
    tty: true
"""
    }
  }
  options {
    timeout(time: 20, unit: 'MINUTES')
  }
  stages {
    stage('Build App') {
      steps {
        container("java") {
          sh "mvn --version"
        }
     }
    }
  }
}

Jenkins memory requirements

When deployed by the provided Jenkins Ephemeral or Jenkins Persistent templates, the default memory limit is 1 Gi.

By default, all other process that run in the Jenkins container cannot use more than a total of 512 MiB of memory. If they require more memory, the container halts. It is therefore highly recommended that pipelines run external commands in an agent container wherever possible.

And if Project quotas allow for it, see recommendations from the Jenkins documentation on what a Jenkins master should have from a memory perspective. Those recommendations proscribe to allocate even more memory for the Jenkins master.

It is recommended to specify memory request and limit values on agent containers created by the Jenkins Kubernetes plug-in. Admin users can set default values on a per-agent image basis through the Jenkins configuration. The memory request and limit parameters can also be overridden on a per-container basis.

You can increase the amount of memory available to Jenkins by overriding the MEMORY_LIMIT parameter when instantiating the Jenkins Ephemeral or Jenkins Persistent template.

Additional resources

• See Base image options for more information on the Red Hat Universal Base Images (UBI).