Cloud-Native Devops Best Practices(1) - Continuous Integration (CI) + OpenKruise Image Pre-download

What is Devops？

DevOps is the merging of the three domains of Development, QA, and Operations. DevOps is an idea, a set of best practices, and a culture. DevOps is an extension of CI/CD, and CI/CD is the core foundation of DevOps. Without CI/CD automation tools and processes, DevOps is meaningless.

ci/cd pipeline

Continuous Integration (CI) + OpenKruise Image Pre-download

Core Concepts

  • Continuous Integration(CI) is a hands-on way to bring integration forward to the early stages of the development cycle, allowing builds, tests and integration of code to happen more often and repeatedly.
  • Image Pre-download is provided by OpenKruise to pull application images to specific Node nodes in advance of application deployment, which in turn can greatly improve the efficiency of application deployment.

Architecture

ci+image predownload

User Story

  • Long-term pre-download common sidecar images, base images, such as: istio envoy, log collection containers.
  • In large-scale scenarios, pre-download business app images to a specific K8s Node to reduce the pressure on the image repository during deployment, mainly for Deployment, StatefulSet and other k8s native resources.
  • OpenKruise CloneSet & Advanced StatefulSet InPlace Update with built-in image pre-download capability, refer to CloneSet documentation.

Note: The OpenKruise image pre-download capability is only available for regular kubelet nodes, and not for virtual kubelet.

Tekton(CI) + Image Pre-download Practice

Requirements

  • Install Kubernetes Cluster, Since v1.0.0 (alpha/beta), OpenKruise requires Kubernetes version >= 1.16.
  • Install Tekton, Reference Official Documents。 Tekton is a Google open source Kubernetes native framework for creating continuous integration and continuous deployment/delivery (CI/CD) systems.
  • Helm installation of OpenKruise, Since v0.9.0, Reference Install OpenKruise。

Build-Test-Docker Push

1. Git Repo: This article provides a helloworld http service demo, It contains Code, Dockerfile, and Unit Test, as follows: git helloworld

2. Tekton Build-Test-DockerPush Task, and need to generate the docker registry secret(for docker push image), as follows:

  1. # docker registry secret, for docker push image
  2. apiVersion: v1
  3. data:
  4.   .dockerconfigjson: xxxxxx
  5. kind: Secret
  6. metadata:
  7.   name: dockersecret
  8. type: kubernetes.io/dockerconfigjson
  9. ---
  10. apiVersion: tekton.dev/v1beta1
  11. kind: Task
  12. metadata:
  13.   labels:
  14.     app: helloworld
  15.   name: helloworld-build-push
  16. spec:
  17.   stepTemplate:
  18.     workingDir: /workspace
  19.   params:
  20.   - name: gitrepositoryurl
  21.     type: string
  22.   - name: branch
  23.     type: string
  24.   - name: short_sha
  25.     type: string
  26.   - name: docker_repo
  27.     type: string
  28.   - name: app_name
  29.     type: string
  30.   steps:
  31.   # git clone
  32.   - name: git-clone-and-checkout
  33.     image: bitnami/git:latest
  34.     command: ["sh", "-ce"]
  35.     args:
  36.     - >
  37.       set -e
  39.       echo $(params.gitrepositoryurl)
  41.       git clone $(params.gitrepositoryurl) ./ && git checkout $(params.branch)
  42.   # unit test
  43.   - name: auto-test
  44.     image: golang:1.16
  45.     command: [ "sh", "-ce" ]
  46.     args:
  47.     - >
  48.       set -e
  50.       cp -R /workspace/$(params.app_name) /go/src/ && cd /go/src/$(params.app_name) && pwd;
  52.       go test
  53.   # docker build & push registry
  54.   - name: push-to-registry
  55.     image: gcr.io/kaniko-project/executor:latest
  56.     args:
  57.     - --dockerfile=Dockerfile
  58.     - --destination=$(params.docker_repo):$(params.branch)-$(params.short_sha)
  59.     - --context=./$(params.app_name)
  60.     - --cache=true
  61.     - --cache-dir=/cache
  62.     - --use-new-run
  63.     volumeMounts:
  64.     - name: kaniko-secret
  65.       mountPath: "/kaniko/.docker"
  66.   volumes:
  67.   # docker push secret
  68.   - name: kaniko-secret
  69.     secret:
  70.       secretName: dockersecret
  71.       items:
  72.       - key: .dockerconfigjson
  73.         path: config.json

Image Pre-download

Kruise CloneSet & Advanced StatefulSet InPlace Update Built-in Image Pre-download

Note: This scenario no longer requires to deploy ImagePullJob CRD

If you have enabled the PreDownloadImageForInPlaceUpdate feature-gate during Kruise installation or upgrade, CloneSet & Advanced StatefulSet controller will automatically pre-download the image you want to update to the nodes of all old Pods. It is quite useful to accelerate the progress of applications upgrade.

  1. # Firstly add openkruise charts repository if you haven't do this.
  2. $ helm repo add openkruise https://openkruise.github.io/charts/
  4. # [Optional]
  5. $ helm repo update
  7. # Install the latest version.
  8. $ helm install kruise openkruise/kruise --set featureGates="PreDownloadImageForInPlaceUpdate=true"
  10. # Those that have been installed need to be upgraded
  11. $ helm upgrade kruise openkruise/kruise --set featureGates="PreDownloadImageForInPlaceUpdate=true"

The parallelism of each new image pre-downloading by CloneSet & Advanced StatefulSet is 1, which means the image is downloaded on nodes one by one. You can change the parallelism using the annotation on CloneSet according to the capability of image registry, for registries with more bandwidth and P2P image downloading ability, a larger parallelism can speed up the pre-download process.

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: CloneSet/StatefulSet
  3. metadata:
  4.   annotations:
  5.     apps.kruise.io/image-predownload-parallelism: "5"

Kubernetes Native Workload, e.g. Deployment, StatefulSet, DaemonSet, Job etc.

1. Configure ImagePullJob CRD in k8s configmap, as follows:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: imagePullJob
  5. data:
  6.   imagepulljob.yaml: |
  7.     apiVersion: apps.kruise.io/v1alpha1
  8.     kind: ImagePullJob
  9.     metadata:
  10.       name: APP_NAME
  11.     spec:
  12.       # pre-download image
  13.       image: APP_IMAGE
  14.       parallelism: 10
  15.       # You can write the names or label selector in the selector field to assign Nodes (only one of them can be set).
  16.       # If no selector is set, the image will be pulled on all Nodes in the cluster.
  17.       selector:
  18.         names:
  19.         - node-1
  20.         - node-2
  21.         matchLabels:
  22.           node-type: xxx
  23.       completionPolicy:
  24.         type: Always
  25.         activeDeadlineSeconds: 1200
  26.         ttlSecondsAfterFinished: 300
  27.       pullPolicy:
  28.         backoffLimit: 3
  29.         timeoutSeconds: 300

2. Image Pre-download ImagePullJob TASK, and store kubeconfig in secret, as follows:

  1. # kubeconfig
  2. apiVersion: v1
  3. data:
  4.   kubeconfig: xxxxxx
  5. kind: Secret
  6. metadata:
  7.   name: kubeconfig
  8. ---
  9. apiVersion: tekton.dev/v1beta1
  10. kind: Task
  11. metadata:
  12.   labels:
  13.     app: helloworld
  14.   name: helloworld-image-predownload
  15. spec:
  16.   params:
  17.   - name: branch
  18.     type: string
  19.   - name: short_sha
  20.     type: string
  21.   - name: docker_repo
  22.     type: string
  23.   - name: app_name
  24.     type: string
  25.   steps:
  26.   - name: image-pre-download
  27.     image: bitnami/kubectl:latest
  28.     command: [ "sh", "-ce" ]
  29.     args:
  30.     - >
  31.       set -e
  33.       echo "pre-download image"
  35.       cat /var/crd/imagepulljob.yaml | sed 's#JOB_NAME#$(params.app_name)-$(params.short_sha)#' | sed 's#APP_IMAGE#$(params.docker_repo):$(params.branch)-$(params.short_sha)#' | kubectl apply --kubeconfig=/var/kube/kubeconfig -f -
  36.     volumeMounts:
  37.     - name: kubeconfig
  38.       mountPath: "/var/kube"
  39.     - name: imagepulljob
  40.       mountPath: "/var/crd"
  41.   volumes:
  42.   - name: kubeconfig
  43.     secret:
  44.       secretName: kubeconfig
  45.   - name: imagepulljob
  46.     configmap:
  47.       name: imagepulljob

Tekton Pipeline

1. configure tekton pipeline, first executing the Build-Test-DockerPush Task, and second Image Pre-download Task, as follows:

  1. apiVersion: tekton.dev/v1beta1
  2. kind: Pipeline
  3. metadata:
  4.   name: helloworld-pipeline
  5. spec:
  6.   params:
  7.   - name: gitrepositoryurl
  8.     type: string
  9.   - name: branch
  10.     type: string
  11.   - name: short_sha
  12.     type: string
  13.   - name: docker_repo
  14.     type: string
  15.   - name: app_name
  16.     type: string
  17.   tasks:
  18.   - name: helloworld-build-push
  19.     taskRef:
  20.       name: helloworld-build-push
  21.     params:
  22.     - name: gitrepositoryurl
  23.       value: $(params.gitrepositoryurl)
  24.     - name: short_sha
  25.       value: $(params.short_sha)
  26.     - name: branch
  27.       value: $(params.branch)
  28.     - name: docker_repo
  29.       value: $(params.docker_repo)
  30.     - name: app_name
  31.       value: $(params.app_name)
  32.   - name: helloworld-image-predownload
  33.     taskRef:
  34.       name: helloworld-image-predownload
  35.     params:
  36.     - name: short_sha
  37.       value: $(params.short_sha)
  38.     - name: branch
  39.       value: $(params.branch)
  40.     - name: docker_repo
  41.       value: $(params.docker_repo)
  42.     - name: app_name
  43.       value: $(params.app_name)
  44.     runAfter:
  45.     - helloworld-build-push

2. Configure PipelineRun CRD, and kubectl apply -f in k8s cluster to run Pipeline, as follows:

  1. apiVersion: tekton.dev/v1beta1
  2. kind: PipelineRun
  3. metadata:
  4.   name: helloworld-pipeline-run-1
  5. spec:
  6.   pipelineRef:
  7.     name: helloworld-pipeline
  8.     params:
  9.     - name: gitrepositoryurl
  10.       value: https://github.com/zmberg/samples.git
  11.     - name: branch
  12.       value: hello_world
  13.     - name: short_sha
  14.       value: d92ae174b
  15.     - name: docker_repo
  16.       value: zhaomingshan/kruise
  17.     - name: app_name
  18.       value: helloworld

3. The execution results can be viewed via the tekton command line tool tkn, as follows:

tekton pipeline

Summary

This article aims to combine the image pre-download capability provided by OpenKruise with CI Pipeline, which can greatly improve the deployment efficiency of users in the application deployment phase and reduce the pressure on image repositories in large-scale deployments. A later article will focus on the CD Pipeline application deployment phase, so stay tuned.