K8s

1. Overview

Previously, the deployment environment section had already been set up to make gitlab, jenkins, harbor, k8s, and we’ll write jenkin’s pipline to publish our services fully into k8s through jenkins.

2. Middlewares Deployment

Deploy mysql, redis, es and others outside the k8s, simulate online independent environments (you want to deploy some middleware inside the k8s self-handling, this time focus on how to deploy the microservices developed by go-zero into the k8s cluster). Here I am using the docker-compose-env.yaml of the project and installing all the dependent third-party intermediate environments directly on the srv-data.com (192.168.1.181) server, provided the server is docker, dock-compose.

Login to 192.168.1.181

  1. $ mkdir data && cd data && vim docker-compose.yml
  2. $ docker-compose up -d
  3. $ docker-compose ps #查看确认

3. Independent configuration

Put the configuration of each service independently and uniformly in a git warehouse, so that only one person can be given permission to change the files in this repository directly. When jenkins do cd, you will first pull the code to automatically build the configuration of the corresponding service and see the following pipline.

Why not configure center?

1) Modification of db, redis and so forth requires restart of services, but some configuration does not require reboot of service, the shipping dimension has a record of which is more likely to cause online accidents.

2) Easy to roll back.We found a new version online, and we changed the new version configuration.This is a time when there is a problem with online user feedback. If we build files into a mirror, you can roll back the previous version plus configuration directly using the k8s line command.If the configuration center is used, the code is rolled back, and the previous version will be reconfigured to escort the center back

Independent online repository directory structure is as follows (this structure is related to the writing in pipline)

go-zero official k8s configuration demo address: https://github.com/zeroicro/zero-examples/blob/main/discovery/k8s/client/etc/k8s.yaml

Also reference is made to configuration repository address ： https://github.com/Mikaelemmmm/go-zero-looklook-pro-conf

[NOTE ] 1 Modified Middle,Database, redis etc to change to 192.168.1.181 This machine is used as an intermediate for online environments.

2 another is our service discovery that we are deployed online in k8s, go-zero directly supports k8s service discoveries, so letcd is not required. We need to change target,k8s configuration when we configure zrpc customers.

4. Mirror Repository

Go to harbor to create this project’s mirror repository

View the push command

  1. $ docker push 192.168.1.180:8077/k8scode/REPOSITORY[:TAG]

5. Write the pipline of jenkins

5.1 Configuration parameters

Visit http://192.168.1.1180:8989/ Open jenkins, enter jenkins homepage, click on the left menu to create an Item

We first create a service pipeline

Then click on “General”, select “This project is parameterized”, “Add parameter”, “Choice Parameter”, like the beacon

Then write the following

Save directly.

5.2 Edit pipline

Swipe down to find Pipeline scripts, fill in script content

  1.     pipeline {
  2.     agent any
  3.     parameters {
  4.         gitParameter name: 'branch',
  5.         type: 'PT_BRANCH',
  6.         branchFilter: 'origin/(.*)',
  7.         defaultValue: 'master',
  8.         selectedValue: 'DEFAULT',
  9.         sortMode: 'ASCENDING_SMART',
  10.         description: '选择需要构建的分支'
  11.     }
  13.     stages {
  14.         stage('服务信息')    {
  15.             steps {
  16.                 sh 'echo 分支：$branch'
  17.                 sh 'echo 构建服务类型：${JOB_NAME}-$type'
  18.             }
  19.         }
  22.         stage('拉取代码') {
  23.             steps {
  24.                 checkout([$class: 'GitSCM',
  25.                 branches: [[name: '$branch']],
  26.                 doGenerateSubmoduleConfigurations: false,
  27.                 extensions: [],
  28.                 submoduleCfg: [],
  29.                 userRemoteConfigs: [[credentialsId: 'gitlab-cert', url: 'ssh://git@192.168.1.180:2222/root/k8scode.git']]])
  30.             }
  31.         }
  32.         stage('获取commit_id') {
  33.             steps {
  34.                 echo '获取commit_id'
  35.                 git credentialsId: 'gitlab-cert', url: 'ssh://git@192.168.1.180:2222/root/k8scode.git'
  36.                 script {
  37.                     env.commit_id = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()
  38.                 }
  39.             }
  40.         }
  41.         stage('拉取配置文件') {
  42.                 steps {
  43.                     checkout([$class: 'GitSCM',
  44.                     branches: [[name: '$branch']],
  45.                     doGenerateSubmoduleConfigurations: false,
  46.                     extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'conf']],
  47.                     submoduleCfg: [],
  48.                     userRemoteConfigs: [[credentialsId: 'gitlab-cert', url: 'ssh://git@192.168.1.180:2222/root/k8scode-pro-conf.git']]])
  49.                 }
  50.             }
  52.         stage('goctl版本检测') {
  53.             steps{
  54.                 sh '/usr/local/bin/goctl -v'
  55.             }
  56.         }
  58.         stage('Dockerfile Build') {
  59.             steps{
  60.                     sh 'yes | cp  -rf conf/${JOB_NAME}/${type}/${JOB_NAME}.yaml  app/${JOB_NAME}/cmd/${type}/etc'   //线上配置文件
  61.                     sh 'cd app/${JOB_NAME}/cmd/${type} && /usr/local/bin/goctl docker -go ${JOB_NAME}.go && ls -l'
  62.                     script{
  63.                         env.image = sh(returnStdout: true, script: 'echo ${JOB_NAME}-${type}:${commit_id}').trim()
  64.                     }
  65.                     sh 'echo 镜像名称：${image} && cp app/${JOB_NAME}/cmd/${type}/Dockerfile ./  && ls -l && docker build  -t ${image} .'
  66.             }
  67.         }
  69.         stage('上传到镜像仓库') {
  70.             steps{
  71.                 //docker login 这里要注意，会把账号密码输出到jenkins页面，可以通过port.sh类似方式处理，官网文档有这里我就不详细写了
  72.                 sh 'docker login --username=${docker_username} --password=${docker_pwd} http://${docker_repo}'
  73.                 sh 'docker tag  ${image} ${docker_repo}/k8scode/${image}'
  74.                 sh 'docker push ${docker_repo}/k8scode/${image}'
  75.             }
  76.         }
  78.         stage('部署到k8s') {
  79.             steps{
  80.                 script{
  81.                     env.deployYaml = sh(returnStdout: true, script: 'echo ${JOB_NAME}-${type}-deploy.yaml').trim()
  82.                     env.port=sh(returnStdout: true, script: '/root/port.sh ${JOB_NAME}-${type}').trim()
  83.                 }
  85.                 sh 'echo ${port}'
  87.                 sh 'rm -f ${deployYaml}'
  88.                 sh '/usr/local/bin/goctl kube deploy -secret docker-login -replicas 2 -nodePort 3${port} -requestCpu 200 -requestMem 50 -limitCpu 300 -limitMem 100 -name ${JOB_NAME}-${type} -namespace k8scode -image ${docker_repo}/${image} -o ${deployYaml} -port ${port} -serviceAccount find-endpoints '
  89.                 sh '/usr/local/bin/kubectl apply -f ${deployYaml}'
  90.             }
  91.         }
  93.         stage('Clean') {
  94.             steps{
  95.                 sh 'docker rmi -f ${image}'
  96.                 sh 'docker rmi -f ${docker_repo}/${image}'
  97.                 cleanWs notFailBuild: true
  98.             }
  99.         }
  100.     }
  101.     }
  102.     ```
  104.     :::note is very important
  105.     when building optimization：pipline is using "/usr/local/bin/goctl kube xxxx" we're deploying using k8s yaml without etcds, but this deployment needs to be specified for the built k8s yaml. The idea is to look at the k8s service in this article below go-zero below ：https://mp.weixin.qq.com/s/-WaWJaM_ePEQOf7ExNJe7w
  107.     I have already specified the serviceAccount
  109.     So you need to create find-endpoints on your k8s, this serviceAccount and bind the corresponding permissions. yaml file I am ready. You just need to do it
  111.     Execute command kubectl apply -f auth.yaml, the auth.yaml file as follows:：
  113.     ```yaml
  114.     #创建账号
  115.     apiVersion: v1
  116.     kind: ServiceAccount
  117.     metadata:
  118.     namespace: k8scode
  119.     name: find-endpoints
  121.     ---
  122.     #创建角色对应操作
  123.     apiVersion: rbac.authorization.k8s.io/v1
  124.     kind: ClusterRole
  125.     metadata:
  126.     name: discov-endpoints
  127.     rules:
  128.     - apiGroups: [""]
  129.         resources: ["endpoints"]
  130.         verbs: ["get", "list", "watch"]
  132.     ---
  133.     #给账号绑定角色
  134.     apiVersion: rbac.authorization.k8s.io/v1
  135.     kind: ClusterRoleBinding
  136.     metadata:
  137.     name: find-endpoints-discov-endpoints
  138.     roleRef:
  139.     apiGroup: rbac.authorization.k8s.io
  140.     kind: ClusterRole
  141.     name: discov-endpoints
  142.     subjects:
  143.     - kind: ServiceAccount
  144.         name: find-endpoints
  145.         namespace: k8scode
  146.     ```
  148.     pipline generating k8s yaml files can support serviceAccount without using a template to specify serviceAccount at the time of generation; directly-specify -serviceAcount in pipline and add serviceAccount in the yaml generating k8s
  150.     ```shell
  151.     /usr/local/bin/goctl kube deploy -secret docker-login -replicas 2 -nodePort 3${port} -requestCpu 200 -requestMem 50 -limitCpu 300 -limitMem 100 -name ${JOB_NAME}-${type} -namespace k8scode -image ${docker_repo}/${image} -o ${deployYaml} -port ${port} --serviceAccount find-endpoints
  152.     ```
  154. 2. ${credentialsId}will replace your specific credentials with a string of strings in the "Add Credit" module, so we've filled in gitlab-cert here. If you are not this own replacement,${gitUrl}needs to be replaced with a git repository address for your code, other variables in${xxx}form need no modification, keeping it as it is.
  156. 3. This is a little different from the official document. Since my project folder is different, goctl generated dockerfile files that I adjusted manually because I did not create a dockerfile, when I created the project, placing the dockerfile together in the directory when creating the project, so no goctl was required to build the mirror
  157. :::
  159. ## 6, Configure k8s pull private repository image
  161. k8s, by default, can only pull a public mirror image of a hole in a repository. If a private repository image is pulled out, it is the error of the drill `ErrImagePull` and `ImagePullBackOff`
  163. 1. First Jenkins Publish Machine Login harbor
  165.     ```shell
  166.     $ docker login 192.168.1.180:8077
  167.     $ Username: admin
  168.     $ Password:
  169.     Login Succeeded
  170.     ```
  172. 2. Generate login harbor profile in k8s
  174.     ```shell
  175.     #查看上一步登陆harbor生成的凭证
  176.     $ cat /root/.docker/config.json
  177.     {
  178.         "auths": {
  179.             "192.168.1.180:8077": {
  180.                 "auth": "YWRtaW46SGFyYm9yMTIzNDU="
  181.             }
  182.     }
  183.     ```
  185. 3. Base64 encryption of key files
  187.     ```shell
  188.     $ cat /root/.docker/config.json  | base64 -w 0
  190.     ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuMTgwOjgwNzciOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=
  191.     ```
  193. 4. Create docker-secret.yaml
  195.     ```yml
  196.     apiVersion: v1
  197.     kind: Secret
  198.     metadata:
  199.     name: docker-login
  200.     type: kubernetes.io/dockerconfigjson
  201.     data:
  202.     .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuMTgwOjgwNzciOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=
  203.     ```
  205.     ```shell
  206.     $ kubectl create -f docker-secret.yaml -n k8scode
  208.     secret "docker-login" created
  209.     ```
  211. ## 7. Buildings
  213. We go into the home page, click on the service name "Go to the details page
  215. <Image src={require('../../resource/tutorials/ops/image-20220209201812134.png').default} alt='image-20220209201812134' />
  217. Then see that we have configured an identity service like the shape, click "Build with Parameters", then select rpc, click "Start Build"
  219. <Image src={require('../../resource/tutorials/ops/image-20220209201927466.png').default} alt='image-20220209201927466' />
  221. The first time you build your code will fail. It's what should be initialized and it's good to do again.
  223. Successfully Deployed
  225. <Image src={require('../../resource/tutorials/ops/image-20220211142524599.png').default} alt='image-20220211142524599' />
  227. <Image src={require('../../resource/tutorials/ops/image-20220211142613065.png').default} alt='image-20220211142613065' />
  229. <Image src={require('../../resource/tutorials/ops/image-20220211142729231.png').default} alt='image-20220211142729231' />
  231. In the same way, build identity-api, then configure usercenter services to build usercenter-rpc, build usercenter-api, then configure other services and build. This time we build identity-api, identity-rpc, usercenter-api and usercenter-api.
  233. ## 8. Add gateway
  235. Because our api service will be exposed to nodeport ports through goctl, indexing the nodeport service in k8s in k8scode namespace, and then configuring nodeport in nginx.
  237. This time, we have an independent VM installed nginx in addition to k8s, which exposes the k8s backend api service to nginx via nodeport, then nginx configures this api service in its configuration, so nginx is used as a gateway.
  239. The nginx installation is no longer said here. Remember to have the auth_request module, and you don't have to install it yourself.
  241. nginx configuration
  243. ```conf
  244. server{
  245.     listen 8081;
  246.     access_log /var/log/nginx/k8scode.com_access.log;
  247.     error_log /var/log/nginx//k8scode.com_error.log;
  249.     location ~ /usercenter/ {
  250.        auth_request /auth;
  251.        auth_request_set $user $upstream_http_x_user;
  252.        proxy_set_header x-user $user;
  254.        proxy_set_header Host $http_host;
  255.        proxy_set_header X-Real-IP $remote_addr;
  256.        proxy_set_header REMOTE-HOST $remote_addr;
  257.        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  258.        proxy_pass http://192.168.1.182:31002;
  259.    }
  261.    location ~ /travel/ {
  262.        auth_request /auth;
  263.        auth_request_set $user $upstream_http_x_user;
  264.        proxy_set_header x-user $user;
  266.        proxy_set_header Host $http_host;
  267.        proxy_set_header X-Real-IP $remote_addr;
  268.        proxy_set_header REMOTE-HOST $remote_addr;
  269.        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  270.        proxy_pass http://192.168.1.182:31003;
  271.    }
  274.    ......
  275. }

If you are on the line, you should configure multiple nginx to remain highly available, there will be a slb before nginx and your domain names including the https’ configurations should be parsed to slb, firewalls and so on before slb.

9. Final remarks

The whole series will come to an end and the overall architecture should be shown, as the first chapter shows, and it is the hope of this series to help you.