Using Ingress-Nginx for Grayscale Release

In Managing Canary Release of Microservice App based on Istio, KubeSphere implemented grayscale publishing for the Bookinfo microservices sample application based on Istio. Some users have indicated that their project has not yet been on Istio. How do you implement grayscale publishing?

In Ingress-Nginx (v0.21.0), a new Canary feature has been introduced that can be used to configure multiple backend services for gateway portals, as well as to control traffic distribution between multiple backend services using specified annotations. . In v2.0.2, KubeSphere upgraded the Ingress Controller version to 0.24.1 to support Ingress-Nginx-based grayscale publishing.

In the previous article, several application scenarios of grayscale publishing have been introduced in detail. This article will directly introduce and demonstrate the implementation of grayscale publishing based on KubeSphere using Ingress and Ingress Controller.

Note: The sample yaml source files and code used in this article have been uploaded to GitHub and can be cloned for local convenience.

Introduction to Ingress-Nginx Annotation

Based on the Nginx Ingress Controller, KubeSphere implements the project’s gateway as a proxy for external traffic and a reverse proxy for each service in the project. Ingress-Nginx supports the configuration of Ingress Annotations to implement grayscale publishing and testing in different scenarios. It can meet the business scenarios of Canary Publishing, Blue-Green Deployment, and A/B Testing.

Nginx Annotations supports the following four Canary rules:

  • nginx.ingress.kubernetes.io/canary-by-header：Flow segmentation based on Request Header for grayscale publishing and A/B testing. When the Request Header is set to always , the request will be sent to the Canary version all the time. When set to never, the request will not be sent to the Canary entry. For any other Header value, the Header will be ignored and the request is prioritized by comparison with other Canary rules.
  • nginx.ingress.kubernetes.io/canary-by-header-value：The value of the Request Header to match to tell Ingress to route the request to the service specified in Canary Ingress. When the Request Header is set to this value, it will be routed to the Canary entry. This rule allows the user to customize the value of the Request Header and must be used with the previous annotation (ie: canary-by-header).
  • nginx.ingress.kubernetes.io/canary-weight：Service-weighted traffic segmentation for blue-green deployments, weight range 0 - 100 Route requests to services specified in Canary Ingress. A weight of 0 means that the Canary Rules will not send any requests to the services of the Canary portal. A weight of 100 means that all requests will be sent to the Canary portal.
  • nginx.ingress.kubernetes.io/canary-by-cookie：Cookie-based traffic segmentation for grayscale publishing and A/B testing. A cookie that tells Ingress to route requests to the service specified in Canary Ingress. When the value of cookie is set to always , it will be routed to the Canary portal; when set to never , the request will not be sent to the Canary entry; for any other value, the cookie is ignored and the request is prioritized against other Canary rules.

Note: The Canary Rules are sorted in order of priority: canary-by-header - > canary-by-cookie - > canary-weight

The above four annotation rules can be divided into the following two categories:

  • Weight-based Canary rule

Ingress-Nginx for Grayscale Release - 图1

  • Canary rules based on user requests

Ingress-Nginx for Grayscale Release - 图2

Step 1: Create Project and the application of Production version

1.1. Create a business space (spacespace) and a project (namespace) in KubeSphere. You can refer to Getting Started with Multi-tenant Management. A sample project has been created as follows.

Ingress-Nginx for Grayscale Release - 图3

1.2. To create an application easily, create a workload and service in your project by edit yaml , or use the toolbox in the bottom right corner of KubeSphere to open web kubectl and create a Production version of the application with the following command and yaml file and expose it to the cluster access. Create a deployment and service for the Production version as follows.

Ingress-Nginx for Grayscale Release - 图4

  1. $ kubectl appy -f production.yaml -n ingress-demo
  2. deployment.extensions/production created
  3. service/production created

The yaml file used is as follows:

production. yaml

  1. apiVersion: extensions/v1beta1
  2. kind: Deployment
  3. metadata:
  4.   name: production
  5. spec:
  6.   replicas: 1
  7.   selector:
  8.     matchLabels:
  9.       app: production
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: production
  14.     spec:
  15.       containers:
  16.       - name: production
  17.         image: mirrorgooglecontainers/echoserver:1.10
  18.         ports:
  19.         - containerPort: 8080
  20.         env:
  21.           - name: NODE_NAME
  22.             valueFrom:
  23.               fieldRef:
  24.                 fieldPath: spec.nodeName
  25.           - name: POD_NAME
  26.             valueFrom:
  27.               fieldRef:
  28.                 fieldPath: metadata.name
  29.           - name: POD_NAMESPACE
  30.             valueFrom:
  31.               fieldRef:
  32.                 fieldPath: metadata.namespace
  33.           - name: POD_IP
  34.             valueFrom:
  35.               fieldRef:
  36.                 fieldPath: status.podIP
  38. ---
  40. apiVersion: v1
  41. kind: Service
  42. metadata:
  43.   name: production
  44.   labels:
  45.     app: production
  46. spec:
  47.   ports:
  48.   - port: 80
  49.     targetPort: 8080
  50.     protocol: TCP
  51.     name: http
  52.   selector:
  53.     app: production

1.3. Create a Production version of Ingress.

  1. $ kubectl appy -f production.ingress -n ingress-demo
  2. ingress.extensions/production created

The yaml file used is as follows:

production.ingress

  1. apiVersion: extensions/v1beta1
  2. kind: Ingress
  3. metadata:
  4.   name: production
  5.   annotations:
  6.     kubernetes.io/ingress.class: nginx
  7. spec:
  8.   rules:
  9.   - host: kubesphere.io
  10.     http:
  11.       paths:
  12.       - backend:
  13.           serviceName: production
  14.           servicePort: 80

Step 2: Access the application of Production version

2.1. At this point, you can see all the resources under the ingress-demo project under the enterprise space demo-workspace of the KubeSphere UI.

Deployment

Ingress-Nginx for Grayscale Release - 图5

Service

Ingress-Nginx for Grayscale Release - 图6

Ingress

Ingress-Nginx for Grayscale Release - 图7

2.2. Access the Production version of the application to ensure that the current project has opened the gateway, open the gateway under the external network access, the type is NodePort.

Ingress-Nginx for Grayscale Release - 图8

2.3. Access the Production version of the app below.

  1. $ curl --resolve kubesphere.io:30205:192.168.0.88 kubesphere.io:30205 
  2. # Note that adding the --resolve parameter eliminates the need to configure IP and domain name mappings in /etc/hosts locally, otherwise you need to configure the domain name mapping locally, where 192.168.0.88 is the gateway address within the project.
  4. Hostname: production-6b4bb8d58d-7r889
  6. Pod Information:
  7.     node name:    ks-allinone
  8.     pod name:    production-6b4bb8d58d-7r889
  9.     pod namespace:    ingress-demo
  10.     pod IP:    10.233.87.165
  12. Server values:
  13.     server_version=nginx: 1.12.2 - lua: 10010
  15. Request Information:
  16.     client_address=10.233.87.225
  17.     method=GET
  18.     real path=/
  19.     query=
  20.     request_version=1.1
  21.     request_scheme=http
  22.     request_uri=http://kubesphere.io:8080/
  24. Request Headers:
  25.     accept=*/*
  26.     host=kubesphere.io:30205
  27.     user-agent=curl/7.29.0
  28. apiVersion: extensions/v1beta1
  29.     x-forwarded-for=192.168.0.88
  30.     x-forwarded-host=kubesphere.io:30205
  31.     x-forwarded-port=80
  32.     x-forwarded-proto=http
  33.     x-original-uri=/
  34.     x-real-ip=192.168.0.88
  35.     x-request-id=9596df96e994ea05bece2ebbe689a2cc
  36.     x-scheme=http
  38. Request Body:
  39.     -no body in request-

Step 3: Create a Canary version

Refer to the production.yaml file of the above Production version, and then create a Canary version of the application, including a Canary version of deployment and service (for demonstration, just replace the keyword production in the production.yaml deployment and service production to canary. The actual scenario may involve business code changes).

Step 4: Ingress-Nginx Annotation Rules

Based on Weight

A typical application scenario for weight-based traffic segmentation is a Blue-green deployment, which can be achieved by setting the weight to 0 or 100. For example, you can set the Green version to the main part and the Blue version’s entry to Canary. Initially, the weight is set to 0, so traffic is not proxied to the Blue version. Once the new version has been tested and verified successfully, you can set the weight of the Blue version to 100, which means that all traffic goes from Green to Blue.

4.1. Use the following canary.ingress yaml file to create an application routing (Ingress) based on the weighted Canary version.

Note: To enable the grayscale publishing mechanism, first set nginx.ingress.kubernetes.io/canary: "true" to enable Canary. The following Canary version of the Ingress example uses an annotation rule based on weights for traffic segmentation, which will be assigned 30% traffic request to sent to the Canary version.

  1. apiVersion: extensions/v1beta1
  2. kind: Ingress
  3. metadata:
  4.   name: canary
  5.   annotations:
  6.     kubernetes.io/ingress.class: nginx
  7.     nginx.ingress.kubernetes.io/canary: "true"
  8.     nginx.ingress.kubernetes.io/canary-weight: "30"
  9. spec:
  10.   rules:
  11.   - host: kubesphere.io
  12.     http:
  13.       paths:
  14.       - backend:
  15.           serviceName: canary
  16.           servicePort: 80

Ingress-Nginx for Grayscale Release - 图9

4.2. Access the domain name of the app.

Note: After the application’s Canary version is based on the weight (30%) for traffic segmentation, the probability of accessing the Canary version is close to 30%, and the traffic ratio may have a small range of fluctuations.

Ingress-Nginx for Grayscale Release - 图10

Based on Request Header

4.3. The classic application scenes based on Request Header’s traffic segmentation are greyscale release or A/B test scenes. Refer to the screenshot below. Add an annotation of nginx.ingress.kubernetes.io/canary-by-header: canary for Ingress in KubeSphere’s Canary edition. (The annotation value here can be random.) Realize the current Ingress and segregate the traffic based on the Ingress.

Note: The Canary Rules are prioritized by canary-by-header - > canary-by-cookie - > canary-weight, so in the following case, rules for the original canary-weight will be ignored.

Ingress-Nginx for Grayscale Release - 图11

4.4. Add a different Header value to the request and access the domain name of the app again.

Note:

For examples, as mentioned in the opening paragraph, when the Request Header is set to never or always, the request will not or always be sent or always sent to the Canary version; For any other Header value, the Header is ignored and the request is prioritized by comparison with other Canary rules (the second request has a 30% weight as the first priority).

Ingress-Nginx for Grayscale Release - 图12

4.5. You can add an nginx.ingress.kubernetes.io/canary-by-header-value: user-value to the previous annotation (that is, canary-by-header). Used to notify Ingress to route requests to the service specified in Canary Ingress.

Ingress-Nginx for Grayscale Release - 图13

4.6. The application’s domain name is accessed as follows. When the Request Header satisfies this value, all requests are routed to the Canary version (this rule allows the user to customize the value of the Request Header).

Ingress-Nginx for Grayscale Release - 图14

Based on Cookie

Similar to the Request Header-based annotation usage rules. For example, in the A/B 测试 scenario, you need to let the users in Beijing have access to the Canary version. Then when the cookie’s annotation is set to nginx.ingress.kubernetes.io/canary-by-cookie: "users_from_Beijing". Now the background user can check the logged in user request, and set the value of the cookie users_from_Beijing to always if the user access source is from Beijing. For always, this will ensure that users in Beijing only access the Canary version.

Summary

The grayscale release can ensure the stability of the overall system. When the initial grayscale is used, the new version can be tested, found and adjusted to ensure its influence. This article demonstrates and demonstrates the use of Ingress and Ingress Controller for grayscale publishing based on Kubeel, and details the four Annotations of Ingress-Nginx. Users who have not used Istio can also easily implement grayscale publishing with Ingress-Nginx. Released with the canary.

References