serviceGroup

A management unit for managing application deployment and traffic by regions.

Why is it needed

Problem to solve

  • In edge computing scenarios, multiple edge locations are often managed in the same cluster, and each edge site has one or more computing nodes
  • We hope to run a group of services with business logic relationship in each site, and each site will run the same and complete microservices
  • Due to network restrictions, the access between microservices is not expected and can not be performed in multiple locations

How to solve it

ServiceGroup can easily deploy a group of services in different locations or regions, which belonging to the same cluster, and make the calls between services complete within site or region, avoiding cross regional access of services.

The native k8s needs to plan the affinity of nodes to control the specific location of the pod created by the deployment. When the number of edge locations and the number of services to be deployed are too large, the management and deployment are extremely complex, and even there is only theoretical possibility

In addition, in order to limit the mutual calls between services to the same site, it is necessary to create a dedicated service for each deployment, which brings huge management work and is easy to make mistakes

ServiceGroup is designed for this scenario. The deployment-grid, statefulset-grid and service-grid provided by ServiceGroup can be used to easily deploy and manage edge applications, control service flow, and ensure the number of services in each region and disaster recovery

Key Concepts

Architecture

serviceGroup - 图1

NodeUnit

  • NodeUnit is usually one or more computing resource instances located in the same edge site. It is necessary to ensure that the intranet of nodes in the same NodeUnit is connected
  • Services in ServiceGroup run within a NodeUnit
  • ServiceGroup allows users to set the number of pods for deployment runs in a NodeUnit
  • ServiceGroup keeps calls between services in its own NodeUnit

NodeGroup

  • NodeGroup contains one or more NodeUnits
  • Ensure that the services in ServiceGroup are deployed on each NodeUnit in NodeGroup
  • When nodeunit is added to the cluster, the service in ServiceGroup is automatically deployed to the newly added NodeUnit

ServiceGroup

ServiceGroup contains one or more business services. The applicable scenarios are as follows:

  • Want to package and deploy multiple services
  • Or, the service(s) needs to run in multiple NodeUnits and guarantee the number of pods for each NodeUnit
  • Or, calls between services need to be controlled in the same NodeUnit, and traffic cannot be forwarded to other NodeUnits.

Note: ServiceGroup is an abstract resource. Multiple ServiceGroups can be created in a cluster

Resource types

DeploymentGrid

The format of the DeploymentGrid is similar to that of deployment. The < deployment template > field is the template field of the Kubernetes original deployment, and the more special is the gridUniqKey field, which indicates the key value of the label of the node group.

  1. apiVersion: superedge.io/v1
  2. kind: DeploymentGrid
  3. metadata:
  4.   name:
  5.   namespace:
  6. spec:
  7.   gridUniqKey: <NodeLabel Key>
  8.   <deployment-template>

StatefulSetGrid

The format of the StatefulSetGrid is similar to that of statefulset. The < statefulset template > field is the template field of the Kubernetes original statefulset, and the more special is the gridUniqKey field, which indicates the key value of the label of the node group.

  1. apiVersion: superedge.io/v1
  2. kind: StatefulSetGrid
  3. metadata:
  4.   name:
  5.   namespace:
  6. spec:
  7.   gridUniqKey: <NodeLabel Key>
  8.   <statefulset-template>

ServiceGrid

The format of ServiceGrid is similar to that of service. The < service template > field is the template field of the Kubernetes original service. The special field is the GridUniqKey field, which indicates the key value of the label of the node group.

  1. apiVersion: superedge.io/v1
  2. kind: ServiceGrid
  3. metadata:
  4.   name:
  5.   namespace:
  6. spec:
  7.   gridUniqKey: <NodeLabel Key>
  8.   <service-template>

How to use ServiceGroup

Taking the deployment of echo-service as an example, we hope to deploy echo-service services in multiple node groups. We need to do the following:

Determines the unique identity of the ServiceGroup

This step is logical planning, and no actual operation is required. For example, we use the UniqKey for the ServiceGroup logical tag to be created as: zone.

Group edge nodes

In this step, we need to label the edge nodes with kubectl.

For example, we select node12 and node14 and label them with zone = nodeunit1; node21 and node23 are labeled with zone = nodeunit2.

Note: in the previous step, the key of the label is consistent with the UniqKey of the ServiceGroup. Value is the unique key of the NodeUnit. Nodes with the same value belong to the same NodeUnit

If you want to use more than one ServiceGroup, assign each ServiceGroup a different UniqKey.

Stateless ServiceGroup

Deploy DeploymentGrid

  1. apiVersion: superedge.io/v1
  2. kind: DeploymentGrid
  3. metadata:
  4.   name: deploymentgrid-demo
  5.   namespace: default
  6. spec:
  7.   gridUniqKey: zone
  8.   template:
  9.     replicas: 2
  10.     selector:
  11.       matchLabels:
  12.         appGrid: echo
  13.     strategy: {}
  14.     template:
  15.       metadata:
  16.         creationTimestamp: null
  17.         labels:
  18.           appGrid: echo
  19.       spec:
  20.         containers:
  21.         - image: superedge/echoserver:2.2
  22.           name: echo
  23.           ports:
  24.           - containerPort: 8080
  25.             protocol: TCP
  26.           env:
  27.             - name: NODE_NAME
  28.               valueFrom:
  29.                 fieldRef:
  30.                   fieldPath: spec.nodeName
  31.             - name: POD_NAME
  32.               valueFrom:
  33.                 fieldRef:
  34.                   fieldPath: metadata.name
  35.             - name: POD_NAMESPACE
  36.               valueFrom:
  37.                 fieldRef:
  38.                   fieldPath: metadata.namespace
  39.             - name: POD_IP
  40.               valueFrom:
  41.                 fieldRef:
  42.                   fieldPath: status.podIP
  43.           resources: {}

Deploy ServiceGrid

  1. apiVersion: superedge.io/v1
  2. kind: ServiceGrid
  3. metadata:
  4.   name: servicegrid-demo
  5.   namespace: default
  6. spec:
  7.   gridUniqKey: zone
  8.   template:
  9.     selector:
  10.       appGrid: echo
  11.     ports:
  12.     - protocol: TCP
  13.       port: 80
  14.       targetPort: 8080

Since the gridUniqKey field is set to zone, the key of the label we use when grouping nodes is zone. If there are three NodeUnits, labeled by zone: zone-0, zone: zone-1, zone: zone-2 for them. at this time, each NodeUnit has the deployment of echo-service and the corresponding pod. If a service is accessed through servicename and clusterip in the node, the request will only be sent to the nodes in this group.

  1. [~]## kubectl get dg
  2. NAME                  AGE
  3. deploymentgrid-demo   85s
  5. [~]## kubectl get deploy
  6. NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
  7. deploymentgrid-demo-zone-0   2/2     2            2           85s
  8. deploymentgrid-demo-zone-1   2/2     2            2           85s
  9. deploymentgrid-demo-zone-2   2/2     2            2           85s
  11. [~]## kubectl get svc
  12. NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
  13. kubernetes             ClusterIP   172.19.0.1     <none>        443/TCP   87m
  14. servicegrid-demo-svc   ClusterIP   172.19.0.177   <none>        80/TCP    80s
  16. ## execute on zone-0 nodeunit
  17. [~]## curl 172.19.0.177|grep "node name"
  18.         node name:      node0
  19. ...
  20. ## execute on zone-1 nodeunit
  21. [~]## curl 172.19.0.177|grep "node name"
  22.         node name:      node1
  23. ...
  24. ## execute on zone-2 nodeunit
  25. [~]## curl 172.19.0.177|grep "node name"
  26.         node name:      node2
  27. ...

In addition, if a new NodeUnit that are added to the cluster after the DeploymentGrid and ServiceGrid resources are deployed, the system will automatically create the corresponding deployment for the new NodeUnit.

Stateful ServiceGroup

Deploy StatefulSetGrid

  1. apiVersion: superedge.io/v1
  2. kind: StatefulSetGrid
  3. metadata:
  4.   name: statefulsetgrid-demo
  5.   namespace: default
  6. spec:
  7.   gridUniqKey: zone
  8.   template:
  9.     selector:
  10.       matchLabels:
  11.         appGrid: echo
  12.     serviceName: "servicegrid-demo-svc"
  13.     replicas: 3
  14.     template:
  15.       metadata:
  16.         labels:
  17.           appGrid: echo
  18.       spec:
  19.         terminationGracePeriodSeconds: 10
  20.         containers:
  21.         - image: superedge/echoserver:2.2
  22.           name: echo
  23.           ports:
  24.           - containerPort: 8080
  25.             protocol: TCP
  26.           env:
  27.             - name: NODE_NAME
  28.               valueFrom:
  29.                 fieldRef:
  30.                   fieldPath: spec.nodeName
  31.             - name: POD_NAME
  32.               valueFrom:
  33.                 fieldRef:
  34.                   fieldPath: metadata.name
  35.             - name: POD_NAMESPACE
  36.               valueFrom:
  37.                 fieldRef:
  38.                   fieldPath: metadata.namespace
  39.             - name: POD_IP
  40.               valueFrom:
  41.                 fieldRef:
  42.                   fieldPath: status.podIP
  43.           resources: {}

Note that the serviceName field of statefulset template should be set to service to be created.

Deploy ServiceGrid

  1. apiVersion: superedge.io/v1
  2. kind: ServiceGrid
  3. metadata:
  4.   name: servicegrid-demo
  5.   namespace: default
  6. spec:
  7.   gridUniqKey: zone
  8.   template:
  9.     selector:
  10.       appGrid: echo
  11.     ports:
  12.     - protocol: TCP
  13.       port: 80
  14.       targetPort: 8080

Since the gridUniqKey field is set to zone, the key of the label we use when grouping nodes is zone. If there are three NodeUnits, labeled by zone: zone-0, zone: zone-1, zone: zone-2 for them. at this time, each NodeUnit has the statefulset of echo-service and the corresponding pod. If a service is accessed through servicename and clusterip in the node, the request will only be sent to the nodes in this group.

  1. [~]## kubectl get ssg
  2. NAME                   AGE
  3. statefulsetgrid-demo   21h
  5. [~]## kubectl get statefulset
  6. NAME                          READY   AGE
  7. statefulsetgrid-demo-zone-0   3/3     21h
  8. statefulsetgrid-demo-zone-1   3/3     21h
  9. statefulsetgrid-demo-zone-2   3/3     21h
  11. [~]## kubectl get svc
  12. NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
  13. kubernetes             ClusterIP   192.168.0.1     <none>        443/TCP   22h
  14. servicegrid-demo-svc   ClusterIP   192.168.21.99   <none>        80/TCP    21h
  16. ## execute on zone-0 nodeunit
  17. [~]## curl 192.168.21.99|grep "node name"
  18.         node name:      node0
  19. ...
  20. ## execute on zone-1 nodeunit
  21. [~]## curl 192.168.21.99|grep "node name"
  22.         node name:      node1
  23. ...
  24. ## execute on zone-2 nodeunit
  25. [~]## curl 192.168.21.99|grep "node name"
  26.         node name:      node2
  27. ...

Note that the clusterIP field of service must not be ‘None’ when accessing statefulset by service since it is not currently supported by StatefulSetGrid.

In addition to the normal service, StatefulSetGrid also provides headless service access as below:

serviceGroup - 图2

StatefulSet headless service will be access by domains constructed by {StatefulSetGrid}-{0..N-1}.{ServiceGrid}-svc.ns.svc.cluster.local which corresponds to actual statefulset workload {StatefulSetGrid}-{NodeUnit}-{0..N-1}.{ServiceGrid}-svc.ns.svc.cluster.local of each NodeUnit, aiming to block the difference of NodeUnits.

Each NodeUnit will use the same headless service to access the pod inside its group.(eg: nodes belongs to NodeUnit：zone-1 will have a access to statefulsetgrid-demo-zone-1 statefulset and statefulsetgrid-demo-zone-2 for nodes belongs to NodeUnit：zone-2)

  1. ## execute on zone-0 nodeunit
  2. [~]## curl statefulsetgrid-demo-0.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  3.         pod name:       statefulsetgrid-demo-zone-0-0
  4. [~]## curl statefulsetgrid-demo-1.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  5.         pod name:       statefulsetgrid-demo-zone-0-1
  6. [~]## curl statefulsetgrid-demo-2.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  7.         pod name:       statefulsetgrid-demo-zone-0-2
  8. ...
  9. ## execute on zone-1 nodeunit
  10. [~]## curl statefulsetgrid-demo-0.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  11.         pod name:       statefulsetgrid-demo-zone-1-0
  12. [~]## curl statefulsetgrid-demo-1.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  13.         pod name:       statefulsetgrid-demo-zone-1-1
  14. [~]## curl statefulsetgrid-demo-2.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  15.         pod name:       statefulsetgrid-demo-zone-1-2
  16. ...
  17. ## execute on zone-2 nodeunit
  18. [~]## curl statefulsetgrid-demo-0.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  19.         pod name:       statefulsetgrid-demo-zone-2-0
  20. [~]## curl statefulsetgrid-demo-1.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  21.         pod name:       statefulsetgrid-demo-zone-2-1
  22. [~]## curl statefulsetgrid-demo-2.servicegrid-demo-svc.default.svc.cluster.local|grep "pod name"
  23.         pod name:       statefulsetgrid-demo-zone-2-2
  24. ...

Canary deployment

Both DeploymentGrid and StatefulSetGrid provide build-in support for NodeUnit based canary deployment. User can define multiple workload templates in the templatePool and assign them to different NodeUnits so as to roll out different releases in different NodeUnit.

Configuration

To use canary deployment, additional fields need to be added to the YAML object:

  • spec.templatePool: Defines a template pool for multiple workload templates to be used.
  • spec.templates: Assigns the name of the template, which defined in the templatePool, to be used by each NodeUnit. For NodeUnits that don’t have template assignment, the spec.defaultTemplateName template will be used.
  • spec.defaultTemplateName: (Optional, default to “default”) Defines a default workload template to be used by NodeUnit if not provided in the spec.templates. Setting it to “default” will use the workload template in the spec.template.
  • spec.autoDeleteUnusedTemplate: (Optional, default to false) Setting it to true will delete unused templates in the templatePool. i.e. Unused templates can’t be retrieved from etcd.

Example:

For DeploymentGrid,

  1. apiVersion: superedge.io/v1
  2. kind: DeploymentGrid
  3. metadata:
  4.   name: deploymentgrid-demo
  5.   namespace: default
  6. spec:
  7.   defaultTemplateName: test1 ## (Optional) Default to "default".
  8.   autoDeleteUnusedTemplate: false ## (Optional). Default to false.
  9.   gridUniqKey: zone
  10.   template:
  11.     ...... ## Omit workload spec. If defaultTemplateName is set to default, this template will be used. Otherwise, it will be ignored.
  12.   templatePool: ## Defines workload templates for NodeUnits
  13.     test1:
  14.       replicas: 2
  15.       selector:
  16.         matchLabels:
  17.           appGrid: echo
  18.       strategy: {}
  19.       template:
  20.         metadata:
  21.           creationTimestamp: null
  22.           labels:
  23.             appGrid: echo
  24.         spec:
  25.           containers:
  26.           - image: superedge/echoserver:2.2 ## Old release
  27.             name: echo
  28.             ## Omit container spec
  29.     test2:
  30.       replicas: 3
  31.       selector:
  32.         matchLabels:
  33.           appGrid: echo
  34.       strategy: {}
  35.       template:
  36.         metadata:
  37.           creationTimestamp: null
  38.           labels:
  39.             appGrid: echo
  40.         spec:
  41.           containers:
  42.           - image: superedge/echoserver:2.3  ## New release
  43.             name: echo
  44.             ## Omit container spec
  45.   templates: ## Assigns workload templates to NodeUnits.
  46.     ## <NodeUnit_name>: <template_name_from_templatePool>
  47.     zone1: test1
  48.     zone2: test2

In this example, the NodeGroup

  • Zone1 will use workload template test1 with image version 2.2.
  • Zone2 will use workload template test2 with image version 2.3.
  • Other NodeGroups will use the default template test1, which was specified in the spec.defaultTemplateName.

Refs

Last modified June 15, 2021 : initial commit (974355a)