Edge connection with EdgeMesh

In case network issue between cloud and edge side, we integrate EdgeMesh to support DNS visit at any time.

Currently we only support HTTP1.x, more protocols like HTTPS and gRPC coming later.

EdgeMesh is enabled as default.

Limitation

  • Ensure network interface “docker0” exists, which means that EdgeMesh only works for Docker CRI.

Environment Check

Before run examples, please check environment first.

DNS Order

Modify /etc/nsswitch.conf, make sure dns is first order, like below:

  1. $ grep hosts /etc/nsswitch.conf
  2. hosts:          dns file mdns4_minimal [NOTFOUND=return]

IP Forward Setting

Enable ip forward:

  1. $ sudo echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
  2. $ sudo sysctl -p

Then check it:

  1. $ sudo sysctl -p | grep ip_forward
  2. net.ipv4.ip_forward = 1

Usage

Assume we have two edge nodes in ready state, we call them edge node “a” and “b”:

  1. $ kubectl get nodes
  2. NAME          STATUS     ROLES    AGE   VERSION
  3. edge-node-a   Ready      edge     25m   v1.15.3-kubeedge-v1.1.0-beta.0.358+0b7ac7172442b5-dirty
  4. edge-node-b   Ready      edge     25m   v1.15.3-kubeedge-v1.1.0-beta.0.358+0b7ac7172442b5-dirty
  5. master        NotReady   master   8d    v1.15.0

Deploy a sample pod from Cloud Side:

  1. $ kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/deployment.yaml
  2. deployment.apps/nginx-deployment created

Check the pod is up and is running state, as we could see the pod is running on edge node b:

  1. $ kubectl get pods -o wide
  2. NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
  3. nginx-deployment-54bf9847f8-sxk94   1/1     Running   0          14m   172.17.0.2   edge-node-b   <none>           <none>

Check it works:

  1. $ curl 172.17.0.2
  2. <!DOCTYPE html>
  3. <html>
  4. <head>
  5. <title>Welcome to nginx!</title>
  6. <style>
  7.     body {
  8.         width: 35em;
  9.         margin: 0 auto;
  10.         font-family: Tahoma, Verdana, Arial, sans-serif;
  11.     }
  12. </style>
  13. </head>
  14. <body>
  15. <h1>Welcome to nginx!</h1>
  16. <p>If you see this page, the nginx web server is successfully installed and
  17. working. Further configuration is required.</p>
  19. <p>For online documentation and support please refer to
  20. <a href="https://nginx.org/">nginx.org</a>.<br/>
  21. Commercial support is available at
  22. <a href="https://nginx.com/">nginx.com</a>.</p>
  24. <p><em>Thank you for using nginx.</em></p>
  25. </body>
  26. </html>

172.17.0.2 is the IP of deployment and the output may be different since the version of nginx image.

Then create a service for it:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: nginx-svc
  6.   namespace: default
  7. spec:
  8.   clusterIP: None
  9.   selector:
  10.     app: nginx
  11.   ports:
  12.     - name: http-0
  13.       port: 12345
  14.       protocol: TCP
  15.       targetPort: 80
  16. EOF
  • For L4/L7 proxy, specify what protocol a port would use by the port’s “name”. First HTTP port should be named “http-0” and the second one should be called “http-1”, etc.

Check the service and endpoints:

  1. $ kubectl get service
  2. NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
  3. nginx-svc    ClusterIP   None         <none>        12345/TCP   77m
  4. $ kubectl get endpoints
  5. NAME         ENDPOINTS            AGE
  6. nginx-svc    172.17.0.2:80        81m

To request a server, use url like this: <service_name>.<service_namespace>.svc.<cluster>.<local>:<port>

In our case, from edge node a or b, run following command:

  1. $ curl http://nginx-svc.default.svc.cluster.local:12345
  2. <!DOCTYPE html>
  3. <html>
  4. <head>
  5. <title>Welcome to nginx!</title>
  6. <style>
  7.     body {
  8.         width: 35em;
  9.         margin: 0 auto;
  10.         font-family: Tahoma, Verdana, Arial, sans-serif;
  11.     }
  12. </style>
  13. </head>
  14. <body>
  15. <h1>Welcome to nginx!</h1>
  16. <p>If you see this page, the nginx web server is successfully installed and
  17. working. Further configuration is required.</p>
  19. <p>For online documentation and support please refer to
  20. <a href="https://nginx.org/">nginx.org</a>.<br/>
  21. Commercial support is available at
  22. <a href="https://nginx.com/">nginx.com</a>.</p>
  24. <p><em>Thank you for using nginx.</em></p>
  25. </body>
  26. </html>
  • EdgeMesh supports both Host Networking and Container Networking
  • If you ever used EdgeMesh of old version, check your iptables rules. It might affect your test result.

Sample

sample

Model

model

  1. a headless service (a service with selector but ClusterIP is None)
  2. one or more pods’ labels match the headless service’s selector
  3. to request a server, use: <service_name>.<service_namespace>.svc.<cluster>:<port>:
    1. get the service’s name and namespace from domain name
    2. query all the backend pods from MetaManager by service’s namespace and name
    3. LoadBalance returns the real backend containers’ hostIP and hostPort

Flow

flow

  1. client requests to server by server’s domain name
  2. DNS being hijacked to EdgeMesh by iptables rules, then a fake ip returned
  3. request hijacked to EdgeMesh by iptables rules
  4. EdgeMesh resolves request, gets domain name, protocol, request and so on
  5. EdgeMesh load balances:
    1. get the service’s name and namespace from the domain name
    2. query backend pods of the service from MetaManager
    3. choose a backend based on strategy
  6. EdgeMesh transports request to server, wait for server’s response and then sends response back to client

Last updated on Jan 1, 0001