EdgeSite: Standalone Cluster at edge

Abstract

In Edge computing, there are scenarios where customers would like to have a whole cluster installed at edge location. As a result, admins/users can leverage the local control plane to implement management functionalities and take advantages of all edge computing’s benefits.

EdgeSite helps running lightweight clusters at edge.

Motivation

There are scenarios user need to run a standalone Kubernetes cluster at edge to get full control and improve the offline scheduling capability. There are two scenarios user need to do that:

  • The edge cluster is in CDN instead of the user’s site

    The CDN sites usually be large around the world and the network connectivity and quality cannot be guaranteed. Another factor is that the application deployed in CDN edge do not need to interact with center usually. For those deploy edge cluster in CDN resources, they need to make sure the cluster is workable without the connection with central cloud not only for the deployed applicatons but also the schedule capabilities. So that the CDN edge is manageable regardless the connection to one center.

  • User need to deploy an edge environment with limited resources and offline running for most of the time

    In some IOT scenarios, user need to deploy a full control edge environment and running offline.

For these use cases, a standalone, full controlled, light weight Edge cluster is required. By integrating KubeEdge and standard Kubernetes, this EdgeSite enables customers to run an efficient kubernetes cluster for Edge/IOT computing.

Assumptions

Here we assume a cluster is deployed at edge location including the management control plane. For the management control plane to manage some scale of edge worker nodes, the hosting master node needs to have sufficient resources.

The assumptions are

  1. EdgeSite cluster master node is of no less than 2 CPUs and no less than 1GB memory
  2. If high availability is required, 2-3 master nodes are needed at different edge locations
  3. The same Kubernetes security (authN and authZ) mechanisms are used to ensure the secure handshake between master and worker nodes
  4. The same K8s HA mechanism is to be used to enable HA

Architecture Design

../_images/EdgeSite_arch.PNGEdgeSite Architecture

Advantages

With the integration, the following can be enabled

  1. Full control of Kubernetes cluster at edge
  2. Light weight control plane and agent
  3. Edge worker node autonomy in case of network disconnection/reconnection
  4. All benefits of edge computing including latency, data locality, etc.

Getting Started

Setup

../_images/EdgeSite_Setup.PNGEdgeSite Setup

Steps for K8S (API server) Cluster

  • Install docker

  • Install kubeadm/kubectl

  • Creating cluster with kubeadm

  • KubeEdge supports https connection to Kubernetes apiserver.

    Enter the path to kubeconfig file in controller.yaml

    1. controller:
    2.   kube:
    3.     ...
    4.     kubeconfig: "path_to_kubeconfig_file" #Enter path to kubeconfig file to enable https connection to k8s apiserver

  • (Optional) KubeEdge also supports insecure http connection to Kubernetes apiserver for testing, debugging cases. Please follow below steps to enable http port in Kubernetes apiserver.

    1. vi /etc/kubernetes/manifests/kube-apiserver.yaml
    2. # Add the following flags in spec: containers: -command section
    3. - --insecure-port=8080
    4. - --insecure-bind-address=0.0.0.0

    Enter the master address in controller.yaml

    1. controller:
    2.   kube:
    3.     ...
    4.     master: "http://127.0.0.1:8080" #Note if master and kubeconfig are both set, master will override any value in kubeconfig.

Steps for EdgeSite

Getting EdgeSite Binary

Using Source code

  • Clone KubeEdge (EdgeSite) code

    1. git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge

  • Build EdgeSite

    1. cd $GOPATH/src/github.com/kubeedge/kubeedge/edgesite
    2. make
Download Release packages

TBA

Configuring EdgeSite

Modify edgeSite.yaml configuration file, with the IP address of K8S API server

  • Configure K8S (API Server)

    Replace localhost at controller.kube.master with the IP address

    1. controller:
    2.   kube:
    3.     master: http://localhost:8080
    4.     ...

  • Add EdgeSite (Worker) Node ID/name

    Replace edge-node with an unique edge id/name in below fields :

    • controller.kube.node-id
    • controller.edged.hostname-override
    1. controller:
    2.   kube:
    3.     ...
    4.     node-id: edge-node
    5.     node-name: edge-node
    6.     ...
    7.   edged:
    8.     ...
    9.     hostname-override: edge-node
    10.     ...

  • Configure MQTT (Optional)

    The Edge part of KubeEdge uses MQTT for communication between deviceTwin and devices. KubeEdge supports 3 MQTT modes:

    1. internalMqttMode: internal mqtt broker is enabled. (Default)
    2. bothMqttMode: internal as well as external broker are enabled.
    3. externalMqttMode: only external broker is enabled.

    Use mode field in edgeSite.yaml to select the desired mode.

    1. mqtt:
    2.   ...
    3.   mode: 0 # 0: internal mqtt broker enable only. 1: internal and external mqtt broker enable. 2: external mqtt broker enable only.
    4.   ...

    To use KubeEdge in double mqtt or external mode, you need to make sure that mosquitto or emqx edge is installed on the edge node as an MQTT Broker.

Run EdgeSite

  1.   # run edgesite
  2.   # `conf/` should be in the same directory as the cloned KubeEdge repository
  3.   # verify the configurations before running edgesite
  4.   ./edgesite
  5.   # or
  6.   nohup ./edgesite > edgesite.log 2>&1 &

Note: Please run edgesite using the users who have root permission.

Deploy EdgeSite (Worker) Node to K8S Cluster

We have provided a sample node.json to add a node in kubernetes. Please make sure edgesite (worker) node is added to k8s api-server. Run below steps:

  • Modify node.json

    Replace edge-node in node.json file, to the id/name of the edgesite node. ID/Name should be same as used before while updating edgesite.yaml

    1.   {
    2.     "metadata": {
    3.       "name": "edge-node",
    4.     }
    5.   }

  • Add node in K8S API server

    In the console execute the below command

    1.   kubectl apply -f $GOPATH/src/github.com/kubeedge/kubeedge/build/node.json

  • Check node status

    Below command to check the edgesite node status.

    1.   kubectl get nodes
    3.   NAME         STATUS     ROLES    AGE     VERSION
    4.   testing123   Ready      <none>   6s      0.3.0-beta.0

    Observe the edgesite node is in Ready state

Deploy Application

Try out a sample application deployment by following below steps.

  1. kubectl apply -f $GOPATH/src/github.com/kubeedge/kubeedge/build/deployment.yaml

Note: Currently, for edgesite node, we must use hostPort in the Pod container spec so that the pod comes up normally, or the pod will be always in ContainerCreating status. The hostPort must be equal to containerPort and can not be 0.

Then you can use below command to check if the application is normally running.

  1.   kubectl get pods