Setup KubeEdge from sourcecode

Abstract

KubeEdge is composed of cloud and edge parts. It is built upon Kubernetes and provides core infrastructure support for networking, application deployment and metadata synchronization between cloud and edge. So if we want to setup kubeedge, we need to setup kubernetes cluster, cloud side and edge side.

  • on cloud side, we need to install docker, kubernetes cluster and cloudcore.
  • on edge side, we need to install docker, mqtt and edgecore.

Prerequisites

Cloud side

Edge side

Note: Do not install kubelet and kube-proxy on edge side

Run KubeEdge

Setup cloud side

Clone KubeEdge

  1. git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
  2. cd $GOPATH/src/github.com/kubeedge/kubeedge

Generate Certificates

RootCA certificate and a cert/key pair is required to have a setup for KubeEdge. Same cert/key pair can be used in both cloud and edge.

  1. $GOPATH/src/github.com/kubeedge/kubeedge/build/tools/certgen.sh genCertAndKey edge

The cert/key will be generated in the /etc/kubeedge/ca and /etc/kubeedge/certs respectively, so this command should be run with root or users who have access to those directories. We need to copy these files to the corresponding edge side server directory.

Run as a binary

  • Firstly, make sure gcc is already installed on your host. You can verify it via:

    1. gcc --version

  • Build cloudcore

    1. cd $GOPATH/src/github.com/kubeedge/kubeedge/
    2. make all WHAT=cloudcore

  • Create DeviceModel and Device CRDs.

    1. cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/devices
    2. kubectl create -f devices_v1alpha1_devicemodel.yaml
    3. kubectl create -f devices_v1alpha1_device.yaml

  • Create ClusterObjectSync and ObjectSync CRDs which used in reliable message delivery.

    1. cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/reliablesyncs
    2. kubectl create -f cluster_objectsync_v1alpha1.yaml
    3. kubectl create -f objectsync_v1alpha1.yaml

  • Copy cloudcore binary

    1. cd $GOPATH/src/github.com/kubeedge/kubeedge/cloud
    2. mkdir -p ~/cmd
    3. cp cloudcore ~/cmd/

    Note ~/cmd/ dir is an example, in the following examples we continue to use ~/cmd/ as the binary startup directory. You can move cloudcore or edgecore binary to anywhere.

  • Create and set cloudcore config file

    1. # the default configration file path is '/etc/kubeedge/config/cloudcore.yaml'
    2. # also you can specify it anywhere with '--config'
    3. mkdir -p /etc/kubeedge/config/ 
    5. # create a minimal configuration with command `~/cmd/cloudcore --minconfig`
    6. # or a full configuration with command `~/cmd/cloudcore --defaultconfig`
    7. ~/cmd/cloudcore --minconfig > /etc/kubeedge/config/cloudcore.yaml 
    8. vim /etc/kubeedge/config/cloudcore.yaml

    verify the configurations before running cloudcore

    1. apiVersion: cloudcore.config.kubeedge.io/v1alpha1
    2. kind: CloudCore
    3. kubeAPIConfig:
    4.   kubeConfig: /root/.kube/config #Enter absolute path to kubeconfig file to enable https connection to k8s apiserver,if master and kubeconfig are both set, master will override any value in kubeconfig.
    5.   master: "" # kube-apiserver address (such as:http://localhost:8080)
    6. modules:
    7.   cloudhub:
    8.     nodeLimit: 10
    9.     tlsCAFile: /etc/kubeedge/ca/rootCA.crt
    10.     tlsCertFile: /etc/kubeedge/certs/edge.crt
    11.     tlsPrivateKeyFile: /etc/kubeedge/certs/edge.key
    12.     unixsocket:
    13.       address: unix:///var/lib/kubeedge/kubeedge.sock # unix domain socket address
    14.       enable: true # enable unix domain socket protocol
    15.     websocket:
    16.       address: 0.0.0.0
    17.       enable: true # enable websocket protocol
    18.       port: 10000 # open port for websocket server

    cloudcore use https connection to Kubernetes apiserver as default, so you should make sure the kubeAPIConfig.kubeConfig exist, but if master and kubeConfig are both set, master will override any value in kubeconfig. Check whether the cert files for modules.cloudhub.tlsCAFile, modules.cloudhub.tlsCertFile,modules.cloudhub.tlsPrivateKeyFile exists.

  • Run cloudcore

    1. cd ~/cmd/
    2. nohup ./cloudcore &

  • Run cloudcore with systemd

    It is also possible to start the cloudcore with systemd. If you want, you could use the example systemd-unit-file. The following command will show you how to setup this:

    1. sudo ln build/tools/cloudcore.service /etc/systemd/system/cloudcore.service
    2. sudo systemctl daemon-reload
    3. sudo systemctl start cloudcore

    Note: Please fix ExecStart path in cloudcore.service. Do NOT use relative path, use absoulte path instead.

    If you also want also an autostart, you have to execute this, too:

    1. sudo systemctl enable cloudcore

  • (Optional)Run admission, this feature is still being evaluated. please read the docs in install the admission webhook

Deploy the edge node

Edge node can be registered automatically. But if you want to deploy edge node manually, here is an example.

Setup edge side

  • Transfer certificate files from cloud side to edge node, because edgecore use these certificate files to connection cloudcore

Clone KubeEdge

  1. git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
  2. cd $GOPATH/src/github.com/kubeedge/kubeedge

Run Edge

Configuring MQTT mode

The Edge part of KubeEdge uses MQTT for communication between deviceTwin and devices. KubeEdge supports 3 MQTT modes:

  1. internalMqttMode: internal mqtt broker is enabled.
  2. bothMqttMode: internal as well as external broker are enabled.
  3. externalMqttMode: only external broker is enabled.

To use KubeEdge in double mqtt or external mode, you need to make sure that mosquitto or emqx edge is installed on the edge node as an MQTT Broker.

Run as a binary

  • Build Edge

    1. cd $GOPATH/src/github.com/kubeedge/kubeedge
    2. make all WHAT=edgecore

    KubeEdge can also be cross compiled to run on ARM based processors. Please follow the instructions given below or click Cross Compilation for detailed instructions.

    1. cd $GOPATH/src/github.com/kubeedge/kubeedge/edge
    2. make edge_cross_build

    KubeEdge can also be compiled with a small binary size. Please follow the below steps to build a binary of lesser size:

    1. apt-get install upx-ucl
    2. cd $GOPATH/src/github.com/kubeedge/kubeedge/edge
    3. make edge_small_build

    Note: If you are using the smaller version of the binary, it is compressed using upx, therefore the possible side effects of using upx compressed binaries like more RAM usage, lower performance, whole code of program being loaded instead of it being on-demand, not allowing sharing of memory which may cause the code to be loaded to memory more than once etc. are applicable here as well.

  • Copy edgecore binary

    1. cd $GOPATH/src/github.com/kubeedge/kubeedge/edge
    2. mkdir -p ~/cmd
    3. cp edgecore ~/cmd/

    Note: ~/cmd/ dir is also an example as well as cloudcore

  • Create and set edgecore config file

    1. # the default configration file path is '/etc/kubeedge/config/edgecore.yaml'
    2. # also you can specify it anywhere with '--config'
    3. mkdir -p /etc/kubeedge/config/ 
    5. # create a minimal configuration with command `~/cmd/edgecore --minconfig`
    6. # or a full configuration with command `~/cmd/edgecore --defaultconfig`
    7. ~/cmd/edgecore --minconfig > /etc/kubeedge/config/edgecore.yaml 
    8. vim /etc/kubeedge/config/edgecore.yaml

    verify the configurations before running edgecore

    1. apiVersion: edgecore.config.kubeedge.io/v1alpha1
    2. database:
    3.   dataSource: /var/lib/kubeedge/edgecore.db
    4. kind: EdgeCore
    5. modules:
    6.   edged:
    7.     cgroupDriver: cgroupfs
    8.     clusterDNS: ""
    9.     clusterDomain: ""
    10.     devicePluginEnabled: false
    11.     dockerAddress: unix:///var/run/docker.sock
    12.     gpuPluginEnabled: false
    13.     hostnameOverride: $your_hostname
    14.     interfaceName: eth0
    15.     nodeIP: $your_ip_address
    16.     podSandboxImage: kubeedge/pause:3.1  # kubeedge/pause:3.1 for x86 arch , kubeedge/pause-arm:3.1 for arm arch, kubeedge/pause-arm64 for arm64 arch
    17.     remoteImageEndpoint: unix:///var/run/dockershim.sock
    18.     remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
    19.     runtimeType: docker
    20.   edgehub:
    21.     heartbeat: 15  # second
    22.     tlsCaFile: /etc/kubeedge/ca/rootCA.crt
    23.     tlsCertFile: /etc/kubeedge/certs/edge.crt
    24.     tlsPrivateKeyFile: /etc/kubeedge/certs/edge.key
    25.     websocket:
    26.       enable: true
    27.       handshakeTimeout: 30  # second
    28.       readDeadline: 15  # second
    29.       server: 127.0.0.1:10000  # cloudcore address
    30.       writeDeadline: 15  # second
    31.   eventbus:
    32.     mqttMode: 2  # 0: internal mqtt broker enable only. 1: internal and external mqtt broker enable. 2: external mqtt broker
    33.     mqttQOS: 0  # 0: QOSAtMostOnce, 1: QOSAtLeastOnce, 2: QOSExactlyOnce.
    34.     mqttRetain: false  # if the flag set true, server will store the message and can be delivered to future subscribers.
    35.     mqttServerExternal: tcp://127.0.0.1:1883  # external mqtt broker url.
    36.     mqttServerInternal: tcp://127.0.0.1:1884  # internal mqtt broker url.
    • Check modules.edged.podSandboxImage
      • kubeedge/pause-arm:3.1 for arm arch
      • kubeedge/pause-arm64:3.1 for arm64 arch
      • kubeedge/pause:3.1 for x86 arch
    • Check whether the cert files for modules.edgehub.tlsCaFile and modules.edgehub.tlsCertFile and modules.edgehub.tlsPrivateKeyFile exists. If those files not exist, you need to copy them from cloud side.
    • Check modules.edgehub.websocket.server. It should be your cloudcore ip address.

  • Run edgecore

    1. # run mosquitto
    2. mosquitto -d -p 1883
    3. # or run emqx edge
    4. # emqx start
    6. cd ~/cmd
    7. ./edgecore
    8. # or
    9. nohup ./edgecore > edgecore.log 2>&1 &

    Note: Please run edgecore using the users who have root permission.

  • Run edgecore with systemd

    It is also possible to start the edgecore with systemd. If you want, you could use the example systemd-unit-file. The following command will show you how to setup this:

    1. sudo ln build/tools/edgecore.service /etc/systemd/system/edgecore.service
    2. sudo systemctl daemon-reload
    3. sudo systemctl start edgecore

    Note: Please fix ExecStart path in edgecore.service. Do NOT use relative path, use absoulte path instead.

    If you also want also an autostart, you have to execute this, too:

    1. sudo systemctl enable edgecore

Check status

After the Cloud and Edge parts have started, you can use below command to check the edge node status.

  1. kubectl get nodes

Please make sure the status of edge node you created is ready.

Deploy Application on cloud side

Try out a sample application deployment by following below steps.

  1. kubectl apply -f $GOPATH/src/github.com/kubeedge/kubeedge/build/deployment.yaml

Note: Currently, for applications running on edge nodes, we don’t support kubectl logs and kubectl exec commands(will support in future release), support pod to pod communication running on edge nodes in same subnet using edgemesh.

Then you can use below command to check if the application is normally running.

  1. kubectl get pods

Run Tests

Run Edge Unit Tests

  1. make edge_test

To run unit tests of a package individually.

  1. export GOARCHAIUS_CONFIG_PATH=$GOPATH/src/github.com/kubeedge/kubeedge/edge
  2. cd <path to package to be tested>
  3. go test -v

Run Edge Integration Tests

  1. make edge_integration_test

Details and use cases of integration test framework

Please find the link to use cases of intergration test framework for KubeEdge.