为 KubeEdge 创建一个 crun 示例

1. 云端设置（KubeEdge 主节点）

安装 Go

  1. $ wget https://golang.org/dl/go1.17.3.linux-amd64.tar.gz
  2. $ tar xzvf go1.17.3.linux-amd64.tar.gz
  4. $ export PATH=/home/${user}/go/bin:$PATH
  5. $ go version
  6. go version go1.17.3 linux/amd64

安装 CRI-O

请参阅 CRI-O Installation Instructions.

  1. # 创建 .conf 文件以在启动时加载模块
  2. $ cat <<EOF | sudo tee /etc/modules-load.d/crio.conf
  3. overlay
  4. br_netfilter
  5. EOF
  7. $ sudo modprobe overlay
  8. $ sudo modprobe br_netfilter
  10. # 设置所需的 sysctl 参数，这些参数在重启后仍然存在。
  11. $ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
  12. net.bridge.bridge-nf-call-iptables  = 1
  13. net.ipv4.ip_forward                 = 1
  14. net.bridge.bridge-nf-call-ip6tables = 1
  15. EOF
  17. $ sudo sysctl --system
  18. $ export OS="xUbuntu_20.04"
  19. $ export VERSION="1.21"
  20. $ cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
  21. deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
  22. EOF
  23. $ cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
  24. deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
  25. EOF
  27. $ curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
  28. $ curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -
  30. $ sudo apt-get update
  31. $ sudo apt-get install cri-o cri-o-runc
  33. $ sudo systemctl daemon-reload
  34. $ sudo systemctl enable crio --now
  35. $ sudo systemctl status cri-o

输出：

  1. $ sudo systemctl status cri-o
  2. ● crio.service - Container Runtime Interface for OCI (CRI-O)
  3.      Loaded: loaded (/lib/systemd/system/crio.service; enabled; vendor preset: enabled)
  4.      Active: active (running) since Mon 2021-12-06 13:46:29 UTC; 16h ago
  5.        Docs: https://github.com/cri-o/cri-o
  6.    Main PID: 6868 (crio)
  7.       Tasks: 14
  8.      Memory: 133.2M
  9.      CGroup: /system.slice/crio.service
  10.              └─6868 /usr/bin/crio
  12. Dec 07 06:04:13 master crio[6868]: time="2021-12-07 06:04:13.694226800Z" level=info msg="Checking image status: k8s.gcr.io/pause:3.4.1" id=1dbb722e-f031-410c-9f45-5d4b5760163e name=/runtime.v1alpha2.ImageServic>
  13. Dec 07 06:04:13 master crio[6868]: time="2021-12-07 06:04:13.695739507Z" level=info msg="Image status: &{0xc00047fdc0 map[]}" id=1dbb722e-f031-410c-9f45-5d4b5760163e name=/runtime.v1alpha2.ImageService/ImageSta>
  14. Dec 07 06:09:13 master crio[6868]: time="2021-12-07 06:09:13.698823984Z" level=info msg="Checking image status: k8s.gcr.io/pause:3.4.1" id=661b754b-48a4-401b-a03f-7f7a553c7eb6 name=/runtime.v1alpha2.ImageServic>
  15. Dec 07 06:09:13 master crio[6868]: time="2021-12-07 06:09:13.703259157Z" level=info msg="Image status: &{0xc0004d98f0 map[]}" id=661b754b-48a4-401b-a03f-7f7a553c7eb6 name=/runtime.v1alpha2.ImageService/ImageSta>
  16. Dec 07 06:14:13 master crio[6868]: time="2021-12-07 06:14:13.707778419Z" level=info msg="Checking image status: k8s.gcr.io/pause:3.4.1" id=8c7e4d36-871a-452e-ab55-707053604077 name=/runtime.v1alpha2.ImageServic>
  17. Dec 07 06:14:13 master crio[6868]: time="2021-12-07 06:14:13.709379469Z" level=info msg="Image status: &{0xc000035030 map[]}" id=8c7e4d36-871a-452e-ab55-707053604077 name=/runtime.v1alpha2.ImageService/ImageSta>
  18. Dec 07 06:19:13 master crio[6868]: time="2021-12-07 06:19:13.713158978Z" level=info msg="Checking image status: k8s.gcr.io/pause:3.4.1" id=827b6315-f145-4f76-b8da-31653d5892a2 name=/runtime.v1alpha2.ImageServic>
  19. Dec 07 06:19:13 master crio[6868]: time="2021-12-07 06:19:13.714030148Z" level=info msg="Image status: &{0xc000162bd0 map[]}" id=827b6315-f145-4f76-b8da-31653d5892a2 name=/runtime.v1alpha2.ImageService/ImageSta>
  20. Dec 07 06:24:13 master crio[6868]: time="2021-12-07 06:24:13.716746612Z" level=info msg="Checking image status: k8s.gcr.io/pause:3.4.1" id=1d53a917-4d98-4723-9ea8-a2951a472cff name=/runtime.v1alpha2.ImageServic>
  21. Dec 07 06:24:13 master crio[6868]: time="2021-12-07 06:24:13.717381882Z" level=info msg="Image status: &{0xc00042ce00 map[]}" id=1d53a917-4d98-4723-9ea8-a2951a472cff name=/runtime.v1alpha2.ImageService/ImageSta>

使用 K8s Kubeadm 安装和创建集群

请参阅 使用 kubeadm 创建集群.

安装 K8s

  1. sudo apt-get update
  2. sudo apt-get install -y apt-transport-https curl
  3. echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
  5. sudo apt update
  6. K_VER="1.21.0-00"
  7. sudo apt install -y kubelet=${K_VER} kubectl=${K_VER} kubeadm=${K_VER}
  8. sudo apt-mark hold kubelet kubeadm kubectl

使用 kubeadm 创建集群

  1. # kubernetes 调度程序需要进行此设置。
  2. $ sudo swapoff -a
  3. $ sudo vim /etc/fstab
  4. mark contain swapfile of row
  6. $ cat /etc/cni/net.d/100-crio-bridge.conf
  7. {
  8.     "cniVersion": "0.3.1",
  9.     "name": "crio",
  10.     "type": "bridge",
  11.     "bridge": "cni0",
  12.     "isGateway": true,
  13.     "ipMasq": true,
  14.     "hairpinMode": true,
  15.     "ipam": {
  16.         "type": "host-local",
  17.         "routes": [
  18.             { "dst": "0.0.0.0/0" },
  19.             { "dst": "1100:200::1/24" }
  20.         ],
  21.         "ranges": [
  22.             [{ "subnet": "10.85.0.0/16" }],
  23.             [{ "subnet": "1100:200::/24" }]
  24.         ]
  25.     }
  26. }
  27. $ export CIDR=10.85.0.0/16
  28. $ sudo kubeadm init --apiserver-advertise-address=192.168.122.160 --pod-network-cidr=$CIDR --cri-socket=/var/run/crio/crio.sock
  30. $ mkdir -p $HOME/.kube
  31. $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  32. $ sudo chown $(id -u):$(id -g) $HOME/.kube/config

输出：

  1. Your Kubernetes control-plane has initialized successfully!
  3. To start using your cluster, you need to run the following as a regular user:
  5.   mkdir -p $HOME/.kube
  6.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  7.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  9. You should now deploy a Pod network to the cluster.
  10. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  11.   /docs/concepts/cluster-administration/addons/
  13. You can now join any number of machines by running the following on each node
  14. as root:
  16.   kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

要让 kubectl 为你的非 root 用户工作，请运行这些命令，这些命令也是 kubeadm init 输出的一部分：

  1. mkdir -p $HOME/.kube
  2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3. sudo chown $(id -u):$(id -g) $HOME/.kube/config

设置 KubeEdge 主节点

请参阅 使用 Keadm 进行部署。

重要提醒:

  1. kubeconfig 或 master 中的至少一项必须配置正确，才能用于验证 k8s 集群的版本等信息。
  2. 请确保边缘节点可以使用云节点的本地 IP 连接云节点，或者你需要使用 --advertise-address 标志指定云节点的公共 IP。
  3. --advertise-address（仅 1.3 版本起有效）为云端公开的地址（将添加到 CloudCore 证书的 SAN 中），默认值为本地IP。
  1. wget https://github.com/kubeedge/kubeedge/releases/download/v1.8.0/keadm-v1.8.0-linux-amd64.tar.gz
  2. tar xzvf keadm-v1.8.0-linux-amd64.tar.gz
  3. cd keadm-v1.8.0-linux-amd64/keadm/
  4. sudo ./keadm init --advertise-address=192.168.122.160 --kube-config=/home/${user}/.kube/config

输出：

  1. Kubernetes version verification passed, KubeEdge installation will start...
  2. ...
  3. KubeEdge cloudcore is running, For logs visit:  /var/log/kubeedge/cloudcore.log

2. 设置边缘端（KubeEdge 工作节点）

你可以使用 CRI-O install.sh 脚本 crun 在 Ubuntu 20.04 上安装 CRI-O。

  1. wget -qO- https://raw.githubusercontent.com/second-state/wasmedge-containers-examples/main/crio/install.sh | bash

在工作节点安装 Go

  1. $ wget https://golang.org/dl/go1.17.3.linux-amd64.tar.gz
  2. $ tar xzvf go1.17.3.linux-amd64.tar.gz
  4. $ export PATH=/home/${user}/go/bin:$PATH
  5. $ go version
  6. go version go1.17.3 linux/amd64

从云端获取令牌

在云端运行 keadm gettoken 会返回 token ，在加入边缘节点时使用。

  1. $ sudo ./keadm gettoken --kube-config=/home/${user}/.kube/config
  2. 27a37ef16159f7d3be8fae95d588b79b3adaaf92727b72659eb89758c66ffda2.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTAyMTYwNzd9.JBj8LLYWXwbbvHKffJBpPd5CyxqapRQYDIXtFZErgYE

下载 Kubeedge 并加入边缘节点

请参阅 使用 CRI 设置不同的容器运行时 和 使用 Keadm 部署.

  1. $ wget https://github.com/kubeedge/kubeedge/releases/download/v1.8.0/keadm-v1.8.0-linux-amd64.tar.gz
  2. $ tar xzvf keadm-v1.8.0-linux-amd64.tar.gz
  3. $ cd keadm-v1.8.0-linux-amd64/keadm/
  5. $ sudo ./keadm join \
  6. --cloudcore-ipport=192.168.122.160:10000 \
  7. --edgenode-name=edge \
  8. --token=b4550d45b773c0480446277eed1358dcd8a02a0c214646a8082d775f9c447d81.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2Mzg4ODUzNzd9.A9WOYJFrgL2swVGnydpb4gMojyvyoNPCXaA4rXGowqU \
  9. --remote-runtime-endpoint=unix:///var/run/crio/crio.sock \
  10. --runtimetype=remote \
  11. --cgroupdriver=systemd

输出：

  1. Host has mosquit+ already installed and running. Hence skipping the installation steps !!!
  2. ...
  3. KubeEdge edgecore is running, For logs visit:  /var/log/kubeedge/edgecore.log

从云端获取边缘节点状态

输出：

  1. kubectl get node
  2. NAME       STATUS    ROLES                  AGE   VERSION
  3. edge       Ready     agent,edge             10s   v1.19.3-kubeedge-v1.8.2
  4. master     Ready     control-plane,master   68m   v1.21.0

3. 启用 kubectl 日志功能

在部署 metrics-server 之前，必须激活 kubectl 日志功能，请参阅这里.

4. 运行一个简单的 WebAssembly 应用

我们可以从 Kubernetes 集群中的 Docker Hub 运行基于 WebAssembly 的镜像。

云端

  1. $ kubectl run -it --restart=Never wasi-demo --image=hydai/wasm-wasi-example:with-wasm-annotation --annotations="module.wasm.image/variant=compat" /wasi_example_main.wasm 50000000
  3. Random number: -1694733782
  4. Random bytes: [6, 226, 176, 126, 136, 114, 90, 2, 216, 17, 241, 217, 143, 189, 123, 197, 17, 60, 49, 37, 71, 69, 67, 108, 66, 39, 105, 9, 6, 72, 232, 238, 102, 5, 148, 243, 249, 183, 52, 228, 54, 176, 63, 249, 216, 217, 46, 74, 88, 204, 130, 191, 182, 19, 118, 193, 77, 35, 189, 6, 139, 68, 163, 214, 231, 100, 138, 246, 185, 47, 37, 49, 3, 7, 176, 97, 68, 124, 20, 235, 145, 166, 142, 159, 114, 163, 186, 46, 161, 144, 191, 211, 69, 19, 179, 241, 8, 207, 8, 112, 80, 170, 33, 51, 251, 33, 105, 0, 178, 175, 129, 225, 112, 126, 102, 219, 106, 77, 242, 104, 198, 238, 193, 247, 23, 47, 22, 29]
  5. Printed from wasi: This is from a main function
  6. This is from a main function
  7. The env vars are as follows.
  8. The args are as follows.
  9. /wasi_example_main.wasm
  10. 50000000
  11. File content is This is in a file

WebAssembly 应用的 pod 成功部署到边缘节点。

  1. $ kubectl describe pod wasi-demo
  3. Name:         wasi-demo
  4. Namespace:    default
  5. Priority:     0
  6. Node:         edge/192.168.122.229
  7. Start Time:   Mon, 06 Dec 2021 15:45:34 +0000
  8. Labels:       run=wasi-demo
  9. Annotations:  module.wasm.image/variant: compat
  10. Status:       Succeeded
  11. IP:           
  12. IPs:          <none>
  13. Containers:
  14.   wasi-demo:
  15.     Container ID:  cri-o://1ae4d0d7f671050331a17e9b61b5436bf97ad35ad0358bef043ab820aed81069
  16.     Image:         hydai/wasm-wasi-example:with-wasm-annotation
  17.     Image ID:      docker.io/hydai/wasm-wasi-example@sha256:525aab8d6ae8a317fd3e83cdac14b7883b92321c7bec72a545edf276bb2100d6
  18.     Port:          <none>
  19.     Host Port:     <none>
  20.     Args:
  21.       /wasi_example_main.wasm
  22.       50000000
  23.     State:          Terminated
  24.       Reason:       Completed
  25.       Exit Code:    0
  26.       Started:      Mon, 06 Dec 2021 15:45:33 +0000
  27.       Finished:     Mon, 06 Dec 2021 15:45:33 +0000
  28.     Ready:          False
  29.     Restart Count:  0
  30.     Environment:    <none>
  31.     Mounts:
  32.       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bhszr (ro)
  33. Conditions:
  34.   Type           Status
  35.   Initialized    True 
  36.   Ready          False 
  37.   PodScheduled   True 
  38. Volumes:
  39.   kube-api-access-bhszr:
  40.     Type:                    Projected (a volume that contains injected data from multiple sources)
  41.     TokenExpirationSeconds:  3607
  42.     ConfigMapName:           kube-root-ca.crt
  43.     ConfigMapOptional:       <nil>
  44.     DownwardAPI:             true
  45. QoS Class:                   BestEffort
  46. Node-Selectors:              <none>
  47. Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
  48.                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  49. Events:
  50.   Type    Reason     Age   From               Message
  51.   ----    ------     ----  ----               -------

边缘端

  1. $ sudo crictl ps -a
  2. CONTAINER           IMAGE                                                                                           CREATED             STATE               NAME                ATTEMPT             POD ID
  3. 1ae4d0d7f6710       0423b8eb71e312b8aaa09a0f0b6976381ff567d5b1e5729bf9b9aa87bff1c9f3                                16 minutes ago      Exited              wasi-demo           0                   2bc2ac0c32eda
  4. 1e6c7cb6bc731       k8s.gcr.io/kube-proxy@sha256:2a25285ff19f9b4025c8e54dac42bb3cd9aceadc361f2570489b8d723cb77135   18 minutes ago      Running             kube-proxy          0                   8b7e7388ad866

就是这样。

5. 运行示例的录屏

asciicast