Install Manually

SuperEdge officially supports Kubernetes 1.16 and 1.18. This document walk you through how to bootstrap SuperEdge on your Kubernetes cluster.

1. Install Tunnel

1.1 Deploy Tunnel’s CoreDNS

On master nodes,

  1. $ kubectl apply -f deployment/tunnel-coredns.yaml

1.2 Configure tunnel-cloud

Generate and set the following parameters in the deployment/tunnel-cloud.yaml

  1.   TunnelCloudEdgeToken:  #Used for authentication between tunnel cloud and tunnel edge，No less than 32 characters
  2.   TunnelPersistentConnectionServerKey:  #Tunnel cloud server private key(BASE64 encoded), accessed by Tunnel edge
  3.   TunnelPersistentConnectionServerCrt:  #Tunnel cloud server certificate, X.509 BASE64 encoding (PEM format). It can be generated by OpenSSL，signed tunnel-cloud's service name: "tunnelcloud.io".
  4.   TunnelProxyServerKey:  #Tunnel proxy server private key(BASE64 encoding), accessed by kube-apiserver
  5.   TunnelProxyServerCrt:  #Tunnel proxy server certificate, X.509 BASE64 encoding (PEM format)

How to create TunnelPersistentConnectionServerKey and TunnelPersistentConnectionServerCrt? 

  1. Certifications for authentication between tunnel-cloud and tunnel-edge.

  • Generate tunnel-cloud’s CA (You can choose to reuse the Kubernetes cluster’s CA)

    1. # Generate CA private key
    2. openssl genrsa -out tunnel_ca.key 2048
    4. # Generate CSR
    5. openssl req -new -key tunnel_ca.key -out tunnel_ca.csr
    7. # Add DNS and IP
    8. echo "subjectAltName=DNS:superedge.io,IP:127.0.0.1" > tunnel_ca_cert_extensions
    10. # Generate Self Signed certificate
    11. openssl x509 -req -days 365 -in tunnel_ca.csr -signkey tunnel_ca.key -extfile tunnel_ca_cert_extensions -out tunnel_ca.crt

  • Generate TunnelPersistentConnectionServerKey and TunnelPersistentConnectionServerCrt

    1. # private key
    2. openssl genrsa -des3 -out tunnel_persistent_connectiong_server.key 2048
    4. # generate csr
    5. openssl req -new -key tunnel_persistent_connectiong_server.key -subj "/CN=tunnel-cloud" -out tunnel_persistent_connectiong_server.csr
    7. # Add DNS and IP
    8. echo "subjectAltName=DNS:tunnelcloud.io,IP:127.0.0.1" > tunnel_cloud_cert_extensions
    10. # Generate Self Signed certificate
    11. openssl x509 -req -days 365 -in tunnel_persistent_connectiong_server.csr -CA tunnel-cloud-ca.crt -CAkey tunnel_ca.key -CAcreateserial  -extfile tunnel_cloud_cert_extensions -out tunnel_persistent_connectiong_server.crt

  • Get base64 encoded certifications

    1. # generate TunnelPersistentConnectionServerKey
    2. cat tunnel_persistent_connectiong_server.key | base64 --wrap=0
    3. #generate TunnelPersistentConnectionServerCrt
    4. cat tunnel_persistent_connectiong_server.crt | base64 --wrap=0
  1. <details>
  2. <summary>How to create TunnelProxyServerKey and TunnelProxyServerCrt?</summary>
  3. <br>
  5. Certifications for authentication between kube-apiserver and tunnel-cloud.
  7. - Generate TunnelProxyServerKey and TunnelProxyServerCrt
  9. ```bash
  10. # private key
  11. openssl genrsa -des3 -out tunnel_proxy_server.key 2048
  13. # generate csr
  14. openssl req -new -key tunnel_proxy_server.key -subj "/CN=tunnel-cloud" -out tunnel_proxy_server.csr
  16. # Add DNS and IP
  17. echo "subjectAltName=DNS:superedge.io,IP:127.0.0.1" > cert_extensions
  19. # Generate Self Signed certificate（Notice: It is Kubernetes cluster's ca.crt and ca.key, In Kubeadm install method，ca.crt and ca.key path at /etc/kubernetes/pki）
  20. openssl x509 -req -days 365 -in tunnel_proxy_server.csr -CA ca.crt -CAkey ca.key -CAcreateserial  -extfile cert_extensions -out tunnel_proxy_server.crt
  21. ```
  23. - BASE64 encoding tunnel_proxy_server.key and tunnel_proxy_server.crt, just like encoding tunnel_persistent_connectiong_server.key and tunnel_persistent_connectiong_server.crt above
  24. </details>

1.3 Deploy tunnel-cloud

On master nodes,

  1. $ kubectl apply -f deployment/tunnel-cloud.yaml

1.4 Let kube-apiserver using Tunnel

Point the DNS resolution of kube-apiserver to tunnel-CoreDNS. Through DNS hijacking, tunnel proxies the traffic from kube-apiserver to edge nodes. This solves the problem that kube-apiserver ususally can’t connect to edge nodes directly.

  1. #Get tunnel-coredns's Cluster IP
  2. $ kubectl get service tunnel-coredns -n edge-system
  3. NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
  4. tunnel-coredns   ClusterIP   10.10.47.74   <none>        53/UDP,53/TCP,9153/TCP   140m
  5. #Replace kube-apierver's DNS nameservers with tunnel-coredns's Cluster IP
  6. ...
  7. dnsConfig:
  8.     nameservers:
  9.     - 10.10.47.74 #tunnel-cloud's CLUSTER IP；  
  10. ...

Notice: Avoid using IP address as the name of the edge node to avoid DNS hijacking failure.

1.5 Configure tunnel-edge

Set the following parameters in the deployment/tunnel-edge.yaml

  1. MasterIP:  #Normal Kubernetes master node's IP or domain(currently, only one IP address or domain is supported)
  2. TunnelCloudEdgeToken:  #Fill in the same token as "TunnelCloudEdgeToken" in Tunnel-cloud
  3. TunnelPersistentConnectionPort:  #Tunnel-cloud's Persistent connection server Port
  4. KubernetesCaCert:  #kube-apiserver's ca.crt(base64 encoded)
  5. KubeletClientKey:  #Kubelet client key for Tunnel-edge to access Kubelet
  6. KubeletClientCrt:  #Kubelet client cert for Tunnel-edge to access Kubelet

How to create KubeletClientKey and KubeletClientCrt?
Certifications for anthentication between tunnel-edge and Kubelet.

  1. # private key
  2. openssl genrsa -des3 -out kubelet_client.key 1024
  3. # generate csr
  4. openssl req -new -key kubelet_client.key -out kubelet_client.csr
  6. # Generate Self Signed certificate（Notice: it is Kubernetes cluster's ca.crt and ca.key, In Kubeadm install method，ca.crt and ca.key path at /etc/kubernetes/pki）
  7. openssl ca -in kubelet-client.csr -out kubelet-client.crt -cert ca.crt -keyfile ca.key

BASE64 encoding KubeletClientKey and KubeletClientCrt

1.6 Deploy tunnel-edge

On edge worker nodes,

  1. $ kubectl apply -f deployment/tunnel-edge.yaml

2. Install lite-apiserver

2.1 Deploy lite-apiserver

Use Kubernetes cluster’s ca.crt and ca.key to generate lite_apiserver key and certificate(lite-apiserver.key and lite-apiserver.crt). If the cluster is created via Kubeadm，ca.crt and ca.key can be found at /etc/kubernetes/pki

  1. #get service 'kubernetes' ClusterIP
  2. $ kubectl get service kubernetes
  3. NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
  4. kubernetes   ClusterIP   10.10.0.1    <none>        443/TCP   23d
  6. #Generate lite-apiserver.key
  7. $ openssl genrsa -out lite-apiserver.key 2048
  9. #create lite-apiserver.csr
  10. $ cat << EOF >lite_apiserver.conf
  11. [req]
  12. distinguished_name = req_distinguished_name
  13. req_extensions = v3_req
  14. [req_distinguished_name]
  15. CN = lite-apiserver
  16. [v3_req]
  17. basicConstraints = CA:FALSE
  18. keyUsage = nonRepudiation, digitalSignature, keyEncipherment
  19. subjectAltName = @alt_names
  20. [alt_names]
  21. DNS.1 = localhost
  22. IP.1 = 127.0.0.1
  23. IP.2 = 10.10.0.1 # please change the value to Kubernetes's Cluster IP
  24. EOF
  26. $ openssl req -new -key lite-apiserver.key -subj "/CN=lite-apiserver" -config lite-apiserver.conf -out lite-apiserver.csr
  28. #generating lite-apiserver.crt
  29. openssl x509 -req -in lite-apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 5000 -extensions v3_req -extfile lite-apiserver.conf -out lite-apiserver.crt

  • Copy lite-apiserver.crt and lite-apiserver.key into edge worker node, path at /etc/kubernetes/pki/

  • Modify deployment/lite-apiserver.yaml, set –kube-apiserver-url and –kube-apiserver-port to apiserver’s host and port

  • Set –tls-config-file=/etc/kubernetes/edge/tls.json, create /etc/kubernetes/edge/tls.json in edge worker node, and write：

    We need use lite-apiserver to proxy request from kubelet to kube-apiserver, so need let lite-apiserver known kubelet-client key and certificate

    Because of kube-apiserver client key and kube-apiserver client certificate are in kubelet-client-current.pem, so value of “key” and “cert” are “/var/lib/kubelet/pki/kubelet-client-current.pem”.

    1. [
    2.     {
    3.         "key":"/var/lib/kubelet/pki/kubelet-client-current.pem",
    4.         "cert":"/var/lib/kubelet/pki/kubelet-client-current.pem"
    5.     }
    6. ]

    Notice: kubelet-client-current.pem generated by kubeadm, please do not modify any data of kubelet-client-current.pem

  • Use Static Pod to deploy lite-apiserver in Edge Worker Node, copy deployment/lite-apiserver.yaml to Edge Worker Node’s kubelet manifests directory (the directory of Kubernetes cluster builded by kubeadm usually locate at /etc/kubernetes/manifests/）。

2.2 Configure Kubelet to use lite-apiserver

lite-apiserver listen on port 51003 by default (use parameter –port to assign port, in deployment/lite-apiserver.yaml)，please replace kube-apiserver by https://127.0.0.1:51003

3. Install application grid

3.1 Deploy Application Grid Controller

On master nodes,

  1. $ kubectl apply -f deployment/application-grid-controller.yaml

3.2 Add annotate endpoint Kubernetes

Configure Kubernetes endspints to point to lite-apiserver, all traffic from pod to kube-apiserver would be proxyed by lite-apiserver.

  1. kubectl annotate endpoints kubernetes superedge.io/local-endpoint=127.0.0.1
  2. kubectl annotate endpoints kubernetes superedge.io/local-port=51003

3.3 Deploy application grid wrapper

On edge worker nodes,

  1. $ kubectl apply -f deployment/application-grid-wrapper.yaml

Application-grid-wrapper will access kube-apiserver proxyed by lite-apiserver

3.4 Configure kube-proxy to Use Application Grid Wrapper

Modify kube-proxy’s cluster.server to http://127.0.0.1:51006 (kube-proxy’s configuration file is a configmap resource named kube-proxy in kube-system namespace)

application-grid-wrapper listen on port 51006 by default

4. Install edge-health

4.1 Deploy edge-health admission and webhook

On master nodes,

  1. $ kubectl apply -f deployment/edge-health-admission.yaml
  2. $ kubectl apply -f deployment/edge-health-webhook.yaml

Currently the certificates in the webhook is pre-populated, you can replace them with your certificates.

The caBundle in deployment/edge-health-webhook.yaml can be replaced with your CA certificate.

The server.crt and server.key in validate-admission-control-server-certs Secret of deployment/edge-health-admission.yaml can be replaced with your signed certificate and key.

4.3 Configure edge-health

Set the following parameters in the deployment/edge-health.yaml

  1.   HmacKey:  #Hmackey is used in communication between edge-healths, no less than 16 characters

4.3 Deploy edge-health

On edge worker nodes,

  1. $ kubectl apply -f deployment/edge-health.yaml

Last modified June 15, 2021 : initial commit (974355a)