Installation by Binary

Step-by-step installation of binary high-availability karmada cluster.

Prerequisites

Server

3 servers required. E.g.

  1. +---------------+-----------------+-----------------+
  2. |   HostName    |      Host IP    |    Public IP    |
  3. +---------------+-----------------+-----------------+
  4. |  karmada-01   |  172.31.209.245 |  47.242.88.82   |
  5. +---------------+-----------------+-----------------+
  6. |  karmada-02   |  172.31.209.246 |                 |
  7. +---------------+-----------------+-----------------+
  8. |  karmada-03   |  172.31.209.247 |                 |
  9. +---------------+-----------------+-----------------+

Public IP is not required. It is used to download some karmada dependent components from the public network and connect to karmada ApiServer through the public network

DNS Resolution

Execute operations at karmada-01 karmada-02 karmada-03.

  1. vi /etc/hosts
  2. 172.31.209.245  karmada-01
  3. 172.31.209.246  karmada-02
  4. 172.31.209.247  karmada-03

Alternatively, you can use “Linux Virtual Server” for load balancing, and don’t change /etc/hosts file.

Environment

karmada-01 requires the following environment.

Golang: Compile the karmada binary GCC: Compile nginx (ignore if using cloud load balancing)

Compile and Download Binaries

Execute operations at karmada-01.

Kubernetes Binaries

Download the kubernetes binary package.

Refer to this page to download binaries of different versions and architectures: https://kubernetes.io/releases/download/#binaries

  1. wget https://dl.k8s.io/v1.23.3/kubernetes-server-linux-amd64.tar.gz
  2. tar -zxvf kubernetes-server-linux-amd64.tar.gz --no-same-owner
  3. cd kubernetes/server/bin
  4. mv kube-apiserver kube-controller-manager kubectl /usr/local/sbin/

etcd Binaries

Download the etcd binary package.

You may want to use a newer version of etcd, please refer to this page: https://etcd.io/docs/latest/install/

  1. wget https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz
  2. tar -zxvf etcd-v3.5.1-linux-amd64.tar.gz --no-same-owner
  3. cd etcd-v3.5.1-linux-amd64/
  4. mv etcdctl etcd /usr/local/sbin/

Karmada Binaries

Compile the karmada binaries from source.

  1. git clone https://github.com/karmada-io/karmada
  2. cd karmada
  3. make karmada-aggregated-apiserver karmada-controller-manager karmada-scheduler karmada-webhook karmadactl kubectl-karmada
  4. mv _output/bin/linux/amd64/* /usr/local/sbin/

Nginx Binaries

Compile the nginx binary from source.

  1. wget http://nginx.org/download/nginx-1.21.6.tar.gz
  2. tar -zxvf nginx-1.21.6.tar.gz
  3. cd nginx-1.21.6
  4. ./configure --with-stream --without-http --prefix=/usr/local/karmada-nginx --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
  5. make && make install
  6. mv /usr/local/karmada-nginx/sbin/nginx /usr/local/karmada-nginx/sbin/karmada-nginx

Distribute Binaries

Upload the binary file to the karmada-02 karmada-03 server.

Generate Certificates

Step 1: Create Bash Scripts and Configuration Files

The scripts will generate certificates using the openssl command. Download this directory.

We separate CA & leaf certificates generation scripts, so when you need to change Subject Alternative Name of leaf certificates (aka Load Balancer IP), you can reuse CA certificates, and run generate_leaf.sh to generate only leaf certificates.

There are 3 CAs: front-proxy-ca, server-ca, etcd/ca. Why we need 3 CAs please see: PKI certificates and requirements, CA Reusage and Conflicts.

If you use etcd provided by others, you can ignore generate_etcd.sh and csr_config/etcd.

Step 2: Change <SERVER_IP>

You need to change <SERVER_IP> in csr_config/**/*.conf file to your “Load Balancer IP” and “Server IP”. If you only use Load Balancer to access your servers, you only need to fill in “Load Balancer IP”.

You normally don’t need to change *.sh files.

Step 3: Run Shell Scripts

  1. $ ./generate_ca.sh
  2. $ ./generate_leaf.sh ca_cert/
  3. $ ./generate_etcd.sh

Step 4: Check the Certificates

You can view the configuration of the certificate, take karmada.crt as an example.

  1. openssl x509 -noout -text -in karmada.crt

Step 5: Create the Karmada Configuration Directory

Copy the certificates to the /etc/karmada/pki directory.

  1. mkdir -p /etc/karmada/pki
  3. cd ca_cert
  4. cp -r * /etc/karmada/pki
  6. cd ../cert
  7. cp -r * /etc/karmada/pki

Create the Karmada kubeconfig Files and etcd Encryption Key

Execute operations at karmada-01.

Create kubeconfig Files

Step 1: Download bash script

Download this file.

Step 2: execute bash script

172.31.209.245:5443 is the address of the nginx proxy for karmada-apiserver, we’ll set it up later. You should replace it with your Load Balancer provided “host:port”.

  1. ./create_kubeconfig_file.sh "https://172.31.209.245:5443"

Create etcd Encryption Key

If you don’t need to encrypt contents in etcd, ignore this section and corresponding kube-apiserver start parameter.

  1. export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
  2. cat > /etc/karmada/encryption-config.yaml <<EOF
  3. kind: EncryptionConfig
  4. apiVersion: v1
  5. resources:
  6.   - resources:
  7.       - secrets
  8.     providers:
  9.       - aescbc:
  10.           keys:
  11.             - name: key1
  12.               secret: ${ENCRYPTION_KEY}
  13.       - identity: {}
  14. EOF

Distribute Files

Package the karmada configuration files and copy them to other nodes.

  1. cd /etc
  2. tar -cvf karmada.tar karmada
  3. scp karmada.tar  karmada-02:/etc/
  4. scp karmada.tar  karmada-03:/etc/

karmada-02 karmada-03 need to decompress archives.

  1. cd /etc
  2. tar -xvf karmada.tar

Health Check Script

(1) You can use the following check_status.sh to check if all components are healthy.

Download this file.

Note: If you are using CentOS 7, you need to run yum update -y nss curl to update curl version, then curl may support tls 1.3. If curl still does not support tls 1.3, you need to update OS version which includes newer curl, or simply use a go program to do a health check.

(2) Usage: ./check_status.sh

Install etcd cluster

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

Create Systemd Service

/usr/lib/systemd/system/etcd.service

  1. [Unit]
  2. Description=Etcd Server
  3. After=network.target
  4. After=network-online.target
  5. Wants=network-online.target
  6. Documentation=https://github.com/coreos
  8. [Service]
  9. Type=notify
  10. WorkingDirectory=/var/lib/etcd/
  11. ExecStart=/usr/local/sbin/etcd \
  12.   --advertise-client-urls https://172.31.209.245:2379 \
  13.   --cert-file /etc/karmada/pki/etcd/server.crt \
  14.   --client-cert-auth=true \
  15.   --data-dir /var/lib/etcd \
  16.   --initial-advertise-peer-urls https://172.31.209.245:2380 \
  17.   --initial-cluster "karmada-01=https://172.31.209.245:2380,karmada-02=https://172.31.209.246:2380,karmada-03=https://172.31.209.247:2380" \
  18.   --initial-cluster-state new \
  19.   --initial-cluster-token etcd-cluster \
  20.   --key-file /etc/karmada/pki/etcd/server.key \
  21.   --listen-client-urls "https://172.31.209.245:2379,https://127.0.0.1:2379" \
  22.   --listen-peer-urls "https://172.31.209.245:2380" \
  23.   --name karmada-01 \
  24.   --peer-cert-file /etc/karmada/pki/etcd/peer.crt \
  25.   --peer-client-cert-auth=true \
  26.   --peer-key-file /etc/karmada/pki/etcd/peer.key \
  27.   --peer-trusted-ca-file /etc/karmada/pki/etcd/ca.crt \
  28.   --snapshot-count 10000 \
  29.   --trusted-ca-file /etc/karmada/pki/etcd/ca.crt \
  31. Restart=on-failure
  32. RestartSec=5
  33. LimitNOFILE=65536
  35. [Install]
  36. WantedBy=multi-user.target

Notice:

The parameters that karmada-02 karmada-03 need to change are:

--name

--initial-advertise-peer-urls

--listen-peer-urls

--listen-client-urls

--advertise-client-urls

You can use EnvironmentFile to seperate mutable configs from immutable configs.

Start etcd cluster

3 servers have to execute.

create etcd storage directory

  1. mkdir /var/lib/etcd/
  2. chmod 700 /var/lib/etcd

start etcd

  1. systemctl daemon-reload
  2. systemctl enable etcd.service
  3. systemctl start etcd.service
  4. systemctl status etcd.service

Verify

  1. etcdctl --cacert /etc/karmada/pki/etcd/ca.crt \
  2.     --cert /etc/karmada/pki/etcd/healthcheck-client.crt \
  3.     --key /etc/karmada/pki/etcd/healthcheck-client.key \
  4.     --endpoints "172.31.209.245:2379,172.31.209.246:2379,172.31.209.247:2379" \
  5.     endpoint status --write-out="table"
  7. +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
  8. |      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
  9. +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
  10. | 172.31.209.245:2379 | 689151f8cbf4ee95 |   3.5.1 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
  11. | 172.31.209.246:2379 | 5db4dfb6ecc14de7 |   3.5.1 |   20 kB |      true |      false |         2 |          9 |                  9 |        |
  12. | 172.31.209.247:2379 | 7e59eef3c816aa57 |   3.5.1 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
  13. +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Install kube-apiserver

Configure Nginx

Execute operations at karmada-01.

configure load balancing for karmada apiserver

/usr/local/karmada-nginx/conf/nginx.conf

  1. worker_processes 2;
  3. events {
  4.     worker_connections  1024;
  5. }
  7. stream {
  8.     upstream backend {
  9.         hash consistent;
  10.         server 172.31.209.245:6443        max_fails=3 fail_timeout=30s;
  11.         server 172.31.209.246:6443        max_fails=3 fail_timeout=30s;
  12.         server 172.31.209.247:6443        max_fails=3 fail_timeout=30s;
  13.     }
  15.     server {
  16.         listen 172.31.209.245:5443;
  17.         proxy_connect_timeout 1s;
  18.         proxy_pass backend;
  19.     }
  20. }

/lib/systemd/system/karmada-nginx.service

  1. [Unit]
  2. Description=The karmada karmada-apiserver nginx proxy server
  3. After=syslog.target network-online.target remote-fs.target nss-lookup.target
  4. Wants=network-online.target
  6. [Service]
  7. Type=forking
  8. ExecStartPre=/usr/local/karmada-nginx/sbin/karmada-nginx -t
  9. ExecStart=/usr/local/karmada-nginx/sbin/karmada-nginx
  10. ExecReload=/usr/local/karmada-nginx/sbin/karmada-nginx -s reload
  11. ExecStop=/bin/kill -s QUIT $MAINPID
  12. PrivateTmp=true
  13. Restart=always
  14. RestartSec=5
  15. StartLimitInterval=0
  16. LimitNOFILE=65536
  18. [Install]
  19. WantedBy=multi-user.target

start karmada nginx

  1. systemctl daemon-reload
  2. systemctl enable karmada-nginx.service
  3. systemctl start karmada-nginx.service
  4. systemctl status karmada-nginx.service

Create kube-apiserver Systemd Service

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/kube-apiserver.service

  1. [Unit]
  2. Description=Kubernetes API Server
  3. Documentation=https://kubernetes.io/docs/home/
  4. After=network.target
  6. [Service]
  7. # If you don't need to encrypt etcd, remove --encryption-provider-config
  8. ExecStart=/usr/local/sbin/kube-apiserver \
  9.   --allow-privileged=true \
  10.   --anonymous-auth=false \
  11.   --audit-webhook-batch-buffer-size 30000 \
  12.   --audit-webhook-batch-max-size 800 \
  13.   --authorization-mode "Node,RBAC" \
  14.   --bind-address 0.0.0.0 \
  15.   --client-ca-file /etc/karmada/pki/server-ca.crt \
  16.   --default-watch-cache-size 200 \
  17.   --delete-collection-workers 2 \
  18.   --disable-admission-plugins "StorageObjectInUseProtection,ServiceAccount" \
  19.   --enable-admission-plugins "NodeRestriction" \
  20.   --enable-bootstrap-token-auth \
  21.   --encryption-provider-config "/etc/karmada/encryption-config.yaml" \
  22.   --etcd-cafile /etc/karmada/pki/etcd/ca.crt \
  23.   --etcd-certfile /etc/karmada/pki/etcd/apiserver-etcd-client.crt \
  24.   --etcd-keyfile /etc/karmada/pki/etcd/apiserver-etcd-client.key \
  25.   --etcd-servers "https://172.31.209.245:2379,https://172.31.209.246:2379,https://172.31.209.247:2379" \
  26.   --insecure-port 0 \
  27.   --logtostderr=true \
  28.   --max-mutating-requests-inflight 2000 \
  29.   --max-requests-inflight 4000 \
  30.   --proxy-client-cert-file /etc/karmada/pki/front-proxy-client.crt \
  31.   --proxy-client-key-file /etc/karmada/pki/front-proxy-client.key \
  32.   --requestheader-allowed-names "front-proxy-client" \
  33.   --requestheader-client-ca-file /etc/karmada/pki/front-proxy-ca.crt \
  34.   --requestheader-extra-headers-prefix "X-Remote-Extra-" \
  35.   --requestheader-group-headers "X-Remote-Group" \
  36.   --requestheader-username-headers "X-Remote-User" \
  37.   --runtime-config "api/all=true" \
  38.   --secure-port 6443 \
  39.   --service-account-issuer "https://kubernetes.default.svc.cluster.local" \
  40.   --service-account-key-file /etc/karmada/pki/sa.pub \
  41.   --service-account-signing-key-file /etc/karmada/pki/sa.key \
  42.   --service-cluster-ip-range "10.254.0.0/16" \
  43.   --tls-cert-file /etc/karmada/pki/kube-apiserver.crt \
  44.   --tls-private-key-file /etc/karmada/pki/kube-apiserver.key \
  46. Restart=on-failure
  47. RestartSec=5
  48. Type=notify
  49. LimitNOFILE=65536
  51. [Install]
  52. WantedBy=multi-user.target

Start kube-apiserver

3 servers have to execute.

  1. systemctl daemon-reload
  2. systemctl enable kube-apiserver.service
  3. systemctl start kube-apiserver.service
  4. systemctl status kube-apiserver.service

Verify

  1. $ ./check_status.sh
  2. ###### Start check kube-apiserver
  3. [+]ping ok
  4. [+]log ok
  5. [+]etcd ok
  6. [+]poststarthook/start-kube-apiserver-admission-initializer ok
  7. [+]poststarthook/generic-apiserver-start-informers ok
  8. [+]poststarthook/priority-and-fairness-config-consumer ok
  9. [+]poststarthook/priority-and-fairness-filter ok
  10. [+]poststarthook/start-apiextensions-informers ok
  11. [+]poststarthook/start-apiextensions-controllers ok
  12. [+]poststarthook/crd-informer-synced ok
  13. [+]poststarthook/bootstrap-controller ok
  14. [+]poststarthook/rbac/bootstrap-roles ok
  15. [+]poststarthook/scheduling/bootstrap-system-priority-classes ok
  16. [+]poststarthook/priority-and-fairness-config-producer ok
  17. [+]poststarthook/start-cluster-authentication-info-controller ok
  18. [+]poststarthook/aggregator-reload-proxy-client-cert ok
  19. [+]poststarthook/start-kube-aggregator-informers ok
  20. [+]poststarthook/apiservice-registration-controller ok
  21. [+]poststarthook/apiservice-status-available-controller ok
  22. [+]poststarthook/kube-apiserver-autoregistration ok
  23. [+]autoregister-completion ok
  24. [+]poststarthook/apiservice-openapi-controller ok
  25. livez check passed
  27. ###### kube-apiserver check success

Install karmada-aggregated-apiserver

Create namespace and bind the cluster admin role. Execute operations at karmada-01.

  1. kubectl create ns karmada-system
  2. kubectl create clusterrolebinding cluster-admin:karmada --clusterrole=cluster-admin --user system:karmada

Then, like karmada-webhook, use nginx for high availability.

modify the nginx configuration and add the following configuration,Execute operations at karmada-01.

  1. cat /usr/local/karmada-nginx/conf/nginx.conf
  2. worker_processes 2;
  4. events {
  5.     worker_connections  1024;
  6. }
  8. stream {
  9.     upstream backend {
  10.         hash  consistent;
  11.         server 172.31.209.245:6443        max_fails=3 fail_timeout=30s;
  12.         server 172.31.209.246:6443        max_fails=3 fail_timeout=30s;
  13.         server 172.31.209.247:6443        max_fails=3 fail_timeout=30s;
  14.     }
  16.     upstream webhook {
  17.         hash  consistent;
  18.         server 172.31.209.245:8443        max_fails=3 fail_timeout=30s;
  19.         server 172.31.209.246:8443        max_fails=3 fail_timeout=30s;
  20.         server 172.31.209.247:8443        max_fails=3 fail_timeout=30s;
  21.     }
  23.     upstream aa {
  24.         hash  consistent;
  25.         server 172.31.209.245:7443        max_fails=3 fail_timeout=30s;
  26.         server 172.31.209.246:7443        max_fails=3 fail_timeout=30s;
  27.         server 172.31.209.247:7443        max_fails=3 fail_timeout=30s;
  28.     }
  30.     server {
  31.         listen 172.31.209.245:5443;
  32.         proxy_connect_timeout 1s;
  33.         proxy_pass backend;
  34.     }
  36.     server {
  37.         listen 172.31.209.245:4443;
  38.         proxy_connect_timeout 1s;
  39.         proxy_pass webhook;
  40.     }
  42.     server {
  43.         listen 172.31.209.245:443;
  44.         proxy_connect_timeout 1s;
  45.         proxy_pass aa;
  46.     }
  47. }

Reload nginx configuration

  1. systemctl restart karmada-nginx

Create Systemd Service

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-aggregated-apiserver.service

  1. [Unit]
  2. Description=Karmada Aggregated ApiServer
  3. Documentation=https://github.com/karmada-io/karmada
  5. [Service]
  6. ExecStart=/usr/local/sbin/karmada-aggregated-apiserver \
  7.   --audit-log-maxage 0  \
  8.   --audit-log-maxbackup 0 \
  9.   --audit-log-path -  \
  10.   --authentication-kubeconfig /etc/karmada/karmada.kubeconfig  \
  11.   --authorization-kubeconfig /etc/karmada/karmada.kubeconfig  \
  12.   --etcd-cafile /etc/karmada/pki/etcd/ca.crt \
  13.   --etcd-certfile /etc/karmada/pki/etcd/apiserver-etcd-client.crt \
  14.   --etcd-keyfile /etc/karmada/pki/etcd/apiserver-etcd-client.key \
  15.   --etcd-servers "https://172.31.209.245:2379,https://172.31.209.246:2379,https://172.31.209.247:2379" \
  16.   --feature-gates "APIPriorityAndFairness=false"  \
  17.   --kubeconfig /etc/karmada/karmada.kubeconfig \
  18.   --logtostderr=true \
  19.   --secure-port 7443 \
  20.   --tls-cert-file /etc/karmada/pki/karmada.crt \
  21.   --tls-private-key-file /etc/karmada/pki/karmada.key \
  23. Restart=on-failure
  24. RestartSec=5
  25. LimitNOFILE=65536
  27. [Install]
  28. WantedBy=multi-user.target

Start karmada-aggregated-apiserver

  1. systemctl daemon-reload
  2. systemctl enable karmada-aggregated-apiserver.service
  3. systemctl start karmada-aggregated-apiserver.service
  4. systemctl status karmada-aggregated-apiserver.service

Create APIService

externalName is the host name of the node where nginx is located (karmada-01).

(1) create file: karmada-aggregated-apiserver-apiservice.yaml

  1. apiVersion: apiregistration.k8s.io/v1
  2. kind: APIService
  3. metadata:
  4.   name: v1alpha1.cluster.karmada.io
  5.   labels:
  6.     app: karmada-aggregated-apiserver
  7.     apiserver: "true"
  8. spec:
  9.   insecureSkipTLSVerify: true
  10.   group: cluster.karmada.io
  11.   groupPriorityMinimum: 2000
  12.   service:
  13.     name: karmada-aggregated-apiserver
  14.     namespace: karmada-system
  15.     port: 443
  16.   version: v1alpha1
  17.   versionPriority: 10
  18. ---
  19. apiVersion: v1
  20. kind: Service
  21. metadata:
  22.   name: karmada-aggregated-apiserver
  23.   namespace: karmada-system
  24. spec:
  25.   type: ExternalName
  26.   externalName: karmada-01

(2) kubectl create -f karmada-aggregated-apiserver-apiservice.yaml

Verify

  1. $ ./check_status.sh
  2. ###### Start check karmada-aggregated-apiserver
  3. [+]ping ok
  4. [+]log ok
  5. [+]etcd ok
  6. [+]poststarthook/generic-apiserver-start-informers ok
  7. [+]poststarthook/max-in-flight-filter ok
  8. [+]poststarthook/start-aggregated-server-informers ok
  9. livez check passed
  11. ###### karmada-aggregated-apiserver check success

Install kube-controller-manager

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

Create Systemd Service

/usr/lib/systemd/system/kube-controller-manager.service

  1. [Unit]
  2. Description=Kubernetes Controller Manager
  3. Documentation=https://kubernetes.io/docs/home/
  4. After=network.target
  6. [Service]
  7. ExecStart=/usr/local/sbin/kube-controller-manager \
  8.   --authentication-kubeconfig /etc/karmada/kube-controller-manager.kubeconfig \
  9.   --authorization-kubeconfig /etc/karmada/kube-controller-manager.kubeconfig \
  10.   --bind-address "0.0.0.0" \
  11.   --client-ca-file /etc/karmada/pki/server-ca.crt \
  12.   --cluster-name karmada \
  13.   --cluster-signing-cert-file /etc/karmada/pki/server-ca.crt \
  14.   --cluster-signing-key-file /etc/karmada/pki/server-ca.key \
  15.   --concurrent-deployment-syncs 10 \
  16.   --concurrent-gc-syncs 30 \
  17.   --concurrent-service-syncs 1 \
  18.   --controllers "namespace,garbagecollector,serviceaccount-token" \
  19.   --feature-gates "RotateKubeletServerCertificate=true" \
  20.   --horizontal-pod-autoscaler-sync-period 10s \
  21.   --kube-api-burst 2000 \
  22.   --kube-api-qps 1000 \
  23.   --kubeconfig /etc/karmada/kube-controller-manager.kubeconfig \
  24.   --leader-elect \
  25.   --logtostderr=true \
  26.   --node-cidr-mask-size 24 \
  27.   --pod-eviction-timeout 5m \
  28.   --requestheader-allowed-names "front-proxy-client" \
  29.   --requestheader-client-ca-file /etc/karmada/pki/front-proxy-ca.crt \
  30.   --requestheader-extra-headers-prefix "X-Remote-Extra-" \
  31.   --requestheader-group-headers "X-Remote-Group" \
  32.   --requestheader-username-headers "X-Remote-User" \
  33.   --root-ca-file /etc/karmada/pki/server-ca.crt \
  34.   --service-account-private-key-file /etc/karmada/pki/sa.key \
  35.   --service-cluster-ip-range "10.254.0.0/16" \
  36.   --terminated-pod-gc-threshold 10000 \
  37.   --tls-cert-file /etc/karmada/pki/kube-controller-manager.crt \
  38.   --tls-private-key-file /etc/karmada/pki/kube-controller-manager.key \
  39.   --use-service-account-credentials \
  40.   --v 4 \
  42. Restart=on-failure
  43. RestartSec=5
  44. LimitNOFILE=65536
  46. [Install]
  47. WantedBy=multi-user.target

Start kube-controller-manager

  1. systemctl daemon-reload
  2. systemctl enable kube-controller-manager.service
  3. systemctl start kube-controller-manager.service
  4. systemctl status kube-controller-manager.service

Verify

  1. $ ./check_status.sh
  2. ###### Start check kube-controller-manager
  3. [+]leaderElection ok
  4. healthz check passed
  6. ###### kube-controller-manager check success

Install karmada-controller-manager

Create Systemd Service

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-controller-manager.service

  1. [Unit]
  2. Description=Karmada Controller Manager
  3. Documentation=https://github.com/karmada-io/karmada
  5. [Service]
  6. ExecStart=/usr/local/sbin/karmada-controller-manager \
  7.   --bind-address 0.0.0.0 \
  8.   --cluster-status-update-frequency 10s \
  9.   --kubeconfig /etc/karmada/karmada.kubeconfig \
  10.   --logtostderr=true \
  11.   --metrics-bind-address ":10358" \
  12.   --secure-port 10357 \
  13.   --v=4 \
  15. Restart=on-failure
  16. RestartSec=5
  17. LimitNOFILE=65536
  19. [Install]
  20. WantedBy=multi-user.target

Start karmada-controller-manager

  1. systemctl daemon-reload
  2. systemctl enable karmada-controller-manager.service
  3. systemctl start karmada-controller-manager.service
  4. systemctl status karmada-controller-manager.service

Verify

  1. $ ./check_status.sh
  2. ###### Start check karmada-controller-manager
  3. [+]ping ok
  4. healthz check passed
  6. ###### karmada-controller-manager check success

Install karmada-scheduler

Create Systemd Service

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-scheduler.service

  1. [Unit]
  2. Description=Karmada Scheduler
  3. Documentation=https://github.com/karmada-io/karmada
  5. [Service]
  6. ExecStart=/usr/local/sbin/karmada-scheduler \
  7.   --bind-address 0.0.0.0 \
  8.   --enable-scheduler-estimator=true \
  9.   --kubeconfig /etc/karmada/karmada.kubeconfig \
  10.   --logtostderr=true \
  11.   --scheduler-estimator-port 10352 \
  12.   --secure-port 10511 \
  13.   --v=4 \
  15. Restart=on-failure
  16. RestartSec=5
  17. LimitNOFILE=65536
  19. [Install]
  20. WantedBy=multi-user.target

Start karmada-scheduler

  1. systemctl daemon-reload
  2. systemctl enable karmada-scheduler.service
  3. systemctl start karmada-scheduler.service
  4. systemctl status karmada-scheduler.service

Verify

  1. $ ./check_status.sh
  2. ###### Start check karmada-scheduler
  3. ok
  4. ###### karmada-scheduler check success

Install karmada-webhook

karmada-webhook is different from scheduler and controller-manager, and its high availability needs to be implemented with nginx.

modify the nginx configuration and add the following configuration,Execute operations at karmada-01.

  1. cat /usr/local/karmada-nginx/conf/nginx.conf
  2. worker_processes 2;
  4. events {
  5.     worker_connections  1024;
  6. }
  8. stream {
  9.     upstream backend {
  10.         hash  consistent;
  11.         server 172.31.209.245:6443        max_fails=3 fail_timeout=30s;
  12.         server 172.31.209.246:6443        max_fails=3 fail_timeout=30s;
  13.         server 172.31.209.247:6443        max_fails=3 fail_timeout=30s;
  14.     }
  16.     upstream webhook {
  17.         hash  consistent;
  18.         server 172.31.209.245:8443        max_fails=3 fail_timeout=30s;
  19.         server 172.31.209.246:8443        max_fails=3 fail_timeout=30s;
  20.         server 172.31.209.247:8443        max_fails=3 fail_timeout=30s;
  21.     }
  23.     server {
  24.         listen 172.31.209.245:5443;
  25.         proxy_connect_timeout 1s;
  26.         proxy_pass backend;
  27.     }
  29.     server {
  30.         listen 172.31.209.245:4443;
  31.         proxy_connect_timeout 1s;
  32.         proxy_pass webhook;
  33.     }
  34. }

Reload nginx configuration

  1. systemctl restart karmada-nginx

Create Systemd Service

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-webhook.service

  1. [Unit]
  2. Description=Karmada Webhook
  3. Documentation=https://github.com/karmada-io/karmada
  5. [Service]
  6. ExecStart=/usr/local/sbin/karmada-webhook \
  7.   --bind-address 0.0.0.0 \
  8.   --cert-dir /etc/karmada/pki \
  9.   --health-probe-bind-address ":8444" \
  10.   --kubeconfig /etc/karmada/karmada.kubeconfig \
  11.   --logtostderr=true \
  12.   --metrics-bind-address ":8445" \
  13.   --secure-port 8443 \
  14.   --tls-cert-file-name "karmada.crt" \
  15.   --tls-private-key-file-name "karmada.key" \
  16.   --v=4 \
  18. Restart=on-failure
  19. RestartSec=5
  20. LimitNOFILE=65536
  22. [Install]
  23. WantedBy=multi-user.target

Start karmada-webook

  1. systemctl daemon-reload
  2. systemctl enable karmada-webhook.service
  3. systemctl start karmada-webhook.service
  4. systemctl status karmada-webhook.service

Configurate karmada-webhook

Download the webhook-configuration.yaml file: https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/webhook-configuration.yaml

  1. ca_string=$(cat /etc/karmada/pki/server-ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  2. sed -i "s/{{caBundle}}/${ca_string}/g"  webhook-configuration.yaml
  3. # You need to change 172.31.209.245:4443 to your Load Balancer host:port.
  4. sed -i 's/karmada-webhook.karmada-system.svc:443/172.31.209.245:4443/g'  webhook-configuration.yaml
  6. kubectl create -f  webhook-configuration.yaml

Verify

  1. $ ./check_status.sh
  2. ###### Start check karmada-webhook
  3. ok
  4. ###### karmada-webhook check success

Initialize Karmada

Execute operations at karmada-01.

  1. git clone https://github.com/karmada-io/karmada
  2. cd karmada/charts/karmada/_crds/bases
  4. kubectl apply -f .
  6. cd ../patches/
  7. ca_string=$(cat /etc/karmada/pki/server-ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
  8. sed -i "s/{{caBundle}}/${ca_string}/g" webhook_in_resourcebindings.yaml
  9. sed -i "s/{{caBundle}}/${ca_string}/g"  webhook_in_clusterresourcebindings.yaml
  10. # You need to change 172.31.209.245:4443 to your Load Balancer host:port.
  11. sed -i 's/karmada-webhook.karmada-system.svc:443/172.31.209.245:4443/g' webhook_in_resourcebindings.yaml
  12. sed -i 's/karmada-webhook.karmada-system.svc:443/172.31.209.245:4443/g' webhook_in_clusterresourcebindings.yaml
  14. kubectl patch CustomResourceDefinition resourcebindings.work.karmada.io --patch-file webhook_in_resourcebindings.yaml
  15. kubectl patch CustomResourceDefinition clusterresourcebindings.work.karmada.io --patch-file webhook_in_clusterresourcebindings.yaml

Now, all the required components have been installed, and the member clusters could join Karmada control plane. If you want to use karmadactl to query, please run following command:

  1. cat <<EOF | kubectl --kubeconfig=/etc/karmada/admin.kubeconfig  apply -f -
  2. apiVersion: rbac.authorization.k8s.io/v1
  3. kind: ClusterRole
  4. metadata:
  5.   name: cluster-proxy-clusterrole
  6. rules:
  7. - apiGroups:
  8.   - 'cluster.karmada.io'
  9.   resources:
  10.   - clusters/proxy
  11.   verbs:
  12.   - '*'
  13. ---
  14. apiVersion: rbac.authorization.k8s.io/v1
  15. kind: ClusterRoleBinding
  16. metadata:
  17.   name: cluster-proxy-clusterrolebinding
  18. roleRef:
  19.   apiGroup: rbac.authorization.k8s.io
  20.   kind: ClusterRole
  21.   name: cluster-proxy-clusterrole
  22. subjects:
  23.   - kind: User
  24.     name: admin  # The name should be the user of client certificate
  25.   # The token generated by the serviceaccount can parse the group information. Therefore, you need to specify the group information below.
  26.   - kind: Group
  27.     name: "system:masters" # The name should be the gourp of client certificate
  28. EOF

Install karmada-scheduler-estimator (Optional)

karmada-scheduler uses gRPC to access karmada-scheduler-estimator. You can use “Linux Virtual Server” as the Load Balancer.

You need to deploy above components and join member cluster into Karmada Control Plane first. See: Install Karmada on your own cluster

Create Systemd Service

In the example below, “/etc/karmada/physical-machine-karmada-member-1.kubeconfig” is kubeconfig file of the joined member cluster.

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-scheduler-estimator.service

  1. [Unit]
  2. Description=Karmada Scheduler Estimator
  3. Documentation=https://github.com/karmada-io/karmada
  5. [Service]
  6. # You need to change `--cluster-name` `--kubeconfig`
  7. ExecStart=/usr/local/sbin/karmada-scheduler-estimator \
  8.   --cluster-name "physical-machine-karmada-member-1" \
  9.   --kubeconfig "/etc/karmada/physical-machine-karmada-member-1.kubeconfig" \
  10.   --logtostderr=true \
  11.   --server-port 10352 \
  13. Restart=on-failure
  14. RestartSec=5
  15. LimitNOFILE=65536
  17. [Install]
  18. WantedBy=multi-user.target

Start karmada-scheduler-estimator

  1. systemctl daemon-reload
  2. systemctl enable karmada-scheduler-estimator.service
  3. systemctl start karmada-scheduler-estimator.service
  4. systemctl status karmada-scheduler-estimator.service

Create Service

(1) Create karmada-scheduler-estimator.yaml

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: karmada-scheduler-estimator-{{MEMBER_CLUSTER_NAME}}
  5.   namespace: karmada-system
  6.   labels:
  7.     cluster: {{MEMBER_CLUSTER_NAME}}
  8. spec:
  9.   ports:
  10.     - protocol: TCP
  11.       port: {{PORT}}
  12.       targetPort: {{TARGET_PORT}}
  13.   type: ExternalName
  14.   externalName: {{EXTERNAL_NAME}}

{{PORT}}: “—scheduler-estimator-port” parameter of “karmada-scheduler”.

{{TARGET_PORT}}: LoadBalancer IP.

{{EXTERNAL_NAME}}: LoadBalancer Host.

{{MEMBER_CLUSTER_NAME}}: Member Cluster Name

(2) Create Service

  1. kubectl create --kubeconfig "/etc/karmada/admin.kubeconfig" -f karmada-scheduler-estimator.yaml

Verify

  1. $ ./check_status.sh
  2. ###### Start check karmada-scheduler-estimator
  3. ok
  4. ###### karmada-scheduler-estimator check success

Trouble Shooting

  1. karmada-scheduler takes some time to find the new karmada-scheduler-estimator. If you can’t wait, you can just restart karmada-scheduler.
  1. systemctl restart karmada-scheduler.service

Install karmada-search (Optional)

Create Systemd Service

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-search.service

  1. [Unit]
  2. Description=Karmada Search
  3. Documentation=https://github.com/karmada-io/karmada
  5. [Service]
  6. ExecStart=/usr/local/sbin/karmada-search \
  7.   --audit-log-maxage 0  \
  8.   --audit-log-maxbackup 0 \
  9.   --audit-log-path -  \
  10.   --authentication-kubeconfig /etc/karmada/karmada.kubeconfig  \
  11.   --authorization-kubeconfig /etc/karmada/karmada.kubeconfig  \
  12.   --etcd-cafile /etc/karmada/pki/etcd/ca.crt \
  13.   --etcd-certfile /etc/karmada/pki/etcd/apiserver-etcd-client.crt \
  14.   --etcd-keyfile /etc/karmada/pki/etcd/apiserver-etcd-client.key \
  15.   --etcd-servers "https://172.31.209.245:2379,https://172.31.209.246:2379,https://172.31.209.247:2379" \
  16.   --feature-gates "APIPriorityAndFairness=false"  \
  17.   --kubeconfig /etc/karmada/karmada.kubeconfig \
  18.   --logtostderr=true \
  19.   --secure-port 9443 \
  20.   --tls-cert-file /etc/karmada/pki/karmada.crt \
  21.   --tls-private-key-file /etc/karmada/pki/karmada.key \
  23. Restart=on-failure
  24. RestartSec=5
  25. LimitNOFILE=65536
  27. [Install]
  28. WantedBy=multi-user.target
  1. systemctl daemon-reload
  2. systemctl enable karmada-search.service
  3. systemctl start karmada-search.service
  4. systemctl status karmada-search.service

Verify

  1. $ ./check_status.sh
  2. ###### Start check karmada-search
  3. [+]ping ok
  4. [+]log ok
  5. [+]etcd ok
  6. [+]poststarthook/generic-apiserver-start-informers ok
  7. [+]poststarthook/max-in-flight-filter ok
  8. [+]poststarthook/start-karmada-search-informers ok
  9. [+]poststarthook/start-karmada-informers ok
  10. [+]poststarthook/start-karmada-search-controller ok
  11. [+]poststarthook/start-karmada-proxy-controller ok
  12. livez check passed
  14. ###### karmada-search check success

Configure

Proxy Global Resources