06-0 kube-apiserver 高可用之 nginx 代理

本文档讲解使用 nginx 4 层透明代理功能实现 K8S 节点( master 节点和 worker 节点)高可用访问 kube-apiserver 的步骤。

注意：如果没有特殊指明，本文档的所有操作均在 zhangjun-k8s01 节点上执行，然后远程分发文件和执行命令。

基于 nginx 代理的 kube-apiserver 高可用方案

  • 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署，所以只要有一个实例正常，就可以保证高可用；
  • 集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 IP，所以也是高可用的；
  • 在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡；
  • kubelet、kube-proxy、controller-manager、scheduler 通过本地的 nginx（监听 127.0.0.1）访问 kube-apiserver，从而实现 kube-apiserver 的高可用；

下载和编译 nginx

下载源码：

  1. cd /opt/k8s/work
  2. wget http://nginx.org/download/nginx-1.15.3.tar.gz
  3. tar -xzvf nginx-1.15.3.tar.gz

配置编译参数：

  1. cd /opt/k8s/work/nginx-1.15.3
  2. mkdir nginx-prefix
  3. ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
  • --with-stream：开启 4 层透明转发(TCP Proxy)功能；
  • --without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小；

输出：

  1. Configuration summary
  2.   + PCRE library is not used
  3.   + OpenSSL library is not used
  4.   + zlib library is not used
  6.   nginx path prefix: "/root/tmp/nginx-1.15.3/nginx-prefix"
  7.   nginx binary file: "/root/tmp/nginx-1.15.3/nginx-prefix/sbin/nginx"
  8.   nginx modules path: "/root/tmp/nginx-1.15.3/nginx-prefix/modules"
  9.   nginx configuration prefix: "/root/tmp/nginx-1.15.3/nginx-prefix/conf"
  10.   nginx configuration file: "/root/tmp/nginx-1.15.3/nginx-prefix/conf/nginx.conf"
  11.   nginx pid file: "/root/tmp/nginx-1.15.3/nginx-prefix/logs/nginx.pid"
  12.   nginx error log file: "/root/tmp/nginx-1.15.3/nginx-prefix/logs/error.log"
  13.   nginx http access log file: "/root/tmp/nginx-1.15.3/nginx-prefix/logs/access.log"
  14.   nginx http client request body temporary files: "client_body_temp"
  15.   nginx http proxy temporary files: "proxy_temp"

编译和安装：

  1. cd /opt/k8s/work/nginx-1.15.3
  2. make && make install

验证编译的 nginx

  1. cd /opt/k8s/work/nginx-1.15.3
  2. ./nginx-prefix/sbin/nginx -v

输出：

  1. nginx version: nginx/1.15.3

查看 nginx 动态链接的库：

  1. $ ldd ./nginx-prefix/sbin/nginx

输出：

  1.         linux-vdso.so.1 =>  (0x00007ffc945e7000)
  2.         libdl.so.2 => /lib64/libdl.so.2 (0x00007f4385072000)
  3.         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4384e56000)
  4.         libc.so.6 => /lib64/libc.so.6 (0x00007f4384a89000)
  5.         /lib64/ld-linux-x86-64.so.2 (0x00007f4385276000)
  • 由于只开启了 4 层透明转发功能，所以除了依赖 libc 等操作系统核心 lib 库外，没有对其它 lib 的依赖(如 libz、libssl 等)，这样可以方便部署到各版本操作系统中；

安装和部署 nginx

创建目录结构：

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4.   do
  5.     echo ">>> ${node_ip}"
  6.     mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}
  7.   done

拷贝二进制程序：

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4.   do
  5.     echo ">>> ${node_ip}"
  6.     scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
  7.     ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
  8.     ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
  9.   done
  • 重命名二进制文件为 kube-nginx；

配置 nginx，开启 4 层透明转发功能：

  1. cd /opt/k8s/work
  2. cat > kube-nginx.conf <<EOF
  3. worker_processes 1;
  5. events {
  6.     worker_connections  1024;
  7. }
  9. stream {
  10.     upstream backend {
  11.         hash $remote_addr consistent;
  12.         server 172.27.128.150:6443          max_fails=3 fail_timeout=30s;
  13.         server 172.27.128.149:6443         max_fails=3 fail_timeout=30s;
  14.         server 172.27.128.148:6443        max_fails=3 fail_timeout=30s;
  15.     }
  17.     server {
  18.         listen 127.0.0.1:8443;
  19.         proxy_connect_timeout 1s;
  20.         proxy_pass backend;
  21.     }
  22. }
  23. EOF
  • 需要根据集群 kube-apiserver 的实际情况，替换 backend 中 server 列表；

分发配置文件：

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4.   do
  5.     echo ">>> ${node_ip}"
  6.     scp kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
  7.   done

配置 systemd unit 文件，启动服务

配置 kube-nginx systemd unit 文件：

  1. cd /opt/k8s/work
  2. cat > kube-nginx.service <<EOF
  3. [Unit]
  4. Description=kube-apiserver nginx proxy
  5. After=network.target
  6. After=network-online.target
  7. Wants=network-online.target
  9. [Service]
  10. Type=forking
  11. ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
  12. ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
  13. ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
  14. PrivateTmp=true
  15. Restart=always
  16. RestartSec=5
  17. StartLimitInterval=0
  18. LimitNOFILE=65536
  20. [Install]
  21. WantedBy=multi-user.target
  22. EOF

分发 systemd unit 文件：

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4.   do
  5.     echo ">>> ${node_ip}"
  6.     scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/
  7.   done

启动 kube-nginx 服务：

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4.   do
  5.     echo ">>> ${node_ip}"
  6.     ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
  7.   done

检查 kube-nginx 服务运行状态

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4.   do
  5.     echo ">>> ${node_ip}"
  6.     ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
  7.   done

确保状态为 active (running)，否则查看日志，确认原因：

  1. journalctl -u kube-nginx