Kubernetes

基于 Kubernetes 的服务发现

Kubernetes 服务发现以 List-Watch 方式监听 Kubernetes 集群 Endpoints 资源的实时变化,并将其值存储到 ngx.shared.DICT 中。

同时遵循 APISIX Discovery 规范 提供了节点查询接口。

Kubernetes 服务发现的使用

目前 Kubernetes 服务发现支持单集群和多集群模式,分别适用于待发现的服务分布在单个或多个 Kubernetes 的场景。

单集群模式 Kubernetes 服务发现的配置格式

单集群模式 Kubernetes 服务发现的完整配置如下:

  1. discovery:
  2. kubernetes:
  3. service:
  4. # apiserver schema, options [http, https]
  5. schema: https #default https
  6. # apiserver host, options [ipv4, ipv6, domain, environment variable]
  7. host: ${KUBERNETES_SERVICE_HOST} #default ${KUBERNETES_SERVICE_HOST}
  8. # apiserver port, options [port number, environment variable]
  9. port: ${KUBERNETES_SERVICE_PORT} #default ${KUBERNETES_SERVICE_PORT}
  10. client:
  11. # serviceaccount token or token_file
  12. token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  13. #token: |-
  14. # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif
  15. # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI
  16. default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0
  17. # kubernetes discovery support namespace_selector
  18. # you can use one of [equal, not_equal, match, not_match] filter namespace
  19. namespace_selector:
  20. # only save endpoints with namespace equal default
  21. equal: default
  22. # only save endpoints with namespace not equal default
  23. #not_equal: default
  24. # only save endpoints with namespace match one of [default, ^my-[a-z]+$]
  25. #match:
  26. #- default
  27. #- ^my-[a-z]+$
  28. # only save endpoints with namespace not match one of [default, ^my-[a-z]+$]
  29. #not_match:
  30. #- default
  31. #- ^my-[a-z]+$
  32. # kubernetes discovery support label_selector
  33. # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
  34. label_selector: |-
  35. first="a",second="b"
  36. # reserved lua shared memory size, 1m memory can store about 1000 pieces of endpoint
  37. shared_size: 1m #default 1m
  38. # if watch_endpoint_slices setting true, watch apiserver with endpointslices instead of endpoints
  39. watch_endpoint_slices: false #default false

如果 Kubernetes 服务发现运行在 Pod 内,你可以使用如下最简配置:

  1. discovery:
  2. kubernetes: { }

如果 Kubernetes 服务发现运行在 Pod 外,你需要新建或选取指定的 ServiceAccount, 获取其 Token 值,然后使用如下配置:

  1. discovery:
  2. kubernetes:
  3. service:
  4. schema: https
  5. host: # enter apiserver host value here
  6. port: # enter apiServer port value here
  7. client:
  8. token: # enter serviceaccount token value here
  9. #token_file: # enter token file path here

单集群模式 Kubernetes 服务发现的查询接口

单集群模式 Kubernetes 服务发现遵循 APISIX Discovery 规范 提供节点查询接口。

函数: nodes(service_name)

说明: service_name 必须满足格式:[namespace]/[name]:[portName]

  • namespace: Endpoints 所在的命名空间

  • name: Endpoints 的资源名

  • portName: Endpoints 定义包含的 ports.name 值,如果 Endpoints 没有定义 ports.name,请依次使用 targetPort, port 代替。设置了 ports.name 的情况下,不能使用后两者。

返回值: 以如下 Endpoints 为例:

  1. apiVersion: v1
  2. kind: Endpoints
  3. metadata:
  4. name: plat-dev
  5. namespace: default
  6. subsets:
  7. - addresses:
  8. - ip: "10.5.10.109"
  9. - ip: "10.5.10.110"
  10. ports:
  11. - port: 3306
  12. name: port

nodes(“default/plat-dev:port”) 调用会得到如下的返回值:

  1. {
  2. {
  3. host="10.5.10.109",
  4. port= 3306,
  5. weight= 50,
  6. },
  7. {
  8. host="10.5.10.110",
  9. port= 3306,
  10. weight= 50,
  11. },
  12. }

多集群模式 Kubernetes 服务发现的配置格式

多集群模式 Kubernetes 服务发现的完整配置如下:

  1. discovery:
  2. kubernetes:
  3. - id: release # a custom name refer to the cluster, pattern ^[a-z0-9]{1,8}
  4. service:
  5. # apiserver schema, options [http, https]
  6. schema: https #default https
  7. # apiserver host, options [ipv4, ipv6, domain, environment variable]
  8. host: "1.cluster.com"
  9. # apiserver port, options [port number, environment variable]
  10. port: "6443"
  11. client:
  12. # serviceaccount token or token_file
  13. token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  14. #token: |-
  15. # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif
  16. # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI
  17. default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0
  18. # kubernetes discovery support namespace_selector
  19. # you can use one of [equal, not_equal, match, not_match] filter namespace
  20. namespace_selector:
  21. # only save endpoints with namespace equal default
  22. equal: default
  23. # only save endpoints with namespace not equal default
  24. #not_equal: default
  25. # only save endpoints with namespace match one of [default, ^my-[a-z]+$]
  26. #match:
  27. #- default
  28. #- ^my-[a-z]+$
  29. # only save endpoints with namespace not match one of [default, ^my-[a-z]+$]
  30. #not_match:
  31. #- default
  32. #- ^my-[a-z]+$
  33. # kubernetes discovery support label_selector
  34. # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
  35. label_selector: |-
  36. first="a",second="b"
  37. # reserved lua shared memory size,1m memory can store about 1000 pieces of endpoint
  38. shared_size: 1m #default 1m
  39. # if watch_endpoint_slices setting true, watch apiserver with endpointslices instead of endpoints
  40. watch_endpoint_slices: false #default false

多集群模式 Kubernetes 服务发现没有为 serviceclient 域填充默认值,你需要根据集群配置情况自行填充。

多集群模式 Kubernetes 服务发现的查询接口

多集群模式 Kubernetes 服务发现遵循 APISIX Discovery 规范 提供节点查询接口。

函数: nodes(service_name)

说明: service_name 必须满足格式:[id]/[namespace]/[name]:[portName]

  • id: Kubernetes 服务发现配置中定义的集群 id 值

  • namespace: Endpoints 所在的命名空间

  • name: Endpoints 的资源名

  • portName: Endpoints 定义包含的 ports.name 值,如果 Endpoints 没有定义 ports.name,请依次使用 targetPort, port 代替。设置了 ports.name 的情况下,不能使用后两者。

返回值: 以如下 Endpoints 为例:

  1. apiVersion: v1
  2. kind: Endpoints
  3. metadata:
  4. name: plat-dev
  5. namespace: default
  6. subsets:
  7. - addresses:
  8. - ip: "10.5.10.109"
  9. - ip: "10.5.10.110"
  10. ports:
  11. - port: 3306
  12. name: port

nodes(“release/default/plat-dev:port”) 调用会得到如下的返回值:

  1. {
  2. {
  3. host="10.5.10.109",
  4. port= 3306,
  5. weight= 50,
  6. },
  7. {
  8. host="10.5.10.110",
  9. port= 3306,
  10. weight= 50,
  11. },
  12. }

Q&A

Q: 为什么只支持配置 token 来访问 Kubernetes APIServer?

A: 一般情况下,我们有三种方式可以完成与 Kubernetes APIServer 的认证:

  • mTLS
  • Token
  • Basic authentication

因为 lua-resty-http 目前不支持 mTLS, Basic authentication 不被推荐使用,所以当前只实现了 Token 认证方式。

Q: APISIX 继承了 NGINX 的多进程模型,是否意味着每个 APISIX 工作进程都会监听 Kubernetes Endpoints?

A: Kubernetes 服务发现只使用特权进程监听 Kubernetes Endpoints,然后将其值存储到 ngx.shared.DICT 中,工作进程通过查询 ngx.shared.DICT 来获取结果。

Q: ServiceAccount 需要的权限有哪些?

A: ServiceAccount 需要集群级 [ get,list,watch ] endpoints 资源的的权限,其声明式定义如下:

  1. kind: ServiceAccount
  2. apiVersion: v1
  3. metadata:
  4. name: apisix-test
  5. namespace: default
  6. ---
  7. kind: ClusterRole
  8. apiVersion: rbac.authorization.k8s.io/v1
  9. metadata:
  10. name: apisix-test
  11. rules:
  12. - apiGroups: [ "" ]
  13. resources: [ endpoints,endpointslices ]
  14. verbs: [ get,list,watch ]
  15. ---
  16. apiVersion: rbac.authorization.k8s.io/v1
  17. kind: ClusterRoleBinding
  18. metadata:
  19. name: apisix-test
  20. roleRef:
  21. apiGroup: rbac.authorization.k8s.io
  22. kind: ClusterRole
  23. name: apisix-test
  24. subjects:
  25. - kind: ServiceAccount
  26. name: apisix-test
  27. namespace: default

Q: 怎样获取指定 ServiceAccount 的 Token 值?

A: 假定你指定的 ServiceAccount 资源名为“kubernetes-discovery“, 命名空间为“apisix”, 请按如下步骤获取其 Token 值。

  1. 获取 Secret 资源名。执行以下命令,输出的第一列内容就是目标 Secret 资源名:

    1. kubectl -n apisix get secrets | grep kubernetes-discovery
  2. 获取 Token 值。假定你获取到的 Secret 资源名为 “kubernetes-discovery-token-c64cv”, 执行以下命令,输出内容就是目标 Token 值:

    1. kubectl -n apisix get secret kubernetes-discovery-token-c64cv -o jsonpath={.data.token} | base64 -d

调试 API

它还提供了用于调试的控制 api。

内存 Dump API

  1. GET /v1/discovery/kubernetes/dump

例子

  1. # curl http://127.0.0.1:9090/v1/discovery/kubernetes/dump | jq
  2. {
  3. "endpoints": [
  4. {
  5. "endpoints": [
  6. {
  7. "value": "{\"https\":[{\"host\":\"172.18.164.170\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":6443,\"weight\":50}]}",
  8. "name": "default/kubernetes"
  9. },
  10. {
  11. "value": "{\"metrics\":[{\"host\":\"172.18.164.170\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":2379,\"weight\":50}]}",
  12. "name": "kube-system/etcd"
  13. },
  14. {
  15. "value": "{\"http-85\":[{\"host\":\"172.64.89.2\",\"port\":85,\"weight\":50}]}",
  16. "name": "test-ws/testing"
  17. }
  18. ],
  19. "id": "first"
  20. }
  21. ],
  22. "config": [
  23. {
  24. "default_weight": 50,
  25. "id": "first",
  26. "client": {
  27. "token": "xxx"
  28. },
  29. "service": {
  30. "host": "172.18.164.170",
  31. "port": "6443",
  32. "schema": "https"
  33. },
  34. "shared_size": "1m"
  35. }
  36. ]
  37. }