Kubernetes

Summary

The Kubernetes service discovery List-Watch real-time changes of Endpoints resources, then store theirs value into ngx.shared.DICT.

Discovery also provides a node query interface in accordance with the APISIX Discovery Specification.

How To Use

Kubernetes service discovery both support single-cluster and multi-cluster modes, applicable to the case where the service is distributed in single or multiple Kubernetes clusters.

Single-Cluster Mode Configuration

A detailed configuration for single-cluster mode Kubernetes service discovery is as follows:

  1. discovery:
  2.   kubernetes:
  3.     service:
  4.       # apiserver schema, options [http, https]
  5.       schema: https #default https
  7.       # apiserver host, options [ipv4, ipv6, domain, environment variable]
  8.       host: ${KUBERNETES_SERVICE_HOST} #default ${KUBERNETES_SERVICE_HOST}
  10.       # apiserver port, options [port number, environment variable]
  11.       port: ${KUBERNETES_SERVICE_PORT}  #default ${KUBERNETES_SERVICE_PORT}
  13.     client:
  14.       # serviceaccount token or token_file
  15.       token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  17.       #token: |-
  18.        # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif
  19.        # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI
  21.     default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0
  23.     # kubernetes discovery support namespace_selector
  24.     # you can use one of [equal, not_equal, match, not_match] filter namespace
  25.     namespace_selector:
  26.       # only save endpoints with namespace equal default
  27.       equal: default
  29.       # only save endpoints with namespace not equal default
  30.       #not_equal: default
  32.       # only save endpoints with namespace match one of [default, ^my-[a-z]+$]
  33.       #match:
  34.        #- default
  35.        #- ^my-[a-z]+$
  37.       # only save endpoints with namespace not match one of [default, ^my-[a-z]+$ ]
  38.       #not_match:
  39.        #- default
  40.        #- ^my-[a-z]+$
  42.     # kubernetes discovery support label_selector
  43.     # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
  44.     label_selector: |-
  45.       first="a",second="b"
  47.     # reserved lua shared memory size,1m memory can store about 1000 pieces of endpoint
  48.     shared_size: 1m #default 1m
  50.     # if watch_endpoint_slices setting true, watch apiserver with endpointslices instead of endpoints
  51.     watch_endpoint_slices: false #default false

If the Kubernetes service discovery runs inside a pod, you can use minimal configuration:

  1. discovery:
  2.   kubernetes: { }

If the Kubernetes service discovery runs outside a pod, you need to create or select a specified ServiceAccount, then get its token value, and use following configuration:

  1. discovery:
  2.   kubernetes:
  3.     service:
  4.       schema: https
  5.       host: # enter apiserver host value here
  6.       port: # enter apiserver port value here
  7.     client:
  8.       token: # enter serviceaccount token value here
  9.       #token_file: # enter file path here

Single-Cluster Mode Query Interface

The Kubernetes service discovery provides a query interface in accordance with the APISIX Discovery Specification.

function: nodes(service_name)

description: nodes() function attempts to look up the ngx.shared.DICT for nodes corresponding to servicename, \ service_name should match pattern: [namespace]/[name]:[portName]_

  • namespace: The namespace where the Kubernetes endpoints is located

  • name: The name of the Kubernetes endpoints

  • portName: The ports.name value in the Kubernetes endpoints, if there is no ports.name, use targetPort, port instead. If ports.name exists, then port number cannot be used.

return value: if the Kubernetes endpoints value is as follows:

  1. apiVersion: v1
  2. kind: Endpoints
  3. metadata:
  4.   name: plat-dev
  5.   namespace: default
  6. subsets:
  7.   - addresses:
  8.       - ip: "10.5.10.109"
  9.       - ip: "10.5.10.110"
  10.     ports:
  11.       - port: 3306
  12.         name: port

a nodes(“default/plat-dev:port”) call will get follow result:

  1.  {
  2.      {
  3.          host="10.5.10.109",
  4.          port= 3306,
  5.          weight= 50,
  6.      },
  7.      {
  8.          host="10.5.10.110",
  9.          port= 3306,
  10.          weight= 50,
  11.      },
  12.  }

Multi-Cluster Mode Configuration

A detailed configuration for multi-cluster mode Kubernetes service discovery is as follows:

  1. discovery:
  2.   kubernetes:
  3.   - id: release  # a custom name refer to the cluster, pattern ^[a-z0-9]{1,8}
  4.     service:
  5.       # apiserver schema, options [http, https]
  6.       schema: https #default https
  8.       # apiserver host, options [ipv4, ipv6, domain, environment variable]
  9.       host: "1.cluster.com"
  11.       # apiserver port, options [port number, environment variable]
  12.       port: "6443"
  14.     client:
  15.       # serviceaccount token or token_file
  16.       token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  18.       #token: |-
  19.        # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif
  20.        # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI
  22.     default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0
  24.     # kubernetes discovery support namespace_selector
  25.     # you can use one of [equal, not_equal, match, not_match] filter namespace
  26.     namespace_selector:
  27.       # only save endpoints with namespace equal default
  28.       equal: default
  30.       # only save endpoints with namespace not equal default
  31.       #not_equal: default
  33.       # only save endpoints with namespace match one of [default, ^my-[a-z]+$]
  34.       #match:
  35.        #- default
  36.        #- ^my-[a-z]+$
  38.       # only save endpoints with namespace not match one of [default, ^my-[a-z]+$]
  39.       #not_match:
  40.        #- default
  41.        #- ^my-[a-z]+$
  43.     # kubernetes discovery support label_selector
  44.     # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
  45.     label_selector: |-
  46.       first="a",second="b"
  48.     # reserved lua shared memory size,1m memory can store about 1000 pieces of endpoint
  49.     shared_size: 1m #default 1m
  51.     # if watch_endpoint_slices setting true, watch apiserver with endpointslices instead of endpoints
  52.     watch_endpoint_slices: false #default false

Multi-Kubernetes service discovery does not fill default values for service and client fields, you need to fill them according to the cluster configuration.

Multi-Cluster Mode Query Interface

The Kubernetes service discovery provides a query interface in accordance with the APISIX Discovery Specification.

function: nodes(service_name)

description: nodes() function attempts to look up the ngx.shared.DICT for nodes corresponding to servicename, \ service_name should match pattern: [id]/[namespace]/[name]:[portName]_

  • id: value defined in service discovery configuration

  • namespace: The namespace where the Kubernetes endpoints is located

  • name: The name of the Kubernetes endpoints

  • portName: The ports.name value in the Kubernetes endpoints, if there is no ports.name, use targetPort, port instead. If ports.name exists, then port number cannot be used.

return value: if the Kubernetes endpoints value is as follows:

  1. apiVersion: v1
  2. kind: Endpoints
  3. metadata:
  4.   name: plat-dev
  5.   namespace: default
  6. subsets:
  7.   - addresses:
  8.       - ip: "10.5.10.109"
  9.       - ip: "10.5.10.110"
  10.     ports:
  11.       - port: 3306
  12.         name: port

a nodes(“release/default/plat-dev:port”) call will get follow result:

  1.  {
  2.      {
  3.          host="10.5.10.109",
  4.          port= 3306,
  5.          weight= 50,
  6.      },
  7.      {
  8.          host="10.5.10.110",
  9.          port= 3306,
  10.          weight= 50,
  11.      },
  12.  }

Q&A

Q: Why only support configuration token to access Kubernetes APIServer?

A: Usually, we will use three ways to complete the authentication of Kubernetes APIServer:

  • mTLS
  • Token
  • Basic authentication

Because lua-resty-http does not currently support mTLS, and basic authentication is not recommended, so currently only the token authentication method is implemented.

Q: APISIX inherits Nginx’s multiple process model, does it mean that each nginx worker process will List-Watch kubernetes endpoints resources?

A: The Kubernetes service discovery only uses privileged processes to List-Watch Kubernetes endpoints resources, then store theirs value into ngx.shared.DICT, worker processes get results by querying ngx.shared.DICT.

Q: What permissions do ServiceAccount require?

A: ServiceAccount requires the permissions of cluster-level [ get, list, watch ] endpoints resources, the declarative definition is as follows:

  1. kind: ServiceAccount
  2. apiVersion: v1
  3. metadata:
  4.  name: apisix-test
  5.  namespace: default
  6. ---
  8. kind: ClusterRole
  9. apiVersion: rbac.authorization.k8s.io/v1
  10. metadata:
  11.  name: apisix-test
  12. rules:
  13. - apiGroups: [ "" ]
  14.   resources: [ endpoints,endpointslices ]
  15.   verbs: [ get,list,watch ]
  16. ---
  18. apiVersion: rbac.authorization.k8s.io/v1
  19. kind: ClusterRoleBinding
  20. metadata:
  21.  name: apisix-test
  22. roleRef:
  23.  apiGroup: rbac.authorization.k8s.io
  24.  kind: ClusterRole
  25.  name: apisix-test
  26. subjects:
  27.  - kind: ServiceAccount
  28.    name: apisix-test
  29.    namespace: default

Q: How to get ServiceAccount token value?

A: Assume your ServiceAccount located in namespace apisix and name is Kubernetes-discovery, you can use the following steps to get token value.

  1. Get secret name. You can execute the following command, the output of the first column is the secret name we want:

    1. kubectl -n apisix get secrets | grep kubernetes-discovery

  2. Get token value. Assume secret resources name is kubernetes-discovery-token-c64cv, you can execute the following command, the output is the service account token value we want:

    1. kubectl -n apisix get secret kubernetes-discovery-token-c64cv -o jsonpath={.data.token} | base64 -d

Debugging API

It also offers control api for debugging.

Memory Dump API

To query/list the nodes discoverd by kubernetes discovery, you can query the /v1/discovery/kubernetes/dump control API endpoint like so:

  1. GET /v1/discovery/kubernetes/dump

Which will yield the following response:

  1. {
  2.   "endpoints": [
  3.     {
  4.       "endpoints": [
  5.         {
  6.           "value": "{\"https\":[{\"host\":\"172.18.164.170\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":6443,\"weight\":50}]}",
  7.           "name": "default/kubernetes"
  8.         },
  9.         {
  10.           "value": "{\"metrics\":[{\"host\":\"172.18.164.170\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":2379,\"weight\":50}]}",
  11.           "name": "kube-system/etcd"
  12.         },
  13.         {
  14.           "value": "{\"http-85\":[{\"host\":\"172.64.89.2\",\"port\":85,\"weight\":50}]}",
  15.           "name": "test-ws/testing"
  16.         }
  17.       ],
  18.       "id": "first"
  19.     }
  20.   ],
  21.   "config": [
  22.     {
  23.       "default_weight": 50,
  24.       "id": "first",
  25.       "client": {
  26.         "token": "xxx"
  27.       },
  28.       "service": {
  29.         "host": "172.18.164.170",
  30.         "port": "6443",
  31.         "schema": "https"
  32.       },
  33.       "shared_size": "1m"
  34.     }
  35.   ]
  36. }