Kubernetes

Summary

The Kubernetes service discovery List-Watch real-time changes of Endpoints resources, then store theirs value into ngx.shared.kubernetes \ Discovery also provides a query interface in accordance with the APISIX Discovery Specification

Configuration

A detailed configuration for the kubernetes service discovery is as follows:

  1. discovery:
  2.   kubernetes:
  3.     service:
  4.       # apiserver schema, options [http, https]
  5.       schema: https #default https
  7.       # apiserver host, options [ipv4, ipv6, domain, environment variable]
  8.       host: ${KUBERNETES_SERVICE_HOST} #default ${KUBERNETES_SERVICE_HOST}
  10.       # apiserver port, options [port number, environment variable]
  11.       port: ${KUBERNETES_SERVICE_PORT}  #default ${KUBERNETES_SERVICE_PORT}
  13.     client:
  14.       # serviceaccount token or token_file
  15.       token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  17.       #token: |-
  18.        # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif
  19.        # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI
  21.     # kubernetes discovery plugin support use namespace_selector
  22.     # you can use one of [equal, not_equal, match, not_match] filter namespace
  23.     namespace_selector:
  24.       # only save endpoints with namespace equal default
  25.       equal: default
  27.       # only save endpoints with namespace not equal default
  28.       #not_equal: default
  30.       # only save endpoints with namespace match one of [default, ^my-[a-z]+$]
  31.       #match:
  32.        #- default
  33.        #- ^my-[a-z]+$
  35.       # only save endpoints with namespace not match one of [default, ^my-[a-z]+$ ]
  36.       #not_match:
  37.        #- default
  38.        #- ^my-[a-z]+$
  40.     # kubernetes discovery plugin support use label_selector
  41.     # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
  42.     label_selector: |-
  43.       first="a",second="b"

If the kubernetes service discovery runs inside a pod, you can use minimal configuration:

  1. discovery:
  2.   kubernetes: { }

If the kubernetes service discovery runs outside a pod, you need to create or select a specified ServiceAccount, then get its token value, and use following configuration:

  1. discovery:
  2.   kubernetes:
  3.     service:
  4.       schema: https
  5.       host: # enter apiserver host value here
  6.       port: # enter apiserver port value here
  7.     client:
  8.       token: # enter serviceaccount token value here
  9.       #token_file: # enter file path here

Interface

the kubernetes service discovery provides a query interface in accordance with the APISIX Discovery Specification

function: \ nodes(service_name)

description: \ nodes() function attempts to look up the ngx.shared.kubernetes for nodes corresponding to servicename, \ service_name should match pattern: [namespace]/[name]:[portName]_

  • namespace: The namespace where the kubernetes endpoints is located

  • name: The name of the kubernetes endpoints

  • portName: The portName of the kubernetes endpoints, if there is no portName, use targetPort, port instead

return value: \ if the kubernetes endpoints value is as follows:

  1. apiVersion: v1
  2. kind: Endpoints
  3. metadata:
  4.   name: plat-dev
  5.   namespace: default
  6. subsets:
  7.   - addresses:
  8.       - ip: "10.5.10.109"
  9.       - ip: "10.5.10.110"
  10.     ports:
  11.       - port: 3306

a nodes(“default/plat-dev:3306”) call will get follow result:

  1.  {
  2.      {
  3.          host="10.5.10.109",
  4.          port= 3306,
  5.          weight= 50,
  6.      },
  7.      {
  8.          host="10.5.10.110",
  9.          port= 3306,
  10.          weight= 50,
  11.      },
  12.  }

Q&A

Q: Why only support configuration token to access Kubernetes APIServer \ A: Usually, we will use three ways to complete the authentication of Kubernetes APIServer:

  • mTLS
  • token
  • basic authentication

Because lua-resty-http does not currently support mTLS, and basic authentication is not recommended,\ So currently only the token authentication method is implemented

Q: APISIX inherits Nginx’s multiple process model, does it mean that each nginx worker process will List-Watch kubernetes endpoints resources \ A: The kubernetes service discovery only uses privileged processes to List-Watch kubernetes endpoints resources, then store theirs value \ into ngx.shared.kubernetes, worker processes get results by querying ngx.shared.kubernetes

Q: How to get ServiceAccount token value \ A: Assume your ServiceAccount located in namespace apisix and name is kubernetes-discovery, you can use the following steps to get token value

  1. Get secret name: \ you can execute the following command, the output of the first column is the secret name we want
  1. kubectl -n apisix get secrets | grep kubernetes-discovery
  1. Get token value: \ assume secret resources name is kubernetes-discovery-token-c64cv, you can execute the following command, the output is the service account token value we want
  1. kubectl -n apisix get secret kubernetes-discovery-token-c64cv -o jsonpath={.data.token} | base64 -d