Integration service discovery registry

Summary

When system traffic changes, the number of servers of the upstream service also increases or decreases, or the server needs to be replaced due to its hardware failure. If the gateway maintains upstream service information through configuration, the maintenance costs in the microservices architecture pattern are unpredictable. Furthermore, due to the untimely update of these information, will also bring a certain impact for the business, and the impact of human error operation can not be ignored. So it is very necessary for the gateway to automatically get the latest list of service instances through the service registry。As shown in the figure below：

discovery through service registry

  1. When the service starts, it will report some of its information, such as the service name, IP, port and other information to the registry. The services communicate with the registry using a mechanism such as a heartbeat, and if the registry and the service are unable to communicate for a long time, the instance will be cancel.When the service goes offline, the registry will delete the instance information.
  2. The gateway gets service instance information from the registry in near-real time.
  3. When the user requests the service through the gateway, the gateway selects one instance from the registry for proxy.

How to extend the discovery client?

Basic steps

It is very easy for APISIX to extend the discovery client, the basic steps are as follows

  1. Add the implementation of registry client in the ‘apisix/discovery/‘ directory;

  2. Implement the _M.init_worker() function for initialization and the _M.nodes(service_name) function for obtaining the list of service instance nodes;

  3. If you need the discovery module to export the debugging information online, implement the _M.dump_data() function;

  4. Convert the registry data into data in APISIX;

the example of Eureka

Implementation of Eureka client

First, create a directory eureka under apisix/discovery;

After that, add init.lua in the apisix/discovery/eureka directory;

Then implement the _M.init_worker() function for initialization and the _M.nodes(service_name) function for obtaining the list of service instance nodes in init.lua:

  1. local _M = {
  2.     version = 1.0,
  3. }
  6. function _M.nodes(service_name)
  7.     ... ...
  8. end
  11. function _M.init_worker()
  12.     ... ...
  13. end
  16. function _M.dump_data()
  17.     return {config = your_config, services = your_services, other = ... }
  18. end
  21. return _M

Finally, provide the schema for YAML configuration in the schema.lua under apisix/discovery/eureka.

How convert Eureka’s instance data to APISIX’s node?

Here’s an example of Eureka’s data：

  1. {
  2.   "applications": {
  3.       "application": [
  4.           {
  5.               "name": "USER-SERVICE",                 # service name
  6.               "instance": [
  7.                   {
  8.                       "instanceId": "192.168.1.100:8761",
  9.                       "hostName": "192.168.1.100",
  10.                       "app": "USER-SERVICE",          # service name
  11.                       "ipAddr": "192.168.1.100",      # IP address
  12.                       "status": "UP",
  13.                       "overriddenStatus": "UNKNOWN",
  14.                       "port": {
  15.                           "$": 8761,
  16.                           "@enabled": "true"
  17.                       },
  18.                       "securePort": {
  19.                           "$": 443,
  20.                           "@enabled": "false"
  21.                       },
  22.                       "metadata": {
  23.                           "management.port": "8761",
  24.                           "weight": 100               # Setting by 'eureka.instance.metadata-map.weight' of the spring boot application
  25.                       },
  26.                       "homePageUrl": "http://192.168.1.100:8761/",
  27.                       "statusPageUrl": "http://192.168.1.100:8761/actuator/info",
  28.                       "healthCheckUrl": "http://192.168.1.100:8761/actuator/health",
  29.                       ... ...
  30.                   }
  31.               ]
  32.           }
  33.       ]
  34.   }
  35. }

Deal with the Eureka’s instance data need the following steps :

  1. select the UP instance. When the value of overriddenStatus is “UP” or the value of overriddenStatus is “UNKNOWN” and the value of status is “UP”.
  2. Host. The ipAddr is the IP address of instance; and must be IPv4 or IPv6.
  3. Port. If the value of port["@enabled"] is equal to “true”, using the value of port["\$"], If the value of securePort["@enabled"] is equal to “true”, using the value of securePort["\$"].
  4. Weight. local weight = metadata.weight or local_conf.eureka.weight or 100

The result of this example is as follows:

  1. [
  2.   {
  3.     "host" : "192.168.1.100",
  4.     "port" : 8761,
  5.     "weight" : 100,
  6.     "metadata" : {
  7.       "management.port": "8761"
  8.     }
  9.   }
  10. ]

Configuration for discovery client

Initial service discovery

Add the following configuration to conf/config.yaml to add different service discovery clients for dynamic selection during use:

  1. discovery:
  2.   eureka:
  3.       ...

This name should be consistent with the file name of the implementation registry in the apisix/discovery/ directory.

The supported discovery client: Eureka.

Configuration for Eureka

Add following configuration in conf/config.yaml ：

  1. discovery:
  2.   eureka:
  3.     host:                            # it's possible to define multiple eureka hosts addresses of the same eureka cluster.
  4.       - "http://${username}:${password}@${eureka_host1}:${eureka_port1}"
  5.       - "http://${username}:${password}@${eureka_host2}:${eureka_port2}"
  6.     prefix: "/eureka/"
  7.     fetch_interval: 30               # 30s
  8.     weight: 100                      # default weight for node
  9.     timeout:
  10.       connect: 2000                  # 2000ms
  11.       send: 2000                     # 2000ms
  12.       read: 5000                     # 5000ms

Upstream setting

L7

Here is an example of routing a request with a URL of “/user/*“ to a service which named “user-service” and use eureka discovery client in the registry :

Integration service discovery registry - 图2note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

  1. admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
  1. $ curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -i -d '
  2. {
  3.     "uri": "/user/*",
  4.     "upstream": {
  5.         "service_name": "USER-SERVICE",
  6.         "type": "roundrobin",
  7.         "discovery_type": "eureka"
  8.     }
  9. }'
  11. HTTP/1.1 201 Created
  12. Date: Sat, 31 Aug 2019 01:17:15 GMT
  13. Content-Type: text/plain
  14. Transfer-Encoding: chunked
  15. Connection: keep-alive
  16. Server: APISIX web server
  18. {"node":{"value":{"uri":"\/user\/*","upstream": {"service_name": "USER-SERVICE", "type": "roundrobin", "discovery_type": "eureka"}},"createdIndex":61925,"key":"\/apisix\/routes\/1","modifiedIndex":61925}}

Because the upstream interface URL may have conflict, usually in the gateway by prefix to distinguish:

  1. $ curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -i -d '
  2. {
  3.     "uri": "/a/*",
  4.     "plugins": {
  5.         "proxy-rewrite" : {
  6.             "regex_uri": ["^/a/(.*)", "/${1}"]
  7.         }
  8.     },
  9.     "upstream": {
  10.         "service_name": "A-SERVICE",
  11.         "type": "roundrobin",
  12.         "discovery_type": "eureka"
  13.     }
  14. }'
  16. $ curl http://127.0.0.1:9180/apisix/admin/routes/2 -H "X-API-KEY: $admin_key" -X PUT -i -d '
  17. {
  18.     "uri": "/b/*",
  19.     "plugins": {
  20.         "proxy-rewrite" : {
  21.             "regex_uri": ["^/b/(.*)", "/${1}"]
  22.         }
  23.     },
  24.     "upstream": {
  25.         "service_name": "B-SERVICE",
  26.         "type": "roundrobin",
  27.         "discovery_type": "eureka"
  28.     }
  29. }'

Suppose both A-SERVICE and B-SERVICE provide a /test API. The above configuration allows access to A-SERVICE’s /test API through /a/test and B-SERVICE’s /test API through /b/test.

Notice：When configuring upstream.service_name, upstream.nodes will no longer take effect, but will be replaced by ‘nodes’ obtained from the registry.

L4

Eureka service discovery also supports use in L4, the configuration method is similar to L7.

  1. $ curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -i -d '
  2. {
  3.     "remote_addr": "127.0.0.1",
  4.     "upstream": {
  5.         "scheme": "tcp",
  6.         "discovery_type": "eureka",
  7.         "service_name": "APISIX-EUREKA",
  8.         "type": "roundrobin"
  9.     }
  10. }'
  11. HTTP/1.1 200 OK
  12. Date: Fri, 30 Dec 2022 03:52:19 GMT
  13. Content-Type: application/json
  14. Transfer-Encoding: chunked
  15. Connection: keep-alive
  16. Server: APISIX/3.0.0
  17. Access-Control-Allow-Origin: *
  18. Access-Control-Allow-Credentials: true
  19. Access-Control-Expose-Headers: *
  20. Access-Control-Max-Age: 3600
  21. X-API-VERSION: v3
  23. {"key":"\/apisix\/stream_routes\/1","value":{"remote_addr":"127.0.0.1","upstream":{"hash_on":"vars","type":"roundrobin","discovery_type":"eureka","scheme":"tcp","pass_host":"pass","service_name":"APISIX-EUREKA"},"id":"1","create_time":1672106762,"update_time":1672372339}}

Embedded control api for debugging

Sometimes we need the discovery client to export online data snapshot in memory when running for debugging, and if you implement the _M. dump_data() function:

  1. function _M.dump_data()
  2.     return {config = local_conf.discovery.eureka, services = applications}
  3. end

Then you can call its control api as below:

  1. GET /v1/discovery/{discovery_type}/dump

eg:

  1. curl http://127.0.0.1:9090/v1/discovery/eureka/dump