Introducing APISIX’s testing framework

APISIX uses a testing framework based on our fork of test-nginx: https://github.com/iresty/test-nginx. For details, you can check the documentation of this project.

If you want to test the CLI behavior of APISIX (./bin/apisix), you need to write a shell script in the t/cli directory to test it. You can refer to the existing test scripts for more details.

If you want to test the others, you need to write test code based on the framework.

Here, we briefly describe how to do simple testing based on this framework.

Test file

you need to write test cases in the t/ directory, in a corresponding .t file. Note that a single test file should not exceed 800 lines, and if it is too long, it needs to be divided by a suffix. For example:

  1. t/
  2. ├── admin
  3. │ ├── consumers.t
  4. │ ├── consumers2.t

Both consumers.t and consumers2.t contain tests for consumers in the Admin API.

Some of the test files start with this paragraph:

  1. add_block_preprocessor(sub {
  2.     my ($block) = @_;
  5.     if (! $block->request) {
  6.         $block->set_value("request", "GET /t");
  7.     }
  10.     if (! $block->no_error_log && ! $block->error_log) {
  11.         $block->set_value("no_error_log", "[error]\n[alert]");
  12.     }
  13. });

It means that all tests in this test file that do not define request are set to GET /t. The same is true for error_log.

Preparing the configuration

When testing a behavior, we need to prepare the configuration.

If the configuration is from etcd: We can set up specific configurations through the Admin API.

  1. === TEST 7: refer to empty nodes upstream
  2. --- config
  3.     location /t {
  4.         content_by_lua_block {
  5.             local core = require("apisix.core")
  6.             local t = require("lib.test_admin").test
  7.             local code, message = t('/apisix/admin/routes/1',
  8.                 ngx.HTTP_PUT,
  9.                 [[{
  10.                     "methods": ["GET"],
  11.                     "upstream_id": "1",
  12.                     "uri": "/index.html"
  13.                 }]]
  14.                 )
  17.             if code >= 300 then
  18.                 ngx.status = code
  19.                 ngx.print(message)
  20.                 return
  21.             end
  24.             ngx.say(message)
  25.         }
  26.     }
  27. --- request
  28. GET /t
  29. --- response_body
  30. passed

Then trigger it in a later test:

  1. === TEST 8: hit empty nodes upstream
  2. --- request
  3. GET /index.html
  4. --- error_code: 503
  5. --- error_log
  6. no valid upstream node

Preparing the upstream

To test the code, we need to provide a mock upstream.

For HTTP request, the upstream code is put in t/lib/server.lua. HTTP request with a given path will trigger the method in same name. For example, a call to /server_port will call the _M.server_port.

For TCP request, a dummy upstream is used:

  1. local sock = ngx.req.socket()
  2. local data = sock:receive("1")
  3. ngx.say("hello world")

If you want to custom the TCP upstream logic, you can use:

  1. --- stream_upstream_code
  2. local sock = ngx.req.socket()
  3. local data = sock:receive("1")
  4. ngx.sleep(0.2)
  5. ngx.say("hello world")

Send request

We can initiate a request with request and set the request headers with more_headers.

For example.

  1. --- request
  2. PUT /hello?xx=y&xx=z&&y=&&z
  3. body part of the request
  4. --- more_headers
  5. X-Req: foo
  6. X-Req: bar
  7. X-Resp: cat

Lua code can be used to send multiple requests.

One request after another:

  1.     location /t {
  2.         content_by_lua_block {
  3.             local http = require "resty.http"
  4.             local uri = "http://127.0.0.1:" .. ngx.var.server_port
  5.                         .. "/server_port"
  8.             local ports_count = {}
  9.             for i = 1, 12 do
  10.                 local httpc = http.new()
  11.                 local res, err = httpc:request_uri(uri, {method = "GET"})
  12.                 if not res then
  13.                     ngx.say(err)
  14.                     return
  15.                 end
  16.                 ports_count[res.body] = (ports_count[res.body] or 0) + 1
  17.             end

Sending multiple requests concurrently:

  1.     location /t {
  2.         content_by_lua_block {
  3.             local http = require "resty.http"
  4.             local uri = "http://127.0.0.1:" .. ngx.var.server_port
  5.                         .. "/server_port?var=2&var2="
  10.             local t = {}
  11.             local ports_count = {}
  12.             for i = 1, 180 do
  13.                 local th = assert(ngx.thread.spawn(function(i)
  14.                     local httpc = http.new()
  15.                     local res, err = httpc:request_uri(uri..i, {method = "GET"})
  16.                     if not res then
  17.                         ngx.log(ngx.ERR, err)
  18.                         return
  19.                     end
  20.                     ports_count[res.body] = (ports_count[res.body] or 0) + 1
  21.                 end, i))
  22.                 table.insert(t, th)
  23.             end
  24.             for i, th in ipairs(t) do
  25.                 ngx.thread.wait(th)
  26.             end

Send TCP request

We can use stream_request to send a TCP request, for example:

  1. --- stream_request
  2. hello

To send a TLS over TCP request, we can combine stream_tls_request with stream_sni:

  1. --- stream_tls_request
  2. mmm
  3. --- stream_sni: xx.com

Assertions

The following assertions are commonly used.

Check status (if not set, the framework will check if the request has 200 status code).

  1. --- error_code: 405

Check response headers.

  1. --- response_headers
  2. X-Resp: foo
  3. X-Req: foo, bar

Check response body.

  1. --- response_body
  2. [{"count":12, "port": "1982"}]

Check the TCP response.

When the request is sent via stream_request:

  1. --- stream_response
  2. receive stream response error: connection reset by peer

When the request is sent via stream_tls_request:

  1. --- response_body
  2. receive stream response error: connection reset by peer

Checking the error log (via grep error log with regular expression).

  1. --- grep_error_log eval
  2. qr/hash_on: header|chash_key: "custom-one"/
  3. --- grep_error_log_out
  4. hash_on: header
  5. chash_key: "custom-one"
  6. hash_on: header
  7. chash_key: "custom-one"
  8. hash_on: header
  9. chash_key: "custom-one"
  10. hash_on: header
  11. chash_key: "custom-one"

The default log level is info, but you can get the debug level log with -- log_level: debug.

Upstream

The test framework listens to multiple ports when it is started.

  • 1980/1981/1982/5044: HTTP upstream port
  • 1983: HTTPS upstream port
  • 1984: APISIX HTTP port. Can be used to verify HTTP related gateway logic, such as concurrent access to an API.
  • 1985: APISIX TCP port. Can be used to verify TCP related gateway logic, such as concurrent access to an API.
  • 1994: APISIX HTTPS port. Can be used to verify HTTPS related gateway logic, such as testing certificate matching logic.
  • 1995: TCP upstream port
  • 2005: APISIX TLS over TCP port. Can be used to verify TLS over TCP related gateway logic, such as concurrent access to an API.

The methods in t/lib/server.lua are executed when accessing the upstream port. _M.go is the entry point for this file. When the request accesses the upstream /xxx, the _M.xxx method is executed. For example, a request for /hello will execute _M.hello. This allows us to write methods inside t/lib/server.lua to emulate specific upstream logic, such as sending special responses.

Note that before adding new methods to t/lib/server.lua, make sure that you can reuse existing methods.

Run the test

Assume your current work directory is the root of the apisix source code.

  1. Install our fork of test-nginx to ../test-nginx.
  2. Run the test: prove -I. -I../test-nginx/inc -I../test-nginx/lib -r t/path/to/file.t.

Tips

Debugging test cases

The Nginx configuration and logs generated by the test cases are located in the t/servroot directory. The Nginx configuration template for testing is located in t/APISIX.pm.

Running only some test cases

Three notes can be used to control which parts of the tests are executed.

FIRST & LAST:

  1. === TEST 1: vars rule with ! (set)
  2. --- FIRST
  3. --- config
  4. ...
  5. --- response_body
  6. passed
  13. === TEST 2: vars rule with ! (hit)
  14. --- request
  15. GET /hello?name=jack&age=17
  16. --- LAST
  17. --- error_code: 403
  18. --- response_body
  19. Fault Injection!

ONLY:

  1. === TEST 1: list empty resources
  2. --- ONLY
  3. --- config
  4. ...
  5. --- response_body
  6. {"action":"get","count":0,"node":{"dir":true,"key":"/apisix/upstreams","nodes":{}}}

Executing Shell Commands

It is possible to execute shell commands while writing tests in test-nginx for APISIX. We expose this feature via exec code block. The stdout of the executed process can be captured via response_body code block and stderr (if any) can be captured by filtering error.log through grep_error_log. Here is an example:

  1. === TEST 1: check exec stdout
  2. --- exec
  3. echo hello world
  4. --- response_body
  5. hello world
  10. === TEST 2: when exec returns an error
  11. --- exec
  12. echxo hello world
  13. --- grep_error_log eval
  14. qr/failed to execute the script [ -~]*/
  15. --- grep_error_log_out
  16. failed to execute the script with status: 127, reason: exit, stderr: /bin/sh: 1: echxo: not found