Debug requests

Debug requests

Kong Gateway admins can debug requests by collecting timing information about a given request, on demand. Request debugging is triggered using a secure token and the resulting data is returned in a response header named X-Kong-Request-Debug-Output.

Request debugging provides the following insights:

Time spent in various Kong Gateway components, such as plugins, DNS resolution, and load balancing.
Contextual information, such as the domain name tried during these processes.

Note: This feature is meant for live debugging. The JSON schema of the header containing the timing should never be considered static and is always subject to change.

Enable request debugging

Request debugging is enabled by default and has the following configurations in kong.conf:

request_debug = on | off # enable or disable request debugging
request_debug_token <token> # Set debug token explicitly. Otherwise, it will be generated randomly when Kong starts, restarts, and reloads.

The debug token (X-Kong-Request-Debug-Token) is used to authenticate the client and send the debug request, preventing abuse.

You can find the debug token in the following locations:

Kong Gateway error log: The debug token is logged in the error log (notice level) when Kong Gateway starts, restarts, or reloads. The log line will have the [request-debug] prefix to aid in searching.
Filesystem: The debug token is also stored in a file at {prefix}/.request_debug_token and updated when Kong Gateway starts, restarts, or reloads.

Debug request configuration

To debug a request, add the following request headers:

At a minimum, you should set the X-Kong-Request-Debug header.
If the requests originate from loopback addresses, the X-Kong-Request-Debug-Token header also needs to be set.

X-Kong-Request-Debug header

If the X-Kong-Request-Debug header is set to *, timing information will be collected and exported for the current request.

In Kong Gateway Enterprise, you can also specify a list of filters, separated by commas, to filter the scope of the time information that is collected. Each filters specifies which phase to collect timing information from. The following filters are supported:

rewrite
access
balancer
response
header_filter
body_filter
upstream
log

If this header isn’t present or contains an unknown value, timing information will not be collected for the current request.

X-Kong-Request-Debug-Log header

If the X-Kong-Request-Debug-Token header is set to true, timing information will also be logged in the Kong Gateway error log with a log level of notice. By default, the X-Kong-Request-Debug-Token header is set to false. The log line will have the [request-debug] prefix to aid in searching.

Debug request example

The following is an example debug request:

curl http://localhost:8000/example \
    -H "X-Kong-Request-Debug: *" \
    -H "X-Kong-Request-Debug-Token: xxxxxx"

Here’s an example of the output of the response header:

{
    "request_id": "2effa21fb2d36d31f80ed02635cde86b",
    "workspace_id": "7b7f79f2-8d52-470c-a307-e76f986041cf",
    "child": {
        "rewrite": {
            "total_time": 0
        },
        "access": {
            "child": {
                "dns": {
                    "child": {
                        "example.host": {
                            "child": {
                                "resolve": {
                                    "total_time": 0,
                                    "cache_hit": true
                                }
                            },
                            "total_time": 0
                        }
                    },
                    "total_time": 0
                }
            },
            "total_time": 1
        },
        "header_filter": {
            "total_time": 0
        },
        "balancer": {
            "total_time": 0
        },
        "upstream": {
            "total_time": 100,
            "child": {
                "time_to_first_byte": {
                    "total_time": 20
                },
                "streaming": {
                    "Total_time": 80
                }
            }
        }
    }
}

If you analyze the example debug output, you can see that:

The unit of total_time is millisecond.
The DNS resolution for example.host was cached, which is why it’s so fast in the example.
The upstream took 100ms in this request.
- The elapsed time from Kong Gateway sending the request to the upstream to Kong Gateway receiving the first byte is 20ms.
- The elapsed time from the first byte to the last byte from the upstream is 80ms.

You can also filter the debug output:

curl http://localhost:8000/example \
    -H "X-Kong-Request-Debug: upstream" \
    -H "X-Kong-Request-Debug-Token: xxxxxx"

This will return something like the following:

{
  "request_id": "a1a1530f8ddb6f6f2462916ae002b715",
  "child": {
    "upstream": {
      "total_time": 363,
      "child": {
        "time_to_first_byte": {
          "total_time": 363
        }
      }
    }
  }
}

Truncation for large debugging output

The downstream system may impose a size restriction on response headers, leading Kong Gateway to truncate the X-Kong-Request-Output if it exceeds 2KB. This truncated output will be unconditionally logged in the error_log.

curl http://localhost:8000/large_debugging_output \
    -H "X-Kong-Request-Debug: *" \
    -H "X-Kong-Request-Debug-Token: xxxxxx"