Monitor traffic rate limit data

This topic describes Consul functionality that enables you to monitor read and write request operations taking place in your network. Use the functionality to help you understand normal workloads and set safe limits on the number of requests Consul client agents and services can make to Consul servers.

Access rate limit logs

Consul prints a log line for each rate limit request. The log provides the necessary information for identifying the source of the request and the configured limit. The log provides the information necessary for identifying the source of the request and the configured limit. Consul prints the log DEBUG log level and can drop the log to avoid affecting the server health. Dropping a log line increments the rpc.rate_limit.log_dropped metric.

The following example log shows that RPC request from 127.0.0.1:53562 to KVS.Apply exceeded the limit:

  1. 2023-02-17T10:01:15.565-0500 [DEBUG] agent.server.rpc-rate-limit: RPC 
  2. exceeded allowed rate limit: rpc=KVS.Apply source_addr=127.0.0.1:53562 
  3. limit_type=global/write limit_enforced=false

Refer to log_file for information about where to retrieve log files.

Review rate limit metrics

Consul captures the following metrics associated with rate limits:

  • Type of limit
  • Operation
  • Rate limit mode

Call the /agent/metrics API endpoint to view the metrics associated with rate limits. Refer to View Metrics for API usage information. In the following example, Consul dropped a call to the consul service because it exceeded the limit by one call:

  1. $ curl http://127.0.0.1:8500/v1/agent/metrics
  2. {
  3.   . . .  
  4.   "Counters": [
  5.     {
  6.       "Name": "consul.rpc.rate_limit.exceeded",
  7.       "Count": 1,
  8.       "Sum": 1,
  9.       "Min": 1,
  10.       "Max": 1,
  11.       "Mean": 1,
  12.       "Stddev": 0,
  13.       "Labels": {
  14.         "service": "consul"
  15.       }
  16.     },
  17.     {
  18.       "Name": "consul.rpc.rate_limit.log_dropped",
  19.       "Count": 1,
  20.       "Sum": 1,
  21.       "Min": 1,
  22.       "Max": 1,
  23.       "Mean": 1,
  24.       "Stddev": 0,
  25.       "Labels": {}
  26.     }
  27.   ],
  28.   . . .
  29. }

Refer to Telemetry for additional information.

Request denials

When an HTTP request is denied for rate limiting reason, Consul returns one of the following errors:

  • 429 Resource Exhausted: Indicates that a server is not able to perform the request but that another server could potentially fulfill it. This error is most common on stale reads because any server may fulfill stale read requests. To resolve this type of error, we recommend immediately retrying the request to another server. If the request came from a Consul client agent, the agent automatically retries the request up to the limit set in the rpc_hold_timeout configuration .

  • 503 Service Unavailable: Indicates that server is unable to perform the request and that no other server can fulfill the request, either. This usually occurs on consistent reads or for writes. In this case we recommend retrying according to an exponential backoff schedule. If the request came from a Consul client agent, the agent automatically retries the request according to the rpc_hold_timeout configuration.

Refer to Rate limit reached on the server for additional information.