API Rate Limiting

If your Spinnaker deployment interacts with API clients, the API rate limiter can help keep it reliable during traffic spikes and protect it from rogue clients.

If your Spinnaker deployment has API clients interacting with it, enabling and knowing how to operate the API rate limiter can help keep Spinnaker reliable through spikes of heavy traffic or rogue clients.

Intro

The API Rate Limiter currently supports limiting individual authenticated and anonymous principals, bucketing requests into windows that are refreshed every interval as well as global & per-principal learning mode. Both the capacity, as well as the window size, are configurable globally and per-principal.

Some metrics are generated by the rate limiter:

  • rateLimit.throttling: Counter. Number of enforced throttled requests.
  • rateLimit.throttlingLearning: Counter. Number of un-enforced throttled requests.
  • rateLimit.principal.throttled: Counter. A tagged metric recording the number of throttled requests per-principal. Tag key is principal.
  • rateLimit.principal.remaining: Gauge. A tagged metric recording the number of remaining requests per-principal. Tag key is principal.

Combined with controller.invocations, these metrics can give you a good sense of your on-going rate limiting configuration.

In the case of failures (Redis unavailable) or configuration errors (principal included in both enforcing and ignoring lists), a request will gracefully fallback to learning mode.

Enabling in gate.yml

The rate limiter requires Redis to be available for Gate.

  1. rateLimit:
  2.   enabled: true
  3.   learning: true
  4.   redis:
  5.     enabled: true
  6.   # The rate (in seconds) that the bucket will be replenished
  7.   rateSeconds: 10
  8.   # The number of allowed requests in the window of rateSeconds
  9.   capacity: 500
  10.   # Specific principals can be given different capacities
  11.   capacityByPrincipal:
  12.   - principal: anonymous
  13.     override: 1000
  14.   # Similarly, rateSeconds can be overridden per-principal
  15.   rateSecondsByPrincipal:
  16.   - principal: anonymous
  17.     override: 5
  18.   # A list of principals that are being enforced. Handy for cases where you want
  19.   # to incrementally enable the rate limiter
  20.   enforcing:
  21.   - example@example.com
  22.   # A list of principals that are in learning mode. This can be useful if you
  23.   # want to give some principals unlimited power. Reconsider doing this :)
  24.   ignoring:
  25.   - unmetered-example@example.com

Integrating clients

The rate limiter exposes four HTTP Headers on each request that a client can ingest to play nicely with the rate limiter:

  • X-RateLimit-Capacity: Int. Total capacity assigned to the principal.
  • X-RateLimit-Remaining: Int. Total number of requests left before reset.
  • X-RateLimit-Reset: Date. When the bucket will be replenished.
  • X-RateLimit-Learning: Bool. Whether or not the rate limiter is enforcing the principal.

Anonymous clients

If your Spinnaker deployment allows anonymous API access, by default all requests will be bucketed into the same rate limit window. With a little trust, you can allow your users to assign themselves their own anonymous rate limit bucket by sending a X-RateLimit-App header, where the value is the name of their application.

Dynamic configuration

Re-deploying gate to respond to real-world events is non-ideal. Redis has the capability to override any configuration defined by the static configuration. First, a list of key patterns:

  • rateLimit* - All rate limiter-related keys.
  • rateLimit:* - Principal rate limiter.
  • rateLimit:learning - Global learning flag.
  • rateLimit:enforcing - A list of principals being enforced.
  • rateLimit:ignoring - A list of principals in learning mode.
  • rateLimit:capacity:* - Per-principal capacity.
  • rateLimit:rateSeconds:* - Per-principal rateSeconds.

Examples

  1. $ redis-cli
  3. # Set a new capacity
  4. 127.0.0.1:6379> set rateLimit:capacity:chaosmonkey@example.com 2000
  6. # Add the chaosmonkey@example.com principal to the enforcing list
  7. 127.0.0.1:6379> sadd rateLimit:enforcing chaosmonkey@example.com
  8. (integer) 1

Reference

Code backing the rate limiter can be found in the gate repository .

Last modified October 8, 2020: docs(headers): shorten linkTitle and description where applicable (68701bc)