Store Limits

The store_limits section in the configuration file (or the command line parameters -mc, -mm, etc..) allow you to configure the global limits.

These limits offer some upper bounds on the size of the storage. By multiplying the limits per channel with the maximum number of channels, you will get a total limit.

It is not the case, though, if you override limits of some channels. Indeed, it is possible to define specific limits per channel. Here is how:

  1. ...
  2. store_limits: {
  3.     # Override some global limits
  4.     max_channels: 10
  5.     max_msgs: 10000
  6.     max_bytes: 10MB
  7.     max_age: "1h"
  9.     # Per channel configuration.
  10.     # Can be channels, channels_limits, per_channel, per_channel_limits or ChannelsLimits
  11.     channels: {
  12.         "foo": {
  13.             # Possible options are the same than in the store_limits section, except
  14.             # for max_channels. Not all limits need to be specified.
  15.             max_msgs: 300
  16.             max_subs: 50
  17.         }
  18.         "bar": {
  19.             max_msgs:50
  20.             max_bytes:1KB
  21.         }
  22.         "baz": {
  23.             # Set to 0 for ignored (or unlimited)
  24.             max_msgs: 0
  25.             # Override with a lower limit
  26.             max_bytes: 1MB
  27.             # Override with a higher limit
  28.             max_age: "2h"
  29.         }
  30.         # When using partitioning, channels need to be listed.
  31.         # They don't have to override any limit.
  32.         "bozo": {}
  34.         # Wildcards are possible in configuration. This says that any channel
  35.         # that will start with "foo" but with at least 2 tokens, will be
  36.         # able to store 400 messages. Other limits are inherited from global.
  37.         "foo.>": {
  38.             max_msgs:400
  39.         }
  40.         # This one says that if the channel name starts with "foo.bar" but has
  41.         # at least one more token, the sever will hold it for 2 hours instead
  42.         # of one. The max number of messages is inherited from "foo.>", so the
  43.         # limit will be 400. All other limits are inherited from global.
  44.         "foo.bar.>": {
  45.             max_age: "2h"
  46.         }
  47.         # Delete channels with this prefix once they don't have any
  48.         # subscription and no new message for more than 1 hour
  49.         "temp.>": {
  50.             max_inactivity: "1h"
  51.         }
  52.     }
  53. }
  54. ...

Note that the number of defined channels cannot be greater than the stores’ maximum number of channels. This is true only for channels without wildcards.

Channels limits can override global limits by being either higher, lower or even set to unlimited.

An unlimited value applies to the specified limit, not to the whole channel.

That is, in the configuration above, baz has the maximum number of messages set to 0, which means ignored or unlimited. Yet, other limits such as max bytes, max age and max subscriptions (inherited in this case) still apply. What that means is that the store will not check the number of messages but still check the other limits.

For a truly unlimited channel all limits need to be set to 0.

Limits Inheritance

When starting the server from the command line, global limits that are not specified (configuration file or command line parameters) are inherited from default limits selected by the server.

Per-channel limits that are not explicitly configured inherit from the corresponding global limit (which can itself be inherited from default limit).

If a per-channel limit is set to 0 in the configuration file (or negative value programmatically), then it becomes unlimited, regardless of the corresponding global limit.

On startup the server displays the store limits. Notice the * at the right of a limit to indicate that the limit was inherited from the default store limits.

For channels that have been configured, their name is displayed and only the limits being specifically set are displayed to minimize the output.

Wildcards

Wildcards are allowed for channel configuration. Limits for foo.> will apply to any channel that starts with foo (but has at least one more token). If foo.bar.> is specified, it will inherit from foo.> and from global limits.

Below is what would be displayed with the above store limits configuration. Notice how foo.bar.> is indented compared to foo.> to show the inheritance.

  1. [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.16.0
  2. [INF] STREAM: ServerID: bFHdJP9hesjHIR0RheCl7W
  3. [INF] STREAM: Go version: go1.11.13
  4. [INF] STREAM: Git commit: [not set]
  5. [INF] Starting nats-server version 2.0.2
  6. [INF] Git commit [not set]
  7. [INF] Listening for client connections on 0.0.0.0:4222
  8. [INF] Server id is NDMRMUBKS37GDEPOZP4YVMTRLS6ROZS4O2JQVFOGDRJTGIY44OUV7ZSD
  9. [INF] Server is ready
  10. [INF] STREAM: Recovering the state...
  11. [INF] STREAM: No recovered state
  12. [INF] STREAM: Message store is MEMORY
  13. [INF] STREAM: ---------- Store Limits ----------
  14. [INF] STREAM: Channels:                   10
  15. [INF] STREAM: --------- Channels Limits --------
  16. [INF] STREAM:   Subscriptions:          1000 *
  17. [INF] STREAM:   Messages     :         10000
  18. [INF] STREAM:   Bytes        :      10.00 MB
  19. [INF] STREAM:   Age          :        1h0m0s
  20. [INF] STREAM:   Inactivity   :     unlimited *
  21. [INF] STREAM: -------- List of Channels ---------
  22. [INF] STREAM: baz
  23. [INF] STREAM:  |-> Messages             unlimited
  24. [INF] STREAM:  |-> Bytes                  1.00 MB
  25. [INF] STREAM:  |-> Age                     2h0m0s
  26. [INF] STREAM: bozo
  27. [INF] STREAM: temp.>
  28. [INF] STREAM:  |-> Inactivity              1h0m0s
  29. [INF] STREAM: foo.>
  30. [INF] STREAM:  |-> Messages                   400
  31. [INF] STREAM:  foo.bar.>
  32. [INF] STREAM:   |-> Age                    2h0m0s
  33. [INF] STREAM: bar
  34. [INF] STREAM:  |-> Messages                    50
  35. [INF] STREAM:  |-> Bytes                  1.00 KB
  36. [INF] STREAM: -----------------------------------

Limits are Retroactive

Suppose you have set a channel limit to hold at most 100 messages, and the channel currently holds 72 messages. The server is stopped and the message limit for this channel is lowered to 50 messages, then the server is restarted.

On startup, the server will apply the store limits, which means that this channel will now hold the maximum number of messages (50) and the 22 oldest messages will be removed due to the channel limit.

We strongly recommend not raising the limit back to the higher limit if messages have been removed in the previous step because those removed messages may or may not become available again depending on the store implementation or if running in clustering mode or not.

Clustering

When running --clustered, messages are kept in a RAFT consensus log file (under --cluster_log_path) in addition to the configured store.

This directory will grow as more messages keep coming within the 2-4 minute intervals of RAFT Snapshotting. After snapshotting the RAFT log will not shrink, but the space for more messages will be allocated internally. As a result - in addition to max_bytes multiplied by max_channels - disk space needs to be provisioned for RAFT Log, which can grow up to the size of payloads that channels could receive per minute * 4.