Monitoring rqlite

How to monitor rqlite

Status API

rqlite serves diagnostic and statistical information, as well as detailed information about the underlying Raft system, at /status. Assuming the rqlite node is started with default settings you can issue a curl command like so to retrieve this information:

  1. curl localhost:4001/status?pretty

The use of the URL param pretty is optional, and results in pretty-printed JSON responses. The output of this endpoint could be periodically written to a monitoring system, allowing the performance of rqlite to be tracked over time.

You can also request the same status information via the rqlite shell:

  1. $ ./rqlite 
  2. Welcome to the rqlite CLI. Enter ".help" for usage hints.
  3. 127.0.0.1:4001> .status
  4. build:
  5.   build_time: unknown
  6.   commit: unknown
  7.   version: 5
  8.   branch: unknown
  9. http:
  10.   addr: 127.0.0.1:4001
  11.   auth: disabled
  12.   redirect: 
  13. node:
  14.   start_time: 2019-12-23T22:34:46.215507011-05:00
  15.   uptime: 16.963009139s
  16. runtime:
  17.   num_goroutine: 9
  18.   version: go1.13

Nodes API

The nodes API returns basic information for nodes in the cluster, as seen by the node receiving the nodes request. The receiving node will also check whether it can actually connect to all other nodes in the cluster. This is an effective way to determine the cluster leader, and the leader’s HTTP API address. It can also be used to check if the cluster is basically running – if the other nodes are reachable, it probably is.

By default, the node only checks if voting nodes are contactable.

  1. curl localhost:4001/nodes?pretty
  3. # Also check non-voting nodes.
  4. curl localhost:4001/nodes?nonvoters&pretty
  6. # Give up if all nodes don't respond within 5 seconds. Default is 30 seconds.
  7. curl localhost:4001/nodes?timeout=5s

You can also request the same nodes information via the rqlite shell:

  1. $ ./rqlite
  2. Welcome to the rqlite CLI. Enter ".help" for usage hints.
  3. 127.0.0.1:4001> .nodes
  4. 1:
  5.   api_addr: http://localhost:4001
  6.   addr: 127.0.0.1:4002
  7.   reachable: true
  8.   leader: true
  9. 2:
  10.   api_addr: http://localhost:4003
  11.   addr: 127.0.0.1:4004
  12.   reachable: true
  13.   leader: false
  14. 3:
  15.   api_addr: http://localhost:4005
  16.   addr: 127.0.0.1:4006
  17.   reachable: true
  18.   leader: false

Readiness checks

rqlite nodes serve a “ready” status at /readyz. The endpoint will return HTTP 200 OK if the node is ready to respond to database requests and cluster management operations. An example access is shown below.

  1. $ curl localhost:4001/readyz
  2. [+]node ok
  3. [+]leader ok
  4. [+]store ok

If you wish to check if the node is running, and responding to HTTP requests, regardless of Leader status, add noleader to the URL. This form may be more useful for automated deployments, which simply need to know if the node is responsive.

  1. $ curl localhost:4001/readyz?noleader
  2. [+]node ok

expvar support

rqlite also exports expvar information, which are mostly counters of various rqlite activity. The standard expvar information, as well as some custom information, is exposed. This data can be retrieved like so (assuming the node is started in its default configuration):

  1. curl localhost:4001/debug/vars

You can optionally set the query parmameter key if you wish to retrieve just a subsection of the expvar output e.g. the URL localhost:4001/debug/vars?key=http will return just HTTP information.

You can also request the same expvar information via the CLI:

  1. $ rqlite
  2. 127.0.0.1:4001> .expvar
  3. cmdline: [./rqlited data]
  4. db:
  5.   execute_transactions: 0
  6.   execution_errors: 1
  7.   executions: 1
  8.   queries: 0
  9.   query_transactions: 0
  10. http:
  11.   backups: 0
  12.   executions: 0
  13.   queries: 0
  14. memstats:
  15.   Mallocs: 8950
  16.   HeapSys: 2.588672e+06
  17.   StackInuse: 557056
  18.   LastGC: 0...

Similar to the status output, the output of this endpoint could be periodically written to a monitoring system, allowing the performance of rqlite to be tracked over time.

pprof support

pprof information is available by default and can be accessed as follows:

  1. curl localhost:4001/debug/pprof/cmdline
  2. curl localhost:4001/debug/pprof/profile
  3. curl localhost:4001/debug/pprof/symbol

Last modified May 29, 2023: Update _index.md (3b4a4a7)