Read Consistency

rqlite support various levels of Read Consistency

You do not need to know the information on this page to use rqlite well, it’s mostly for advanced users. rqlite has also been run through Jepsen-style testing. You can read about that here.

Even though serving queries does not require Raft consensus (because the database is not changed), queries should generally be served by the cluster Leader. Why is this? Because, without this check, queries on a node could return results that are out-of-date i.e. stale. This could happen for one, or both, of the following two reasons:

  • The node that received your request , while still part of the cluster, has fallen behind the Leader in terms of updates to its underlying database.
  • The node is no longer part of the cluster, and has stopped receiving Raft log updates.

This is why rqlite offers selectable read consistency levels of weak (the default), strong, and none. Each is explained below, and examples of each are shown at the end of this page.

Weak

Weak consistency is used if you don’t specify any level.

Weak instructs the node receiving the read request to check that it is the Leader, before querying the local SQLite file. Checking Leader state only involves checking state local to the node, so is very fast. There is, however, a very small window of time (milliseconds by default) during which the node may return stale data if a Leader-election is in progress. This is because after the local Leader check, but before the local SQLite database is read, another node could be elected Leader and make changes to the cluster. As result the node may not be quite up-to-date with the rest of cluster.

If the node determines it is not the Leader, the node will transparently forward the request to the Leader, which will in turn perform a Weak read of its database. The node then waits for the response from the Leader, and then returns that response to the client.

Strong

To avoid even the issues associated with weak consistency, rqlite also offers strong. In this mode, the node receiving the request sends the query through the Raft consensus system, ensuring that the cluster Leader remains the Leader at all times during the processing of the query. When using strong you can be sure that the database reflects every change sent to it prior to the query. However, this will involve the Leader contacting at least a quorum of nodes, and will therefore increase query response times.

If a query request is sent to a Follower, and strong consistency is specified, the Follower will transparently forward the request to the Leader. The Follower waits for the response from the Leader, and then returns that response to the client.

None

With none, the node receving your read request simply queries its local SQLite database, and does not perform any Leadership check. This offers the fastest query response, but suffers from the potential issues outlined above, whereby there is a chance of Stale Reads if the Leader changes during the query, of if the node is disconnected from the clsuter.

Limiting read staleness

You can tell the receiving node not to return results staler than a certain duration, however. If a read request sets the query parameter freshness to a Go duration string, the node serving the read will check that less time has passed since it was last in contact with the Leader, than that specified via freshness. If more time has passed the node will return an error. freshness is ignored for all consistency levels except none, and is also ignored if set to zero.

The freshness parameter is always ignored if the node serving the query is the Leader. Any read, when served by the Leader, is always going to be within any possible freshness bound.

If you decide to deploy read-only nodes however, none combined with freshness can be a particularly effective at adding read scalability to your system. You can use lots of read-only nodes, yet be sure that a given node serving a request has not fallen too far behind the Leader (or even become disconnected from the cluster).

Which should I use?

Weak is almost certainly sufficient for your application, and is the default read consistency level. Unless your cluster Leader is continually changing while you’re actually executing queries there will be never be any difference between weak and strong – but using strong will result in slower queries, which is not what most people want.

How do I specify read consistency?

To explicitly select consistency, set the query param level to the desired level. However, you should use none with read-only nodes, unless you want those nodes to actually forward the query to the Leader.

Example queries

Examples of enabling each read consistency level for a simple query is shown below.

  1. # Default query options. The read request will be served by the node if it believes
  2. # it is the leader, otherwise it forwards the request to the Leader. Same as weak.
  3. curl -G 'localhost:4001/db/query' --data-urlencode 'q=SELECT * FROM foo'
  5. # Query the node, telling it simply to read the SQLite database directly.
  6. # No guarantees on how old the data is. In fact, the node may not even be
  7. # connected to the cluster. Provides the shortest possible query response time.
  8. curl -G 'localhost:4001/db/query?level=none' --data-urlencode 'q=SELECT * FROM foo'
  10. # Query the node, telling it simply to read the SQLite database directly.
  11. # The read request will be successful only if the node last heard from the
  12. # Leader no more than 1 second ago. This provides very fast reads, but sets
  13. # an upper bound of 1 seconod on how old the returned data iis.
  14. curl -G 'localhost:4001/db/query?level=none&freshness=1s' --data-urlencode 'q=SELECT * FROM foo'
  16. # The read request will be served by the node if it believes it is the Leader,
  17. # otherwise it wil forward the request to the Leader.
  18. curl -G 'localhost:4001/db/query?level=weak' --data-urlencode 'q=SELECT * FROM foo'
  20. # The read request will be successful only if the node maintained cluster
  21. # leadership during the entirety of query processing. Zero chance of stale reads
  22. # but query processing will be relatively slow.
  23. curl -G 'localhost:4001/db/query?level=strong' --data-urlencode 'q=SELECT * FROM foo'

Last modified May 29, 2023: Update _index.md (b5c530a)