Read Consistency

rqlite support various levels of Read Consistency

You do not need to know the information on this page to use rqlite well, it’s mostly for advanced users. rqlite has also been run through Jepsen-style testing. You can read about that here.

Even though serving queries does not require Raft consensus (because the database is not changed), queries should generally be served by the cluster Leader. Why is this? Because, without this check, queries on a node could return results that are out-of-date i.e. stale. This could happen for one, or both, of the following two reasons:

  • The node that received your request , while still part of the cluster, has fallen behind the Leader in terms of updates to its underlying database.
  • The node is no longer part of the cluster, and has stopped receiving Raft log updates.

This is why rqlite offers selectable read consistency levels of weak (the default), linearizable, strong, and none. Each is explained below, and examples of each are shown at the end of this page.

Weak

Weak consistency is used if you don’t specify any level, or if an unrecognized level is specified – and it’s probably the right choice for your application.

Weak instructs the node receiving the read request to check that it is the Leader, and if it is the Leader, the node simply reads its local SQLite database. If the node determines it is not the Leader, the node will transparently forward the request to the Leader, which will in turn perform a Weak read of its database. In that case the node waits for the response from the Leader, and then returns that response to the client.

Weak reads are usually very fast, but have some potential shortcomings, which are described below.

A node checks if it’s the Leader by checking state local to the node, so this check is very fast. However there is a small window of time (less than a second by default) during which a node may think it’s the Leader, but has actually been deposed, a new Leader elected, and other writes have taken place on the cluster. If this happens the node may not be quite up-to-date with the rest of cluster, and stale data may be returned. Technically this means that weak reads are not Linearizable.

Linearizable

Linearizable reads implement the process described in section 6.4 of the Raft dissertation titled Processing read-only queries more efficiently.

To avoid the issues associated with weak consistency, rqlite also offers linearizable.

This type of read is, as the name suggests, linearizable because these types of reads reflect a state of the system sometime after the read was initiated; each read will at least return the results of the latest committed write1. Linearizable reads are reasonably fast, though measurably slower than weak.

How does the node guarantee linearizable reads? It does this as follows: when the node receives the read request it records the Raft Commit Index, and as well as checking local state to see if it is the Leader. Next the node heartbeats with the Followers, and waits until it receives a quorum of responses. Finally – and this is critical – the Leader waits until at least the write request contained in the previously recorded commit index is applied to the SQLite database. Once this happens it then performs the read.

Linearizable reads means the Leader contacts at least a quorum of nodes, and will therefore increase query response times. But since the Raft log is not actually involved, read performance is only dependant on the network performance between the nodes.

Strong

Strong consistency has little use in production systems, as the reads are costly and do not offer much, if any, benefit over Linearizable reads. Strong reads can be useful in certain testing scenarios however.

rqlite also offers a consistency level known as strong. In this mode, the node receiving the request ensures that all committed entries in the Raft log have also been applied to the SQLite database at the time the query is executed. Strong reads accomplish this by sending the query through the actual Raft log. This will, of course, involve the Leader contacting at least a quorum of nodes, some disk IO, and will therefore increase query response times. Strong reads are linearizable.

If a query request is sent to a Follower, and strong consistency is specified, the Follower will transparently forward the request to the Leader. The Follower waits for the response from the Leader, and then returns that response to the client.

None

With none, the node receiving your read request simply queries its local SQLite database, and does not perform any Leadership check – in fact, the node could be completely disconnected from the rest of the cluster, but the query will still be successful. This offers the absolute fastest query response, but suffers from the potential issues outlined above, whereby there is a chance of Stale Reads if the Leader changes during the query, or if the node has become disconnected from the cluster.

Limiting read staleness

You can tell the node which receives the read rquest not to return results if the node has been disconnected from the cluster for longer than a specified duration. If a read request sets the query parameter freshness to a Go duration string, the node serving the read will check that less time has passed since it was last in contact with the Leader, than that specified via freshness. If more time has passed the node will return an error. This approach can be useful if you want to maximize successful query operations, but are willing to tolerate occassional, short-lived networking issues between nodes.

It’s important to note that the freshness does not guarantee that the node is caught up with the Leader, only that it is contact with the Leader. But if a node is in contact with the Leader, it’s usually caught up with all changes that have taken place on the cluster. To check if node is actually caught up with Leader, use freshness_strict in addition to the freshness query parameter.

The freshness parameter is always ignored if the node serving the query is the Leader. Any read, when served by the Leader, is always going to be within any possible freshness bound. freshness is also ignored for all consistency levels except none, and is also ignored if set to zero.

If you decide to deploy read-only nodes however, none combined with freshness can be a particularly effective at adding read scalability to your system. You can use lots of read-only nodes, yet be sure that a given node serving a request has not been disconnected from the cluster for too long.

freshness_strict

As explained above freshness just checks that that the node has been in contact with the Leader within the specified time. If you also want the node to check that the data it last received is not out-of-date by (at most) the freshness interval, you should also add the freshness_strict flag as a query parameter. Note that this check works by comparing timestamps generated by the Leader to those generated by the node receiving the read request. Any clock skew between the nodes may therefore affect the correctness of the data returned by the node. You are responsible for controlling the amount of clock skew across your rqlite cluster.

See the examples at the end of this page to learn how to control freshness.

Auto

Auto is not an actual Read Consistency level. Instead if a client selects this level during a read request, the receiving node will automatically select the level which is (usually) most sensible for the node’s type. In the case of a read-only node None is chosen as the level. In all other cases Weak is the chosen as the level.

Using auto can simplify clients as clients do not need know ahead of time whether they will be talking to a read-only node or voting node. A client can just select auto.

Which should I use?

Weak is usually the right choice for your application, and is the default read consistency level. Unless your cluster Leader is continually changing while you’re actually executing queries there will be never be any difference between weak and linearizable – but using linearizable will result in slower queries, which is not what most people want. However linearizable has its uses, and you may need it depending on your application requirements.

One exception to the rule above is if you’re querying read-only nodes. In that case you probably want to specify None, possibly setting the freshness controls too. If you set a read consistency level other than None when querying a read-only node then that read-only node will simply forward the request to the Leader (which partially defeats the purpose of read-only nodes).

If you are running a cluster which has some read-only nodes, and you want to implement the Read Consistency policy describe above in an easy manner, check out auto Read Consistency.

Strong is likely unsuitable for production systems, is slow, and puts measurable load on the cluster. However, it can be quite useful in certain testing scenarios, as it removes any uncertainty regarding the difference between committed changes and applied changes. Specifically when you use Strong all currently committed entries in the Raft log have also been applied to the SQLite database at the time the read request is executed.

How do I specify read consistency?

To explicitly select consistency, set the query param level to the desired level. However, you should use none with read-only nodes, unless you want those nodes to actually forward the query to the Leader.

Example queries

Examples of enabling each read consistency level for a simple query is shown below.

  1. # Default query options. The read request will be served by the node if it believes
  2. # it is the leader, otherwise it transparently forwards the request to the Leader, and
  3. # waits for a response from the Leader. Same as weak.
  4. curl -G 'localhost:4001/db/query' --data-urlencode 'q=SELECT * FROM foo'
  6. # The read request will be served by the node if it believes it is the Leader,
  7. # otherwise it wil forward the request to the Leader. This is the default if
  8. # no read consistency is specified.
  9. curl -G 'localhost:4001/db/query?level=weak' --data-urlencode 'q=SELECT * FROM foo'
  11. # The read request will be served by the node if it is the Leader, and if it
  12. # remained the Leader throughout the processing of the read. If the node
  13. # receiving the query is not the the Leader, the request will be transparently
  14. # forwarded to the Leader.
  15. curl -G 'localhost:4001/db/query?level=linearizable' --data-urlencode 'q=SELECT * FROM foo'
  17. # The read request will be served by the node if it is the Leader, and if it
  18. # remained the Leader throughout the processing of the read. If the node
  19. # receiving the query is not the the Leader, the request will be transparently
  20. # forwarded to the Leader. If a linearizable read is not available within 1
  21. # second of receiving the read request, an error will be returned.
  22. curl -G 'localhost:4001/db/query?level=linearizable&linearizable_timeout=1s' --data-urlencode 'q=SELECT * FROM foo'
  24. # Query the node, telling it simply to read the SQLite database directly.
  25. # No guarantees on how old the data is. In fact, the node may not even be
  26. # connected to the cluster. Provides the fastest possible query response.
  27. curl -G 'localhost:4001/db/query?level=none' --data-urlencode 'q=SELECT * FROM foo'
  29. # Query the node, telling it simply to read the SQLite database directly.
  30. # The read request will be successful only if the node last heard from the
  31. # Leader no more than 1 second ago. This provides very fast reads, but sets
  32. # an upper bound of 1 second on how long the node may have been disconnected
  33. # from the cluster.
  34. curl -G 'localhost:4001/db/query?level=none&freshness=1s' --data-urlencode 'q=SELECT * FROM foo'
  36. # Query the node, telling it simply to read the SQLite database directly.
  37. # The read request will be successful only if the node last heard from the
  38. # Leader no more than 1 second ago, and if the most recently received data
  39. # was appended by the Leader to the log within 1 second ago, relative to the
  40. # node's local clock.
  41. curl -G 'localhost:4001/db/query?level=none&freshness=1s&freshness_strict' --data-urlencode 'q=SELECT * FROM foo'
  43. # The read request will be processed by the Leader and will be successful
  44. # only if the Leader maintained leadership during the entirety of query
  45. # processing. Zero chance of stale reads but query processing will be
  46. # relatively slow. If the node receiving the query is not the the Leader,
  47. # the request will be transparently forwarded to the Leader.
  48. curl -G 'localhost:4001/db/query?level=strong' --data-urlencode 'q=SELECT * FROM foo'
  50. # Query the node, enabling 'auto' Read Consistency mode. If the receiving
  51. # node is read-only i.e. non-voting, then 'none' will be set as the Read
  52. # Consistency level, and the read-only node will check that it heard from
  53. # the Leader within the last second. For voting nodes 'weak' is set as the
  54. # Read Consistency level, and the freshness value is ignored.
  55. curl -G 'localhost:4001/db/query?level=auto&freshness=1s' --data-urlencode 'q=SELECT * FROM foo'
  1. This is how the Raft dissertation defines Linearizability. ↩︎

Last modified October 28, 2024: Update _index.md (00008a1)