Security

How to secure your rqlite deployment

rqlite can be secured in various way, and with different levels of control.

SQLite security

SQLite has some documentation on security, which is worth reviewing. Much of it can be applied to rqlite, though implementing some of the practices would need you to recompile rqlite.

File system security

You should control access to the data directory that each rqlite node uses. There is no reason for any user to directly access this directory. File-level security is also very important if you decide to use TLS Certificates and Keys with rqlite (see later).

You are also responsible for securing access to the SQLite database files if you enable “on disk” mode (which is not the default mode). Unless your application requires it, there is no reason for any client to directly access the SQLite file, and writing to the SQLite file may cause rqlite to fail. If your users don’t need to access a SQLite database file, then don’t enable “on disk” mode. This will maximize file-level security.

Network security

Each rqlite node listens on 2 TCP ports – one for the HTTP API, and the other for inter-node (Raft consensus) communications. Only the HTTP API port need be reachable from outside the cluster.

So, if possible, configure the network such that the Raft port on each node is only accessible from other nodes in the cluster. Alternatively run the Raft connections on a physically, or logically, different network from the network the HTTP API is connected to. There is no need for the Raft port to be accessible by rqlite clients, which only need to use the HTTP API.

If the IP addresses (or subnets) of rqlite clients is also known, it may also be possible to limit access to the HTTP API from those addresses only.

AWS EC2 Security Groups, for example, support all this functionality. So if running rqlite in the AWS EC2 cloud you can implement this level of security at the network level.

HTTPS API

rqlite supports secure access via HTTPS, using Transport Layer Security (TLS). Using TLS ensuring that all communication between clients and a cluster is encrypted. The HTTPS API also supports Mutual TLS. If mutual TLS is enabled only clients which present a trusted certificate can access rqlite.

To configure HTTPS, you set the following command-line options when launching rqlite:

  1.   -http-ca-cert string
  2.       Path to X.509 CA certificate for HTTPS.
  3.       If not set, then the host systems CA certificate(s) will be used.
  4.       This certificate must be set however if using mutual TLS.
  5.   -http-cert string
  6.       Path to HTTPS X.509 certificate
  7.   -http-key string
  8.       Path to corresponding HTTPS X.509 private key
  9.   -http-no-verify
  10.       Skip verification of HTTPS certificates when joining a cluster.
  11.       Mostly used for testing.
  12.   -http-verify-client
  13.       Enable mutual TLS for HTTPS. Mutual TLS is disabled by default.

Node-to-node encryption

rqlite supports authentication encryption of all inter-node traffic1. TLS is again used, and mutual TLS is also supported so you can restrict nodes to only accept inter-node connections from other authorized nodes. To use TLS each node must be supplied with the relevant SSL certificate and corresponding private key, in X.509 format. Note that every node in a cluster must operate with inter-node encryption enabled, or none at all.

  1.   -node-ca-cert string
  2.       Path to X.509 CA certificate for node-to-node encryption.
  3.       If not set, then the host systems CA certificate(s) will be used
  4.       when verifying server certificates. This certificate is required
  5.       for verifying client certificates.
  6.   -node-cert string
  7.       Path to X.509 certificate for node-to-node communication
  8.   -node-key string
  9.       Path to X.509 private key for node-to-node communicate
  10.   -node-no-verify
  11.       Skip verification of any node-node certificates in each direction.
  12.       Mostly used for testing.
  13.   -node-verify-client
  14.       Enable mutual TLS for node-to-node communication.
  15.       Mutual TLS is disabled by default.

1There is one time nodes use the HTTP API when communicating – during a Join operation, or during bootstrapping (autoclustering). But once a node joins a cluster, all further communications takes place over the node-to-node connection.

Basic Auth

The HTTP API supports Basic Auth. Each rqlite node can be passed a JSON-formatted configuration file, which configures valid usernames and associated passwords for that node. The password string can be in cleartext or bcrypt hashed.

Since the configuration file only controls the node local to it, it’s important to ensure the configuration is identical on each node for consistent behaviour.

User-level permissions

rqlite, via the configuration file, also supports user-level permissions. Each user can be granted one or more of the following permissions:

  • all: user can perform all operations on a node.
  • execute: user may access the execute endpoint.
  • query: user may access the query endpoint.
  • load: user may load an SQLite dump file into a node.
  • backup: user may perform backups.
  • status: user can retrieve node status and Go runtime information.
  • ready: user can retrieve node readiness.
  • join: user can join a cluster. In practice only a node joins a cluster, so it’s the joining node that must supply the credentials.
  • join-read-only: user can join a cluster, but only as a read-only node.
  • remove: user can remove a node from a cluster.

Note that for a user to be able to access the Unified Endpoint, they must have both execute and query permissions.

Example configuration file

An example configuration file is shown below.

  1. [
  2.   {
  3.     "username": "bob",
  4.     "password": "secret1",
  5.     "perms": ["all"]
  6.   },
  7.   {
  8.     "username": "mary",
  9.     "password": "$2a$10$fKRHxrEuyDTP6tXIiDycr.nyC8Q7UMIfc31YMyXHDLgRDyhLK3VFS",
  10.     "perms": ["query", "backup", "join"]
  11.   },
  12.   {
  13.     "username": "*",
  14.     "perms": ["status", "ready", "join-read-only"]
  15.   }
  16. ]

This configuration file sets authentication for three usernames, bob, mary, and *. It sets a password for the first two.

This configuration also sets permissions for all usernames. bob has permission to perform all operations, but mary can only query the cluster, as well as backup and join the cluster. * is a special username, which indicates that all users – even anonymous users (requests without any BasicAuth information) – have permission to check the cluster status and readiness. All users can also join as a read-only node. This can be useful if you wish to leave certain operations open to all accesses.

Secure cluster example

Starting a node with HTTPS enabled, node-to-node encryption, mutual TLS disabled, and with the above configuration file. It is assumed the HTTPS X.509 certificate and key are at the paths server.crt and key.pem respectively, and the node-to-node certificate and key are at node.crt and node-key.pem

  1. rqlited -auth config.json -http-cert server.crt -http-key key.pem \
  2. -node-cert node.crt -node-key node-key.pem ~/node.1

Bringing up a second node on the same host, joining it to the first node. This allows you to block nodes from joining a cluster, unless those nodes supply a password.

  1. rqlited -auth config.json -http-addr localhost:4003 -http-cert server.crt \
  2. -http-key key.pem -raft-addr :4004 -join https://bob:secret1@localhost:4001 \
  3. -node-cert node.crt -node-key node-key.pem ~/node.2

Querying the node, as user mary.

  1. curl -G 'https://mary:secret2@localhost:4001/db/query?pretty&timings' \
  2. --data-urlencode 'q=SELECT * FROM foo'

Avoiding passwords at the command line

The above example suffer from one shortcoming – the password for user bob is entered at the command line. This is not ideal, as someone with access to the process table could learn the password. You can avoid this via the -join-as option, which will tell rqlite to retrieve the password from the configuration file.

  1. rqlited -auth config.json -http-addr localhost:4003 -http-cert server.crt \
  2. -http-key key.pem -raft-addr :4004 -join https://localhost:4001 -join-as bob \
  3. -node-cert node.crt -node-key node-key.pem \
  4. ~/node.2

How can I generate certificates and keys?

There are a lot of resources available on the web explaining how to do so. The most popular tool for creating and signing certificates and keys is OpenSSL.

Last modified May 19, 2023: Update _index.md (1c7da61)