Leaf Nodes

A Leaf Node extends an existing NATS system of any size, optionally bridging both operator and security domains. A leafnode server will transparently route messages as needed from local clients to one or more remote NATS system(s) and vice versa. The leaf node authenticates and authorizes clients using a local policy. Messages are allowed to flow to the cluster or into the leaf node based on leaf node connection permissions of either.

Leaf nodes are useful in IoT and edge scenarios and when the local server traffic should be low RTT and local unless routed to the super cluster. NATS’ queue semantics are honored across leaf connections by serving local queue consumers first.

  • Clients to leaf nodes authenticate locally (or just connect if authentication is not required)
  • Traffic between the leaf node and the cluster assumes the restrictions of the user configuration used to create the leaf connection.
    • Subjects that the user is allowed to publish are exported to the cluster.
    • Subjects the user is allowed to subscribe to, are imported into the leaf node.

Unlike cluster or gateway nodes, leaf nodes do not need to be reachable themselves and can be used to explicitly configure any acyclic graph topologies.

If a leaf node connects to a cluster, it is recommended to configure it with knowledge of all seed servers and have each seed server accept connections from leaf nodes. Should the remote cluster’s configuration change, the discovery protocol will gossip peers capable of accepting leaf connections. A leaf node can have multiple remotes, each connecting to a different cluster. Each URL in a remote needs to point to the same cluster. If one node in a cluster is configured as leaf node, all nodes need to. Likewise, if one server in a cluster accepts leaf node connections, all servers need to.

Leaf Nodes are an important component as a way to bridge traffic between local NATS servers you control and servers that are managed by a third-party. Synadia’s NGS allows accounts to use leaf nodes, but gain accessibility to the global network to inexpensively connect geographically distributed servers or small clusters.

LeafNode Configuration Options

LeafNode Configuration Tutorial

The main server is just a standard NATS server. Clients to the main cluster are just using token authentication, but any kind of authentication can be used. The server allows leaf node connections at port 7422 (default port):

  1. leafnodes {
  2.     port: 7422
  3. }
  4. authorization {
  5.     token: "s3cr3t"
  6. }

Start the server:

  1. nats-server -c /tmp/server.conf

Output extract

  1. ...
  2. [5774] 2019/12/09 11:11:23.064276 [INF] Listening for leafnode connections on 0.0.0.0:7422
  3. ...

We create a replier on the server to listen for requests on ‘q’, which it will aptly respond with ‘42’:

  1. nats reply -s nats://s3cr3t@localhost q 42

The leaf node, allows local clients to connect to through port 4111, and doesn’t require any kind of authentication. The configuration specifies where the remote cluster is located, and specifies how to connect to it (just a simple token in this case):

  1. listen: "127.0.0.1:4111"
  2. leafnodes {
  3.     remotes = [
  4.         {
  5.           url: "nats://s3cr3t@localhost"
  6.         },
  7.     ]
  8. }

In the case where the remote leaf connection is connecting with tls:

  1. listen: "127.0.0.1:4111"
  2. leafnodes {
  3.     remotes = [
  4.         {
  5.           url: "tls://s3cr3t@localhost"
  6.         },
  7.     ]
  8. }

Note the leaf node configuration lists a number of remotes. The url specifies the port on the server where leaf node connections are allowed.

Start the leaf node server:

  1. nats-server -c /tmp/leaf.conf

Output extract

  1. ....
  2. [3704] 2019/12/09 09:55:31.548308 [INF] Listening for client connections on 127.0.0.1:4111
  3. ...
  4. [3704] 2019/12/09 09:55:31.549404 [INF] Connected leafnode to "localhost"

Connect a client to the leaf server and make a request to ‘q’:

  1. nats req -s nats://127.0.0.1:4111 q ""
  1. Published [q] : ''
  2. Received  [_INBOX.Ua82OJamRdWof5FBoiKaRm.gZhJP6RU] : '42'

Leaf Node Example Using a Remote Global Service

In this example, we connect a leaf node to Synadia’s NGS. Leaf nodes are supported on free developer and paid accounts. To use NGS, ensure that you’ve signed up and have an account loaded on your local system. It takes less than 30 seconds to grab yourself a free account to follow along if you don’t have one already!

The nsc tool can operate with many accounts and operators, so it’s essential to make sure you’re working with the right operator and account. You can set the account using the nsc tool like below. The DELETE_ME account is used as an example, which is registered with NGS as a free account.

  1. ❯ nsc env -a DELETE_ME
  2. ❯ nsc describe account
  3. +--------------------------------------------------------------------------------------+
  4. |                                   Account Details                                    |
  5. +---------------------------+----------------------------------------------------------+
  6. | Name                      | DELETE_ME                                                |
  7. | Account ID                | ABF3NX7FJLDCUO5QXBH56PV6EU4PR5HFCUJBXAG57AKSDUBTGORDFOLI |
  8. | Issuer ID                 | ODSKBNDIT3LTZWFSRAWOBXSBZ7VZCDQVU6TBJX3TQGYXUWRU46ANJJS4 |
  9. | Issued                    | 2023-03-02 18:18:42 UTC                                  |
  10. | Expires                   |                                                          |
  11. +---------------------------+----------------------------------------------------------+
  12. | Max Connections           | 10                                                       |
  13. | Max Leaf Node Connections | Not Allowed                                              |
  14. | Max Data                  | 1.0 GB (1000000000 bytes)                                |
  15. | Max Exports               | 2                                                        |
  16. | Max Imports               | 7                                                        |
  17. | Max Msg Payload           | 1.0 kB (1000 bytes)                                      |
  18. | Max Subscriptions         | 10                                                       |
  19. | Exports Allows Wildcards  | True                                                     |
  20. | Disallow Bearer Token     | False                                                    |
  21. | Response Permissions      | Not Set                                                  |
  22. +---------------------------+----------------------------------------------------------+
  23. | Jetstream                 | Disabled                                                 |
  24. +---------------------------+----------------------------------------------------------+
  25. | Exports                   | None                                                     |
  26. +---------------------------+----------------------------------------------------------+

The nsc tool is aware of the account, so let’s proceed to create a user for our example.

  1. nsc add user leaftestuser
  1. [ OK ] generated and stored user key "UB5QBEU4LU7OR26JEYSG27HH265QVUFGXYVBRD7SVKQJMEFSZTGFU62F"
  2. [ OK ] generated user creds file "~/.nkeys/creds/synadia/leaftest/leaftestuser.creds"
  3. [ OK ] added user "leaftestuser" to account "leaftest"

Let’s craft a leaf node connection much like we did earlier:

  1. leafnodes {
  2.     remotes = [
  3.         {
  4.           url: "tls://connect.ngs.global"
  5.           credentials: "/Users/alberto/.nkeys/creds/synadia/leaftest/leaftestuser.creds"
  6.         },
  7.     ]
  8. }

The default port for leaf nodes is 7422, so we don’t have to specify it.

Let’s start the leaf server:

  1. nats-server -c /tmp/ngs_leaf.conf
  1. ...
  2. [4985] 2023/03/03 10:55:51.577569 [INF] Listening for client connections on 0.0.0.0:4222
  3. ...
  4. [4985] 2023/03/03 10:55:51.918781 [INF] Connected leafnode to "connect.ngs.global"

Again, let’s connect a replier, but this time to Synadia’s NGS. NSC connects specifying the credentials file:

  1. nsc reply q 42

And now let’s make the request from the local host:

  1. nats-req q ""
  1. Published [q] : ''
  2. Received  [_INBOX.hgG0zVcVcyr4G5KBwOuyJw.uUYkEyKr] : '42'

Leaf Authorization

In some cases you may want to restrict what messages can be exported from the leaf node or imported from the leaf connection. You can specify restrictions by limiting what the leaf connection client can publish and subscribe to. See NATS Authorization for how you can do this.

TLS-first Handshake

As of NATS v2.10.0

Leafnode connections follow the model where when a TCP connection is created to the server, the server will immediately send an INFO protocol message in clear text. This INFO protocol provides metadata, including whether the server requires a secure connection.

Some environments prefer not to want a server that is configured to accept TLS connections for leafnodes having any traffic sent in clear text. It was possible to by-pass this using a websocket connection. However, if websocket is not desired, the accepting and remote servers can be configured to perform a TLS handshake before sending the INFO protocol message.