Leaf Nodes

A Leaf Node extends an existing NATS system of any size, optionally bridging both operator and security domains. A leafnode server will transparently route messages as needed from local clients to one or more remote NATS system(s) and vice versa. The leaf node authenticates and authorizes clients using a local policy. Messages are allowed to flow to the cluster or into the leaf node based on leaf node connection permissions of either.

Leaf nodes are useful in IoT and edge scenarios and when the local server traffic should be low RTT and local unless routed to the super cluster. NATS’ queue semantics are honored across leaf connections by serving local queue consumer first.

  • Clients to leaf nodes authenticate locally (or just connect if authentication is not required)
  • Traffic between the leaf node and the cluster assumes the restrictions of the user configuration used to create the leaf connection.
    • Subjects that the user is allowed to publish are exported to the cluster.
    • Subjects the user is allowed to subscribe to, are imported into the leaf node.

Unlike cluster or gateway nodes, leaf nodes do not need to be reachable themselves and can be used to explicitly configure any acyclic graph topologies.

If a leaf node connects to a cluster, it is recommended to configure it with knowledge of all seed servers and have each seed server accept connections from leaf nodes. Should the remote cluster’s configuration change, the discovery protocol will gossip peers capable of accepting leaf connections. A leaf node can have multiple remotes, each connecting to a different cluster. Each URL in a remote needs to point to the same cluster. If one node in a cluster is configured as leaf node, all nodes need to. Likewise, if one server in a cluster accepts leaf node connections, all servers need to.

Leaf Nodes are an important component as a way to bridge traffic between local NATS servers you control and servers that are managed by a third-party. Synadia’s NATS Global Service (NGS) allows accounts to use leaf nodes, but gain accessibility to the global network to inexpensively connect geographically distributed servers or small clusters.

LeafNode Configuration Options

LeafNode Configuration Tutorial

The main server is just a standard NATS server. Clients to the main cluster are just using token authentication, but any kind of authentication can be used. The server allows leaf node connections at port 7422 (default port):

  1. leafnodes {
  2.     port: 7422
  3. }
  4. authorization {
  5.     token: "s3cr3t"
  6. }

Start the server:

  1. nats-server -c /tmp/server.conf
  2. ...
  3. [5774] 2019/12/09 11:11:23.064276 [INF] Listening for leafnode connections on 0.0.0.0:7422
  4. ...

We create a replier on the server to listen for requests on ‘q’, which it will aptly respond with ‘42’:

  1. nats-rply -s nats://s3cr3t@localhost q 42

The leaf node, allows local clients to connect to through port 4111, and doesn’t require any kind of authentication. The configuration specifies where the remote cluster is located, and specifies how to connect to it (just a simple token in this case):

  1. listen: "127.0.0.1:4111"
  2. leafnodes {
  3.     remotes = [ 
  4.         { 
  5.           url: "nats://s3cr3t@localhost"
  6.         },
  7.     ]
  8. }

In the case where the remote leaf connection is connecting with tls:

  1. listen: "127.0.0.1:4111"
  2. leafnodes {
  3.     remotes = [ 
  4.         { 
  5.           url: "tls://s3cr3t@localhost"
  6.         },
  7.     ]
  8. }

Note the leaf node configuration lists a number of remotes. The url specifies the port on the server where leaf node connections are allowed.

Start the leaf node server:

  1. nats-server -c /tmp/leaf.conf 
  2. ....
  3. [3704] 2019/12/09 09:55:31.548308 [INF] Listening for client connections on 127.0.0.1:4111
  4. ...
  5. [3704] 2019/12/09 09:55:31.549404 [INF] Connected leafnode to "localhost"

Connect a client to the leaf server and make a request to ‘q’:

  1. nats-req -s nats://127.0.0.1:4111 q ""
  2. Published [q] : ''
  3. Received  [_INBOX.Ua82OJamRdWof5FBoiKaRm.gZhJP6RU] : '42'

Leaf Node Example Using a Remote Global Service

In this example, we connect a leaf node to NGS. Leaf nodes are supported on developer and paid accounts. To sign up for a developer account, you’ll need the ngs tool which you can install by following instructions in https://github.com/ConnectEverything/ngs-cli.

Once you have the ngs tool installed, you can go ahead and import the synadia operator from ngs:

  1. > nsc add operator -u synadia
  2. [ OK ] imported operator "synadia"
  4. > nsc add account leaftest
  5. [ OK ] generated and stored account key "ACR4E2VU2ZC4GPTGOLL6GLO3WHUBBIQBM2JWOGRCEJJQEV6SVXL64JWD"
  6. [ OK ] push jwt to account server:
  7.     [ OK ] pushed account jwt to the account server
  8.     > NGS created a new free billing account for your JWT, leaftest [ACR4E2VU2ZC4].
  9.     > Use the 'ngs' command to manage your billing plan.
  10.     > If your account JWT is *not* in ~/.nsc, use the -d flag on ngs commands to locate it.
  11. [ OK ] pull jwt from account server
  12. [ OK ] added account "leaftest" to operator "Synadia Communications Inc."

In order to use leaf nodes, you’ll have to upgrade the account to the developer plan. The developer plan has zero cost, but requires specifying an email and providing a credit card number:

  1. > ngs edit
  3. Please select your new plan. For a complete description of offerings,
  4. please visit our website at https://www.https://www.synadia.com/.
  6. ? Select a Messaging Plan Developer $0.00/month
  8. Synadia will report service notifications and billing updates with the
  9. email address you associate with your account. This address will be
  10. verified if changed.
  12. ? Email natsuser@test.com
  14. ╭────────────────────────────────╮
  15. │        Account Details         │
  16. ├────────┬───────────────────────┤
  17. │ Email: │ natsuser@test.com     |
  18. │ Plan:  │ Developer $0.00/month │
  19. ╰────────┴───────────────────────╯
  22. ? Check your account details OK
  24. Your changes were sent to Synadia, but it looks like we need to verify
  25. your email and credit card before updating your account. You should
  26. receive a welcome email shortly.
  27. Once the update succeeds use nsc to sync the latest version of your
  28. synadia account JWT to disk.

Check your email, verify the email, and specify an credit card, after that:

  1. > nsc pull
  2. [ OK ] pulled "leaftest" from the account server
  4. > nsc describe account
  5. ╭──────────────────────────────────────────────────────────────────────────────────────╮
  6. │                                   Account Details                                    │
  7. ├───────────────────────────┬──────────────────────────────────────────────────────────┤
  8. │ Name                      │ leaftest                                                 │
  9. │ Account ID                │ ACR4E2VU2ZC4GPTGOLL6GLO3WHUBBIQBM2JWOGRCEJJQEV6SVXL64JWD │
  10. │ Issuer ID                 │ ODSKBNDIT3LTZWFSRAWOBXSBZ7VZCDQVU6TBJX3TQGYXUWRU46ANJJS4 │
  11. │ Issued                    │ 2019-12-09 14:44:55 UTC                                  │
  12. │ Expires                   │                                                          │
  13. ├───────────────────────────┼──────────────────────────────────────────────────────────┤
  14. │ Max Connections           │ 50                                                       │
  15. │ Max Leaf Node Connections │ 2                                                        │
  16. │ Max Data                  │ 5.0 GB (5000000000 bytes)                                │
  17. │ Max Exports               │ Unlimited                                                │
  18. │ Max Imports               │ Unlimited                                                │
  19. │ Max Msg Payload           │ 4.0 kB (4000 bytes)                                      │
  20. │ Max Subscriptions         │ 50                                                       │
  21. │ Exports Allows Wildcards  │ False                                                    │
  22. ├───────────────────────────┼──────────────────────────────────────────────────────────┤
  23. │ Exports                   │ None                                                     │
  24. ╰───────────────────────────┴──────────────────────────────────────────────────────────╯
  26. ....

Note the limits on the account, specify that the account can have up-to 2 leaf node connections. Let’s use them:

  1. > nsc add user leaftestuser
  2. [ OK ] generated and stored user key "UB5QBEU4LU7OR26JEYSG27HH265QVUFGXYVBRD7SVKQJMEFSZTGFU62F"
  3. [ OK ] generated user creds file "~/.nkeys/creds/synadia/leaftest/leaftestuser.creds"
  4. [ OK ] added user "leaftestuser" to account "leaftest"

Let’s craft a leaf node connection much like we did earlier:

  1. leafnodes {
  2.     remotes = [ 
  3.         { 
  4.           url: "tls://connect.ngs.global"
  5.           credentials: "/Users/alberto/.nkeys/creds/synadia/leaftest/leaftestuser.creds"
  6.         },
  7.     ]
  8. }

The default port for leaf nodes is 7422, so we don’t have to specify it.

Let’s start the leaf server:

  1. > nats-server -c /tmp/ngs_leaf.conf 
  2. ...
  3. [4985] 2019/12/09 10:55:51.577569 [INF] Listening for client connections on 0.0.0.0:4222
  4. ...
  5. [4985] 2019/12/09 10:55:51.918781 [INF] Connected leafnode to "connect.ngs.global"

Again, let’s connect a replier, but this time to NGS. NSC connects specifying the credentials file:

  1. nsc reply q 42

And now let’s make the request from the local host:

  1. > nats-req q ""
  2. Published [q] : ''
  3. Received  [_INBOX.hgG0zVcVcyr4G5KBwOuyJw.uUYkEyKr] : '42'

Leaf Authorization

In some cases you may want to restrict what messages can be exported from the leaf node or imported from the leaf connection. You can specify restrictions by limiting what the leaf connection client can publish and subscribe to. See NATS Authorization for how you can do this.