description: Using JetStream on Leaf Nodes

Leaf Nodes

If you want to see a demonstration of the full range of this functionality, check out our video

One of the use cases for a NATS server configured as a leaf node is to provide a local NATS network even when the connection to a hub or the cloud is down. To support such a disconnected use case with JetStream, independent JetStream islands are also supported and available through the same NATS network.

The general issue with multiple, independent JetStreams, accessible from the same client is that you need to be able to tell them apart. As an example, consider a leaf node with a non-clustered JetStream in each server. You connect to one of them, but which JetStream responds when you use the JetStream API $JS.API.> ?

To disambiguate between servers, the option domain was added to the JetStream configuration block. When using it, follow these rules: Every server in a cluster and super cluster needs to have the same domain name. This means that domain names can only change between two servers if they are connected via a leaf node connection. As a result of this the JetStream API $JS.API.> will also be available under a disambiguated name $JS.<domain>.API.>. Needless to say, domain names need to be unique.

There are reasons to connect system accounts on either end of your leaf node connection. You probably don’t want to connect your cloud and edge device system accounts, but you might connect them when the only reason keeping you from using a super cluster are firewall rules.

The benefits are:

  • Monitoring of all connected nats-servers
  • nats-account-resolver working on the entire network
  • extended JetStream cluster

When domain is set, JetStream-related traffic on the system account is suppressed. This is what causes JetStream not to be extended.

In addition, traffic on $JS.API.> is also suppressed. This causes clients to use the local JetStream that is available in the nats-servers they are connected to. To communicate with another JetStream, that JetStream’s unique domain specific prefix $JS.<domain>.API needs to be specified.

Please be aware that each domain is an independent name space. Meaning, inside the same account it is legal to use the same stream name in different domains.

Furthermore, regular message flow is not restricted. Thus, if the same subject is subscribed to by different streams in the same account in different domains, as long as the underlying leaf node was connected at the time, each stream will store the message. This can be resolved by using the same account but use different subjects in each domain or use different accounts in each domain or isolate accounts used in leaf nodes.

Known issue: if you have more than one JetStream enabled leaf node in a different cluster, the cluster you connect to also needs JetStream enabled and a domain set.

Known issue: when you intend to extend a central JetStream, by not supplying a domain name in leaf nodes, that central JetStream needs to be in clustered mode.

Configuration

Below is the config needed to connect two JetStream enabled servers via a leaf node connection. In the example, the system accounts are connected for demonstration purposes (you do not have to do that).

accounts.conf Imported by Both Servers

  1. accounts {
  2. SYS: {
  3. users: [{user: admin, password: admin}]
  4. },
  5. ACC: {
  6. users: [{user: acc, password: acc}],
  7. jetstream: enabled
  8. }
  9. }
  10. system_account: SYS

hub.conf

To be started with nats-server -c hub.conf:

  1. port: 4222
  2. server_name: hub-server
  3. jetstream {
  4. store_dir="./store_server"
  5. domain=hub
  6. }
  7. leafnodes {
  8. port: 7422
  9. }
  10. include ./accounts.conf

leaf.conf

To be started with nats-server -c leaf.conf:

  1. port: 4111
  2. server_name: leaf-server
  3. jetstream {
  4. store_dir="./store_leaf"
  5. domain=leaf
  6. }
  7. leafnodes {
  8. remotes = [
  9. {
  10. urls: ["nats://admin:admin@0.0.0.0:7422"]
  11. account: "SYS"
  12. },
  13. {
  14. urls: ["nats://acc:acc@0.0.0.0:7422"]
  15. account: "ACC"
  16. }
  17. ]
  18. }
  19. include ./accounts.conf

Usage

Because the system account is connected, you can obtain the JetStream server report from both servers.

  1. nats --server nats://admin:admin@localhost:4222 server report jetstream
  1. ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
  2. JetStream Summary
  3. ├─────────────┬─────────────┬────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
  4. Server Cluster Domain Streams Consumers Messages Bytes Memory File API Req API Err
  5. ├─────────────┼─────────────┼────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
  6. leaf-server leaf-server leaf 0 0 0 0 B 0 B 0 B 0 0
  7. hub-server hub 0 0 0 0 B 0 B 0 B 0 0
  8. ├─────────────┼─────────────┼────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
  9. 0 0 0 0 B 0 B 0 B 0 0
  10. ╰─────────────┴─────────────┴────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯

Create a stream named test subscribing to subject test in the JetStream domain, the program is connected to. As a result, this stream will be created in the domain hub which is the domain of the server listening on localhost:4222.

  1. nats --server nats://acc:acc@localhost:4222 stream add
  1. ? Stream Name test
  2. ? Subjects to consume test
  3. ? Storage backend file
  4. ? Retention Policy Limits
  5. ? Discard Policy Old
  6. ? Stream Messages Limit -1
  7. ? Per Subject Messages Limit -1
  8. ? Message size limit -1
  9. ? Maximum message age limit -1
  10. ? Maximum individual message size -1
  11. ? Duplicate tracking time window 2m
  12. ? Replicas 1
  13. Stream test was created
  14. Information for Stream test created 2021-06-28T12:52:29-04:00
  15. Configuration:
  16. Subjects: test
  17. Acknowledgements: true
  18. Retention: File - Limits
  19. Replicas: 1
  20. Discard Policy: Old
  21. Duplicate Window: 2m0s
  22. Maximum Messages: unlimited
  23. Maximum Bytes: unlimited
  24. Maximum Age: 0.00s
  25. Maximum Message Size: unlimited
  26. Maximum Consumers: unlimited
  27. State:
  28. Messages: 0
  29. Bytes: 0 B
  30. FirstSeq: 0
  31. LastSeq: 0
  32. Active Consumers: 0

To create a stream in a different domain while connected somewhere else, just provide the js-domain argument. While connected to the same server as before, now the stream is created in leaf.

  1. nats --server nats://acc:acc@localhost:4222 stream add --js-domain leaf
  1. ? Stream Name test
  2. ? Subjects to consume test
  3. ? Storage backend file
  4. ? Retention Policy Limits
  5. ? Discard Policy Old
  6. ? Stream Messages Limit -1
  7. ? Per Subject Messages Limit -1
  8. ? Message size limit -1
  9. ? Maximum message age limit -1
  10. ? Maximum individual message size -1
  11. ? Duplicate tracking time window 2m
  12. ? Replicas 1
  13. Stream test was created
  14. Information for Stream test created 2021-06-28T12:59:18-04:00
  15. Configuration:
  16. Subjects: test
  17. Acknowledgements: true
  18. Retention: File - Limits
  19. Replicas: 1
  20. Discard Policy: Old
  21. Duplicate Window: 2m0s
  22. Maximum Messages: unlimited
  23. Maximum Bytes: unlimited
  24. Maximum Age: 0.00s
  25. Maximum Message Size: unlimited
  26. Maximum Consumers: unlimited
  27. State:
  28. Messages: 0
  29. Bytes: 0 B
  30. FirstSeq: 0
  31. LastSeq: 0
  32. Active Consumers: 0

Publish a message so there is something to retrieve.

  1. nats --server nats://acc:acc@localhost:4222 pub test "hello world"

Because both streams subscribe to the same subject, each one now reports one message. This is done to demonstrate the issue. If you want to avoid that, you need to either use different subjects, different accounts, or one isolated account.

  1. nats --server nats://acc:acc@localhost:4222 stream report --js-domain leaf
  1. Obtaining Stream stats
  2. ╭─────────────────────────────────────────────────────────────────────────────╮
  3. Stream Report
  4. ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
  5. Stream Storage Consumers Messages Bytes Lost Deleted Replicas
  6. ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
  7. test File 0 1 45 B 0 0
  8. ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯
  1. nats --server nats://acc:acc@localhost:4222 stream report --js-domain hub

Output

  1. Obtaining Stream stats
  2. ╭─────────────────────────────────────────────────────────────────────────────╮
  3. Stream Report
  4. ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
  5. Stream Storage Consumers Messages Bytes Lost Deleted Replicas
  6. ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
  7. test File 0 1 45 B 0 0
  8. ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

Copying across domains via source or mirror

In order to copy a stream from one domain into another, specify the JetStream domain when creating a mirror. If you want to connect a leaf to the hub and get commands, even when the leaf node connection is offline, mirroring a stream located in the hub is the way to go.

  1. nats --server nats://acc:acc@localhost:4222 stream add --js-domain hub --mirror test
  1. ? Stream Name backup-test-leaf
  2. ? Storage backend file
  3. ? Retention Policy Limits
  4. ? Discard Policy Old
  5. ? Stream Messages Limit -1
  6. ? Message size limit -1
  7. ? Maximum message age limit -1
  8. ? Maximum individual message size -1
  9. ? Replicas 1
  10. ? Adjust mirror start No
  11. ? Import mirror from a different JetStream domain Yes
  12. ? Foreign JetStream domain name leaf
  13. ? Delivery prefix
  14. Stream backup-test-leaf was created
  15. Information for Stream backup-test-leaf created 2021-06-28T14:00:43-04:00
  16. Configuration:
  17. Acknowledgements: true
  18. Retention: File - Limits
  19. Replicas: 1
  20. Discard Policy: Old
  21. Duplicate Window: 2m0s
  22. Maximum Messages: unlimited
  23. Maximum Bytes: unlimited
  24. Maximum Age: 0.00s
  25. Maximum Message Size: unlimited
  26. Maximum Consumers: unlimited
  27. Mirror: test, API Prefix: $JS.leaf.API, Delivery Prefix:
  28. State:
  29. Messages: 0
  30. Bytes: 0 B
  31. FirstSeq: 0
  32. LastSeq: 0
  33. Active Consumers: 0

Similarly, if you want to aggregate streams located in any number of leaf nodes use source. If the streams located in each leaf are used for the same reasons, it is recommended to aggregate them in the hub for processing via source.

  1. nats --server nats://acc:acc@localhost:4222 stream add --js-domain hub --source test
  1. ? Stream Name aggregate-test-leaf
  2. ? Storage backend file
  3. ? Retention Policy Limits
  4. ? Discard Policy Old
  5. ? Stream Messages Limit -1
  6. ? Message size limit -1
  7. ? Maximum message age limit -1
  8. ? Maximum individual message size -1
  9. ? Duplicate tracking time window 2m
  10. ? Replicas 1
  11. ? Adjust source "test" start No
  12. ? Import "test" from a different JetStream domain Yes
  13. ? test Source foreign JetStream domain name leaf
  14. ? test Source foreign JetStream domain delivery prefix
  15. Stream aggregate-test-leaf was created
  16. Information for Stream aggregate-test-leaf created 2021-06-28T14:02:36-04:00
  17. Configuration:
  18. Acknowledgements: true
  19. Retention: File - Limits
  20. Replicas: 1
  21. Discard Policy: Old
  22. Duplicate Window: 2m0s
  23. Maximum Messages: unlimited
  24. Maximum Bytes: unlimited
  25. Maximum Age: 0.00s
  26. Maximum Message Size: unlimited
  27. Maximum Consumers: unlimited
  28. Sources: test, API Prefix: $JS.leaf.API, Delivery Prefix:
  29. State:
  30. Messages: 0
  31. Bytes: 0 B
  32. FirstSeq: 0
  33. LastSeq: 0
  34. Active Consumers: 0

source as well as mirror take a copy of the messages. Once copied, accessing the data is independent of the leaf node connection being online. Copying this way also avoids having to run a dedicated program of your own. This is the recommended way to exchange persistent data across domains.

  1. nats --server nats://acc:acc@localhost:4222 stream report --js-domain hub
  1. Obtaining Stream stats
  2. ╭──────────────────────────────────────────────────────────────────────────────────────────╮
  3. Stream Report
  4. ├─────────────────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
  5. Stream Storage Consumers Messages Bytes Lost Deleted Replicas
  6. ├─────────────────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
  7. backup-test-leaf File 0 1 45 B 0 0
  8. test File 0 1 45 B 0 0
  9. aggregate-test-leaf File 0 1 98 B 0 0
  10. ╰─────────────────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯
  11. ╭────────────────────────────────────────────────────────────────────────────────────╮
  12. Replication Report
  13. ├─────────────────────┬────────┬──────────────┬───────────────┬────────┬─────┬───────┤
  14. Stream Kind API Prefix Source Stream Active Lag Error
  15. ├─────────────────────┼────────┼──────────────┼───────────────┼────────┼─────┼───────┤
  16. backup-test-leaf Mirror $JS.leaf.API test 0.21s 0
  17. aggregate-test-leaf Source $JS.leaf.API test 1.23s 0
  18. ╰─────────────────────┴────────┴──────────────┴───────────────┴────────┴─────┴───────╯

Cross account & domain import

All of the above happened in the same account. To share domain access across accounts the account.conf from above needs to be modified and the server restarted or reloaded. This example exports the consumer and FC API as well as a delivery subject which is used by the internal push consumer created by source and mirror.

In support of another example on how to share a durable pull consumer for client access across domains and accounts, the NEXT and ACK API are exported as well.

Known issue: Currently, across accounts, push consumer are not supported.

On import, the JetStream API prefix $JS.hub.API is renamed to JS.test@hub.API. This is to, once more, disambiguate which JetStream a client in the importing account might want to interact with. When using domains, the general recommendation is to export the domain specific API $JS.<domain>.API as this allows you to pin the export to a particular domain.

Furthermore, the delivery subject is extended on import. This is to allow for easier export into multiple accounts.

This example also exports the absolute minimum necessary. It is possible to give access to the entire consumer API $JS.hub.API.CONSUMER.> or the entire API in a domain $JS.hub.API.> or the entire API $JS.API.> wherever the importing client connects.

  1. accounts {
  2. SYS: {
  3. users: [{user: admin, password: admin}]
  4. },
  5. ACC: {
  6. users: [{user: acc, password: acc}],
  7. jetstream: enabled
  8. exports: [
  9. # minimum export needed to allow source/mirror to create a consumer on the fly
  10. {service: "$JS.hub.API.CONSUMER.CREATE.*", response_type: "stream"}
  11. # minimum export needed for push consumer. This includes source and mirror!
  12. {stream: "deliver.acc.hub.>"}
  13. # minimum export needed for durable pull consumer `dur` in stream `aggregate-test-leaf`. (clients only - source and mirror do not use this)
  14. {service: "$JS.hub.API.CONSUMER.MSG.NEXT.aggregate-test-leaf.dur", response_type: "stream"}
  15. # minimum export needed to ack messages for durable consumer `dur` in stream `aggregate-test-leaf`. (clients only - source and mirror do not use this)
  16. {service: "$JS.ACK.aggregate-test-leaf.dur.>"}
  17. # minimum export needed for flow control of source/mirror
  18. {service: "$JS.FC.aggregate-test-leaf.dur.>"}
  19. ]
  20. }
  21. IMPORT_MIRROR: {
  22. users: [{user: import_mirror, password: import_mirror}],
  23. jetstream: enabled
  24. imports: [
  25. {service: {account: ACC, subject: "$JS.hub.API.CONSUMER.CREATE.*"}, to: "JS.acc@hub.API.CONSUMER.CREATE.*" }
  26. {service: {account: ACC, subject: "$JS.FC.aggregate-test-leaf.dur.>"}}
  27. {stream: {account: ACC, subject: deliver.acc.hub.import_mirror.>}}
  28. ]
  29. }
  30. # As of now, cross account, only pull consumer are supported.
  31. IMPORT_CLIENT: {
  32. users: [{user: import_client, password: import_client}],
  33. jetstream: enabled
  34. imports: [
  35. {service: {account: ACC, subject: "$JS.hub.API.CONSUMER.MSG.NEXT.aggregate-test-leaf.dur"}, to: "JS.acc@hub.API.CONSUMER.MSG.NEXT.aggregate-test-leaf.dur" }
  36. {service: {account: ACC, subject: "$JS.ACK.aggregate-test-leaf.dur.>"}}
  37. ]
  38. }
  39. }
  40. system_account: SYS

Copying via source and mirror

Once the servers have been restarted or reloaded, a mirror can be created as follows (same applies to source): On import from a different account the renamed prefix JS.acc@hub.API is provided. In addition, the delivery subject name is extended to also include the importing domain and stream. This makes it unique to that particular import. If every delivery prefix follows the pattern <static type>.<exporting account>.<exporting domain>.<importing account>.<importing domain>.<importing domain>.<importing stream name> overlaps caused by multiple imports are avoided.

  1. nats --server nats://import_mirror:import_mirror@localhost:4222 stream add --js-domain hub --mirror aggregate-test-leaf
  1. ? Stream Name aggregate-test-leaf-from-acc
  2. ? Storage backend file
  3. ? Retention Policy Limits
  4. ? Discard Policy Old
  5. ? Stream Messages Limit -1
  6. ? Message size limit -1
  7. ? Maximum message age limit -1
  8. ? Maximum individual message size -1
  9. ? Replicas 1
  10. ? Adjust mirror start No
  11. ? Import mirror from a different JetStream domain No
  12. ? Import mirror from a different account Yes
  13. ? Foreign account API prefix JS.acc@hub.API
  14. ? Foreign account delivery prefix deliver.acc.hub.import_mirror.hub.aggregate-test-leaf-from-acc
  15. Stream aggregate-test-leaf-from-acc was created
  16. Information for Stream aggregate-test-leaf-from-acc created 2021-06-28T16:59:15-04:00
  17. Configuration:
  18. Acknowledgements: true
  19. Retention: File - Limits
  20. Replicas: 1
  21. Discard Policy: Old
  22. Duplicate Window: 2m0s
  23. Maximum Messages: unlimited
  24. Maximum Bytes: unlimited
  25. Maximum Age: 0.00s
  26. Maximum Message Size: unlimited
  27. Maximum Consumers: unlimited
  28. Mirror: aggregate-test-leaf, API Prefix: JS.acc@hub.API, Delivery Prefix: deliver.acc.hub.import_mirror.hub.aggregate-test-leaf-from-acc
  29. State:
  30. Messages: 0
  31. Bytes: 0 B
  32. FirstSeq: 0
  33. LastSeq: 0
  34. Active Consumers: 0

A subsequent check shows that the one message stored in the stream aggregate in account ACC got copied to the new stream in the account IMPORTER.

  1. nats --server nats://import_mirror:import_mirror@localhost:4222 stream report --js-domain hub
  1. Obtaining Stream stats
  2. ╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
  3. Stream Report
  4. ├──────────────────────────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
  5. Stream Storage Consumers Messages Bytes Lost Deleted Replicas
  6. ├──────────────────────────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
  7. aggregate-test-leaf-from-acc File 0 1 98 B 0 0
  8. ╰──────────────────────────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯
  9. ╭─────────────────────────────────────────────────────────────────────────────────────────────────────╮
  10. Replication Report
  11. ├──────────────────────────────┬────────┬────────────────┬─────────────────────┬────────┬─────┬───────┤
  12. Stream Kind API Prefix Source Stream Active Lag Error
  13. ├──────────────────────────────┼────────┼────────────────┼─────────────────────┼────────┼─────┼───────┤
  14. aggregate-test-leaf-from-acc Mirror JS.acc@hub.API aggregate-test-leaf 0.59s 0
  15. ╰──────────────────────────────┴────────┴────────────────┴─────────────────────┴────────┴─────┴───────╯

Direct access of a durable pull consumer

The modified accounts.conf also includes a separate import for an existing pull consumer. Let’s create a consumer by the name dur in the stream aggregate-test-leaf in the account acc.

  1. nats --server nats://acc:acc@localhost:4222 consumer add --js-domain hub
  1. ? Consumer name dur
  2. ? Delivery target (empty for Pull Consumers)
  3. ? Start policy (all, new, last, 1h, msg sequence) all
  4. ? Replay policy instant
  5. ? Filter Stream by subject (blank for all)
  6. ? Maximum Allowed Deliveries -1
  7. ? Maximum Acknowledgements Pending 0
  8. ? Select a Stream aggregate-test-leaf
  9. Information for Consumer aggregate-test-leaf > dur created 2021-06-28T17:16:51-04:00
  10. Configuration:
  11. Durable Name: dur
  12. Pull Mode: true
  13. Deliver All: true
  14. Ack Policy: Explicit
  15. Ack Wait: 30s
  16. Replay Policy: Instant
  17. Max Ack Pending: 20,000
  18. Max Waiting Pulls: 512
  19. State:
  20. Last Delivered Message: Consumer sequence: 0 Stream sequence: 0
  21. Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
  22. Outstanding Acks: 0 out of maximum 20000
  23. Redelivered Messages: 0
  24. Unprocessed Messages: 1
  25. Waiting Pulls: 0 of maximum 512
  1. nats --server nats://acc:acc@localhost:4222 stream report --js-domain hub
  1. Obtaining Stream stats
  2. ╭──────────────────────────────────────────────────────────────────────────────────────────╮
  3. Stream Report
  4. ├─────────────────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
  5. Stream Storage Consumers Messages Bytes Lost Deleted Replicas
  6. ├─────────────────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
  7. backup-test-leaf File 0 1 45 B 0 0
  8. test File 0 1 45 B 0 0
  9. aggregate-test-leaf File 1 1 98 B 0 0
  10. ╰─────────────────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯
  11. ╭────────────────────────────────────────────────────────────────────────────────────╮
  12. Replication Report
  13. ├─────────────────────┬────────┬──────────────┬───────────────┬────────┬─────┬───────┤
  14. Stream Kind API Prefix Source Stream Active Lag Error
  15. ├─────────────────────┼────────┼──────────────┼───────────────┼────────┼─────┼───────┤
  16. backup-test-leaf Mirror $JS.leaf.API test 1.85s 0
  17. aggregate-test-leaf Source $JS.leaf.API test 1.85s 0
  18. ╰─────────────────────┴────────┴──────────────┴───────────────┴────────┴─────┴───────╯
  1. nats --server nats://acc:acc@localhost:4222 consumer report --js-domain hub

Output

  1. ? Select a Stream aggregate-test-leaf
  2. ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
  3. Consumer report for aggregate-test-leaf with 1 consumers
  4. ├──────────┬──────┬────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬─────────────┤
  5. Consumer Mode Ack Policy Ack Wait Ack Pending Redelivered Unprocessed Ack Floor Cluster
  6. ├──────────┼──────┼────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼─────────────┤
  7. dur Pull Explicit 30.00s 0 0 1 / 100% 0 hub-server*
  8. ╰──────────┴──────┴────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴─────────────╯

To retrieve the messages stored in the domain hub using nats while connected to the leaf node, provide the correct stream and durable name as well as the API prefix JS.acc@hub.API

  1. nats --server nats://import_client:import_client@localhost:4111 consumer next aggregate-test-leaf dur --js-api-prefix JS.acc@hub.API
  1. [17:44:16] subj: test / tries: 1 / cons seq: 1 / str seq: 1 / pending: 0
  2. Headers:
  3. Nats-Stream-Source: test:mSx7q4yJ 1
  4. Data:
  5. hello world
  6. Acknowledged message
  1. nats --server nats://acc:acc@localhost:4222 consumer report --js-domain hub
  1. ? Select a Stream aggregate-test-leaf
  2. ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
  3. Consumer report for aggregate-test-leaf with 1 consumers
  4. ├──────────┬──────┬────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬─────────────┤
  5. Consumer Mode Ack Policy Ack Wait Ack Pending Redelivered Unprocessed Ack Floor Cluster
  6. ├──────────┼──────┼────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼─────────────┤
  7. dur Pull Explicit 30.00s 0 0 0 1 hub-server*
  8. ╰──────────┴──────┴────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴─────────────╯

This works similarly when writing your own client. To avoid waiting for the ack timeout, a new message is sent on test from where it is copied into aggregate-test-leaf.

  1. nats --server nats://acc:acc@localhost:4222 pub test "hello world 2"

The client is connected to the leaf node and receives the message just sent.

  1. ./main nats://import_client:import_client@localhost:4111
  1. starting
  2. &{Sequence:{Consumer:3 Stream:3} NumDelivered:1 NumPending:0 Timestamp:2021-06-28 17:51:05.186878 -0400 EDT Stream:aggregate-test-leaf Consumer:dur}
  3. hello world 2
  4. nats: timeout
  5. ^Cnats: timeout

There the API prefix is communicated with setting the option nats.APIPrefix("JS.acc@hub.API") when obtaining the JetStream object. Because the API access is limited, the subscribe call provides the option nats.Bind("aggregate-test-leaf", "dur") which prevents calls to infer the stream and durable name.

  1. package main
  2. import (
  3. "fmt"
  4. "os"
  5. "os/signal"
  6. "syscall"
  7. "time"
  8. "github.com/nats-io/nats.go"
  9. )
  10. func main() {
  11. nc, err := nats.Connect(os.Args[1], nats.Name("JS test"))
  12. defer nc.Close()
  13. if err != nil {
  14. fmt.Printf("nats connect: %v\n", err)
  15. return
  16. }
  17. js, err := nc.JetStream(nats.APIPrefix("JS.acc@hub.API"))
  18. if err != nil {
  19. fmt.Printf("JetStream: %v\n", err)
  20. if js == nil {
  21. return
  22. }
  23. }
  24. s, err := js.PullSubscribe("", "dur", nats.Bind("aggregate-test-leaf", "dur"))
  25. if err != nil {
  26. fmt.Printf("PullSubscribe: %v\n", err)
  27. return
  28. }
  29. shutdown := make(chan os.Signal, 1)
  30. signal.Notify(shutdown, os.Interrupt, syscall.SIGTERM)
  31. fmt.Printf("starting\n")
  32. for {
  33. select {
  34. case <-shutdown:
  35. return
  36. default:
  37. if m, err := s.Fetch(1, nats.MaxWait(time.Second)); err != nil {
  38. fmt.Println(err)
  39. } else {
  40. if meta, err := m[0].Metadata(); err == nil {
  41. fmt.Printf("%+v\n", meta)
  42. }
  43. fmt.Println(string(m[0].Data))
  44. if err := m[0].Ack(); err != nil {
  45. fmt.Printf("ack error: %+v\n", err)
  46. }
  47. }
  48. }
  49. }
  50. }

A push subscriber will need a similar setup. It will require the ACK subject. However, instead of exporting/importing the NEXT subject, the delivery subject shown for source/mirror needs to be used.

See Also

JetStream Leaf Nodes demo and its companion video recording.