Creating a NATS Super Cluster in Digital Ocean with Helm

Let’s create a super cluster using NATS Gateways. First let’s create 3 different clusters in NYC, Amsterdam, and San Francisco:

  1. doctl kubernetes cluster create nats-k8s-nyc1 --count 3 --region nyc1
  2. doctl kubernetes cluster create nats-k8s-sfo2 --count 3 --region sfo2
  3. doctl kubernetes cluster create nats-k8s-ams3 --count 3 --region ams3

Next, open up the firewall across the 3 regions to be able to access the client, leafnode and gateways ports:

  1. for firewall in `doctl compute firewall list | tail -n 3 | awk '{print $1}'`; do
  2.   doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:4222,address:0.0.0.0/0
  3.   doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7422,address:0.0.0.0/0
  4.   doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7522,address:0.0.0.0/0
  5. done

For this setup, we will create a super cluster using the external IPs from the nodes of the 3 clusters. For a production type of setup, it is recommended to use a DNS entry and an A record for each one of the servers.

  1. for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
  2.   echo "name: $ctx"
  3.   for externalIP in `kubectl --context $ctx get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'`; do 
  4.     echo "- nats://$externalIP:7522"; 
  5.   done
  6.   echo
  7. done

The Helm definition would look as follows for the 3 clusters:

  1. # super-cluster.yaml
  2. nats:
  3.   externalAccess: true
  4.   logging:
  5.     debug: false
  6.     trace: false
  8. cluster:
  9.   enabled: true
  11. gateway:
  12.   enabled: true
  14.   # NOTE: defined via --set gateway.name="$ctx"
  15.   # name: $ctx
  17.   gateways:
  18.   - name: do-ams3-nats-k8s-ams3
  19.     urls:
  20.     - nats://142.93.251.181:7522
  21.     - nats://161.35.12.245:7522
  22.     - nats://161.35.2.153:7522
  24.   - name: do-nyc1-nats-k8s-nyc1
  25.     urls:
  26.     - nats://142.93.251.181:7522
  27.     - nats://161.35.12.245:7522
  28.     - nats://161.35.2.153:7522
  30.   - name: do-sfo2-nats-k8s-sfo2
  31.     urls:
  32.     - nats://142.93.251.181:7522
  33.     - nats://161.35.12.245:7522
  34.     - nats://161.35.2.153:7522
  36. natsbox:
  37.   enabled: true

Let’s deploy the super cluster with Helm using the name of cluster as the name of the gateway:

  1. for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
  2.   helm --kube-context $ctx install nats nats/nats -f super-cluster.yaml --set gateway.name=$ctx
  3. done

That’s it! It should now be possible to send some messages across regions:

  1. # Start subscription in Amsterdam
  2. nats-box:~# kubectl --context do-ams3-nats-k8s-ams3 exec -it nats-box -- /bin/sh -l
  3. nats-box:~# nats sub -s nats hello
  5. # Send messages from San Francisco region
  6. nats-box:~# kubectl --context do-sfo2-nats-k8s-sfo2 exec -it nats-box -- /bin/sh -l
  7. nats-box:~# nats pub -s nats hello 'Hello World!'
  9. # From outside of k8s can use the external IPs
  10. nats sub -s 142.93.251.181 hello
  11. nats pub -s 161.35.2.153 hello 'Hello World!'

Using leafnodes and NATS super clusters to communicate across regions

You can also create a multi-region NATS topology by using leafnodes connecting to a NATS super cluster (which could also be a much simpler way!).

  1. doctl kubernetes cluster create nats-k8s-nyc1 --count 3 --region nyc1
  2. doctl kubernetes cluster create nats-k8s-sfo2 --count 3 --region sfo2
  3. doctl kubernetes cluster create nats-k8s-ams3 --count 3 --region ams3

Next, open up the firewall across the 3 regions to be able to access the client, leafnode and gateways ports:

  1. for firewall in `doctl compute firewall list | tail -n 3 | awk '{print $1}'`; do
  2.   doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:4222,address:0.0.0.0/0
  3. done

The Helm definition would look as follows for the 3 clusters:

  1. # nats.yaml
  2. leafnodes:
  3.   enabled: true
  4.   remotes:
  5.     - url: tls://connect.ngs.global:7422
  6.       credentials:
  7.         secret:
  8.           name: ngs-creds
  9.           key: NGS.creds
  11. natsbox:
  12.   enabled: true

Let’s deploy the super cluster with Helm using the name of cluster as the name of the gateway:

  1. for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
  2.   kubectl --context $ctx create secret generic ngs-creds --from-file $HOME/.nkeys/creds/synadia/NGS/NGS.creds
  3.   helm --kube-context $ctx install nats nats/nats -f nats.yaml
  4. done

It should now be possible to send some messages across regions:

  1. # Start subscription in Amsterdam
  2. nats-box:~# kubectl --context do-ams3-nats-k8s-ams3 exec -it nats-box -- /bin/sh -l
  3. nats-box:~# nats-sub -s nats hello
  5. # Send messages from San Francisco region
  6. nats-box:~# kubectl --context do-sfo2-nats-k8s-sfo2 exec -it nats-box -- /bin/sh -l
  7. nats-box:~# nats-pub -s nats hello 'Hello World!'

Or from outside of k8s can use the external IPs:

  1. # Find the external ips from the nodes
  2. $ for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
  3.   kubectl --context $ctx get nodes -o wide
  4. done
  6. NAME                               STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP      OS-IMAGE                       KERNEL-VERSION         CONTAINER-RUNTIME
  7. nats-k8s-ams3-default-pool-3cifn   Ready    <none>   38m   v1.17.5   10.133.7.58     188.166.32.235   Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2
  8. nats-k8s-ams3-default-pool-3ciq9   Ready    <none>   38m   v1.17.5   10.133.19.19    188.166.40.159   Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2
  9. nats-k8s-ams3-default-pool-3ciqz   Ready    <none>   38m   v1.17.5   10.133.31.211   188.166.43.34    Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2
  10. NAME                               STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP     OS-IMAGE                       KERNEL-VERSION         CONTAINER-RUNTIME
  11. nats-k8s-nyc1-default-pool-3ciq4   Ready    <none>   37m   v1.17.5   10.136.75.202   161.35.125.69   Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2
  12. nats-k8s-nyc1-default-pool-3ciqh   Ready    <none>   38m   v1.17.5   10.136.90.125   161.35.125.70   Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2
  13. nats-k8s-nyc1-default-pool-3ciqk   Ready    <none>   37m   v1.17.5   10.136.65.137   161.35.125.66   Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2
  14. NAME                               STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP      OS-IMAGE                       KERNEL-VERSION         CONTAINER-RUNTIME
  15. nats-k8s-sfo2-default-pool-3ciq0   Ready    <none>   37m   v1.17.5   10.138.20.132    206.189.79.122   Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2
  16. nats-k8s-sfo2-default-pool-3ciqd   Ready    <none>   37m   v1.17.5   10.138.4.194     64.225.124.243   Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2
  17. nats-k8s-sfo2-default-pool-3ciqv   Ready    <none>   37m   v1.17.5   10.138.148.237   206.189.79.131   Debian GNU/Linux 9 (stretch)   4.19.0-0.bpo.6-amd64   docker://18.9.2

Send a message from Amsterdam to SFO via the super cluster connected with leafnodes:

  1. $ nats-sub -s 188.166.32.235 hello        # From Amsterdam
  2. $ nats-pub -s 206.189.79.131 hello world  #   To SFO