NATS Streaming Cluster with FT Mode

NATS Streaming Cluster with FT Mode on AWS

Preparation

First, we need a Kubernetes cluster with a provider that offers a service with a ReadWriteMany filesystem available. In this short guide, we will create the cluster on AWS and then use EFS for the filesystem:

  1. # Create 3 nodes Kubernetes cluster
  2. eksctl create cluster --name stan-k8s \
  3. --nodes 3 \
  4. --node-type=t3.large \ # t3.small
  5. --region=us-east-2
  6. # Get the credentials for your cluster
  7. eksctl utils write-kubeconfig --name stan-k8s --region us-east-2

For the FT mode to work, we will need to create an EFS volume which can be shared by more than one pod. Go into the AWS console and create one and make the sure that it is in a security group where the k8s nodes will have access to it. In case of clusters created via eksctl, this will be a security group named ClusterSharedNodeSecurityGroup:

Screen Shot 2019-12-04 at 11 25 08 AM

Screen Shot 2019-12-04 at 12 40 13 PM

Creating the EFS provisioner

Confirm from the FilesystemID from the cluster and the DNS name, we will use those values to create an EFS provisioner controller within the K8S cluster:

Screen Shot 2019-12-04 at 12 08 35 PM

  1. ---
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5. name: efs-provisioner
  6. ---
  7. kind: ClusterRole
  8. apiVersion: rbac.authorization.k8s.io/v1
  9. metadata:
  10. name: efs-provisioner-runner
  11. rules:
  12. - apiGroups: [""]
  13. resources: ["persistentvolumes"]
  14. verbs: ["get", "list", "watch", "create", "delete"]
  15. - apiGroups: [""]
  16. resources: ["persistentvolumeclaims"]
  17. verbs: ["get", "list", "watch", "update"]
  18. - apiGroups: ["storage.k8s.io"]
  19. resources: ["storageclasses"]
  20. verbs: ["get", "list", "watch"]
  21. - apiGroups: [""]
  22. resources: ["events"]
  23. verbs: ["create", "update", "patch"]
  24. ---
  25. kind: ClusterRoleBinding
  26. apiVersion: rbac.authorization.k8s.io/v1
  27. metadata:
  28. name: run-efs-provisioner
  29. subjects:
  30. - kind: ServiceAccount
  31. name:
  32. efs-provisioner
  33. # replace with namespace where provisioner is deployed
  34. namespace: default
  35. roleRef:
  36. kind: ClusterRole
  37. name: efs-provisioner-runner
  38. apiGroup: rbac.authorization.k8s.io
  39. ---
  40. kind: Role
  41. apiVersion: rbac.authorization.k8s.io/v1
  42. metadata:
  43. name: leader-locking-efs-provisioner
  44. rules:
  45. - apiGroups: [""]
  46. resources: ["endpoints"]
  47. verbs: ["get", "list", "watch", "create", "update", "patch"]
  48. ---
  49. kind: RoleBinding
  50. apiVersion: rbac.authorization.k8s.io/v1
  51. metadata:
  52. name: leader-locking-efs-provisioner
  53. subjects:
  54. - kind: ServiceAccount
  55. name: efs-provisioner
  56. # replace with namespace where provisioner is deployed
  57. namespace: default
  58. roleRef:
  59. kind: Role
  60. name: leader-locking-efs-provisioner
  61. apiGroup: rbac.authorization.k8s.io
  62. ---
  63. apiVersion: v1
  64. kind: ConfigMap
  65. metadata:
  66. name: efs-provisioner
  67. data:
  68. file.system.id: fs-c22a24bb
  69. aws.region: us-east-2
  70. provisioner.name: synadia.com/aws-efs
  71. dns.name: ""
  72. ---
  73. kind: Deployment
  74. apiVersion: apps/v1
  75. metadata:
  76. name: efs-provisioner
  77. spec:
  78. replicas: 1
  79. selector:
  80. matchLabels:
  81. app: efs-provisioner
  82. strategy:
  83. type: Recreate
  84. template:
  85. metadata:
  86. labels:
  87. app: efs-provisioner
  88. spec:
  89. serviceAccountName: efs-provisioner
  90. containers:
  91. - name: efs-provisioner
  92. image: quay.io/external_storage/efs-provisioner:latest
  93. env:
  94. - name: FILE_SYSTEM_ID
  95. valueFrom:
  96. configMapKeyRef:
  97. name: efs-provisioner
  98. key: file.system.id
  99. - name: AWS_REGION
  100. valueFrom:
  101. configMapKeyRef:
  102. name: efs-provisioner
  103. key: aws.region
  104. - name: DNS_NAME
  105. valueFrom:
  106. configMapKeyRef:
  107. name: efs-provisioner
  108. key: dns.name
  109. - name: PROVISIONER_NAME
  110. valueFrom:
  111. configMapKeyRef:
  112. name: efs-provisioner
  113. key: provisioner.name
  114. volumeMounts:
  115. - name: pv-volume
  116. mountPath: /efs
  117. volumes:
  118. - name: pv-volume
  119. nfs:
  120. server: fs-c22a24bb.efs.us-east-2.amazonaws.com
  121. path: /
  122. ---
  123. kind: StorageClass
  124. apiVersion: storage.k8s.io/v1
  125. metadata:
  126. name: aws-efs
  127. provisioner: synadia.com/aws-efs
  128. ---
  129. kind: PersistentVolumeClaim
  130. apiVersion: v1
  131. metadata:
  132. name: efs
  133. annotations:
  134. volume.beta.kubernetes.io/storage-class: "aws-efs"
  135. spec:
  136. accessModes:
  137. - ReadWriteMany
  138. resources:
  139. requests:
  140. storage: 1Mi

Result of deploying the manifest:

  1. serviceaccount/efs-provisioner created
  2. clusterrole.rbac.authorization.k8s.io/efs-provisioner-runner created
  3. clusterrolebinding.rbac.authorization.k8s.io/run-efs-provisioner created
  4. role.rbac.authorization.k8s.io/leader-locking-efs-provisioner created
  5. rolebinding.rbac.authorization.k8s.io/leader-locking-efs-provisioner created
  6. configmap/efs-provisioner created
  7. deployment.extensions/efs-provisioner created
  8. storageclass.storage.k8s.io/aws-efs created
  9. persistentvolumeclaim/efs created

Setting up the NATS Streaming cluster

Now create a NATS Streaming cluster with FT mode enabled and using NATS embedded mode that is mounting the EFS volume:

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5. name: stan
  6. labels:
  7. app: stan
  8. spec:
  9. selector:
  10. app: stan
  11. clusterIP: None
  12. ports:
  13. - name: client
  14. port: 4222
  15. - name: cluster
  16. port: 6222
  17. - name: monitor
  18. port: 8222
  19. - name: metrics
  20. port: 7777
  21. ---
  22. apiVersion: v1
  23. kind: ConfigMap
  24. metadata:
  25. name: stan-config
  26. data:
  27. stan.conf: |
  28. http: 8222
  29. cluster {
  30. port: 6222
  31. routes [
  32. nats://stan-0.stan:6222
  33. nats://stan-1.stan:6222
  34. nats://stan-2.stan:6222
  35. ]
  36. cluster_advertise: $CLUSTER_ADVERTISE
  37. connect_retries: 10
  38. }
  39. streaming {
  40. id: test-cluster
  41. store: file
  42. dir: /data/stan/store
  43. ft_group_name: "test-cluster"
  44. file_options {
  45. buffer_size: 32mb
  46. sync_on_flush: false
  47. slice_max_bytes: 512mb
  48. parallel_recovery: 64
  49. }
  50. store_limits {
  51. max_channels: 10
  52. max_msgs: 0
  53. max_bytes: 256gb
  54. max_age: 1h
  55. max_subs: 128
  56. }
  57. }
  58. ---
  59. apiVersion: apps/v1
  60. kind: StatefulSet
  61. metadata:
  62. name: stan
  63. labels:
  64. app: stan
  65. spec:
  66. selector:
  67. matchLabels:
  68. app: stan
  69. serviceName: stan
  70. replicas: 3
  71. volumeClaimTemplates:
  72. template:
  73. metadata:
  74. labels:
  75. app: stan
  76. spec:
  77. # STAN Server
  78. terminationGracePeriodSeconds: 30
  79. containers:
  80. - name: stan
  81. image: nats-streaming:alpine
  82. ports:
  83. # In case of NATS embedded mode expose these ports
  84. - containerPort: 4222
  85. name: client
  86. - containerPort: 6222
  87. name: cluster
  88. - containerPort: 8222
  89. name: monitor
  90. args:
  91. - "-sc"
  92. - "/etc/stan-config/stan.conf"
  93. # Required to be able to define an environment variable
  94. # that refers to other environment variables. This env var
  95. # is later used as part of the configuration file.
  96. env:
  97. - name: POD_NAME
  98. valueFrom:
  99. fieldRef:
  100. fieldPath: metadata.name
  101. - name: POD_NAMESPACE
  102. valueFrom:
  103. fieldRef:
  104. fieldPath: metadata.namespace
  105. - name: CLUSTER_ADVERTISE
  106. value: $(POD_NAME).stan.$(POD_NAMESPACE).svc
  107. volumeMounts:
  108. - name: config-volume
  109. mountPath: /etc/stan-config
  110. - name: efs
  111. mountPath: /data/stan
  112. resources:
  113. requests:
  114. cpu: 0
  115. livenessProbe:
  116. httpGet:
  117. path: /
  118. port: 8222
  119. initialDelaySeconds: 10
  120. timeoutSeconds: 5
  121. - name: metrics
  122. image: synadia/prometheus-nats-exporter:0.5.0
  123. args:
  124. - -connz
  125. - -routez
  126. - -subz
  127. - -varz
  128. - -channelz
  129. - -serverz
  130. - http://localhost:8222
  131. ports:
  132. - containerPort: 7777
  133. name: metrics
  134. volumes:
  135. - name: config-volume
  136. configMap:
  137. name: stan-config
  138. - name: efs
  139. persistentVolumeClaim:
  140. claimName: efs

Your cluster now will look something like this:

  1. kubectl get pods
  2. NAME READY STATUS RESTARTS AGE
  3. efs-provisioner-6b7866dd4-4k5wx 1/1 Running 0 21m
  4. stan-0 2/2 Running 0 6m35s
  5. stan-1 2/2 Running 0 4m56s
  6. stan-2 2/2 Running 0 4m42s

If everything was setup properly, one of the servers will be the active node.

  1. $ kubectl logs stan-0 -c stan
  2. [1] 2019/12/04 20:40:40.429359 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.16.2
  3. [1] 2019/12/04 20:40:40.429385 [INF] STREAM: ServerID: 7j3t3Ii7e2tifWqanYKwFX
  4. [1] 2019/12/04 20:40:40.429389 [INF] STREAM: Go version: go1.11.13
  5. [1] 2019/12/04 20:40:40.429392 [INF] STREAM: Git commit: [910d6e1]
  6. [1] 2019/12/04 20:40:40.454212 [INF] Starting nats-server version 2.0.4
  7. [1] 2019/12/04 20:40:40.454360 [INF] Git commit [c8ca58e]
  8. [1] 2019/12/04 20:40:40.454522 [INF] Starting http monitor on 0.0.0.0:8222
  9. [1] 2019/12/04 20:40:40.454830 [INF] Listening for client connections on 0.0.0.0:4222
  10. [1] 2019/12/04 20:40:40.454841 [INF] Server id is NB3A5RSGABLJP3WUYG6VYA36ZGE7MP5GVQIQVRG6WUYSRJA7B2NNMW57
  11. [1] 2019/12/04 20:40:40.454844 [INF] Server is ready
  12. [1] 2019/12/04 20:40:40.456360 [INF] Listening for route connections on 0.0.0.0:6222
  13. [1] 2019/12/04 20:40:40.481927 [INF] STREAM: Starting in standby mode
  14. [1] 2019/12/04 20:40:40.488193 [ERR] Error trying to connect to route (attempt 1): dial tcp: lookup stan on 10.100.0.10:53: no such host
  15. [1] 2019/12/04 20:40:41.489688 [INF] 192.168.52.76:40992 - rid:6 - Route connection created
  16. [1] 2019/12/04 20:40:41.489788 [INF] 192.168.52.76:40992 - rid:6 - Router connection closed
  17. [1] 2019/12/04 20:40:41.489695 [INF] 192.168.52.76:6222 - rid:5 - Route connection created
  18. [1] 2019/12/04 20:40:41.489955 [INF] 192.168.52.76:6222 - rid:5 - Router connection closed
  19. [1] 2019/12/04 20:40:41.634944 [INF] STREAM: Server is active
  20. [1] 2019/12/04 20:40:41.634976 [INF] STREAM: Recovering the state...
  21. [1] 2019/12/04 20:40:41.655526 [INF] STREAM: No recovered state
  22. [1] 2019/12/04 20:40:41.671435 [INF] STREAM: Message store is FILE
  23. [1] 2019/12/04 20:40:41.671448 [INF] STREAM: Store location: /data/stan/store
  24. [1] 2019/12/04 20:40:41.671524 [INF] STREAM: ---------- Store Limits ----------
  25. [1] 2019/12/04 20:40:41.671527 [INF] STREAM: Channels: 10
  26. [1] 2019/12/04 20:40:41.671529 [INF] STREAM: --------- Channels Limits --------
  27. [1] 2019/12/04 20:40:41.671531 [INF] STREAM: Subscriptions: 128
  28. [1] 2019/12/04 20:40:41.671533 [INF] STREAM: Messages : unlimited
  29. [1] 2019/12/04 20:40:41.671535 [INF] STREAM: Bytes : 256.00 GB
  30. [1] 2019/12/04 20:40:41.671537 [INF] STREAM: Age : 1h0m0s
  31. [1] 2019/12/04 20:40:41.671539 [INF] STREAM: Inactivity : unlimited *
  32. [1] 2019/12/04 20:40:41.671541 [INF] STREAM: ----------------------------------
  33. [1] 2019/12/04 20:40:41.671546 [INF] STREAM: Streaming Server is ready

NATS Streaming Cluster with FT Mode on Azure

First need to create a PVC (PersistentVolumeClaim), in Azure we can use azurefile to get a volume with ReadWriteMany:

  1. ---
  2. kind: PersistentVolumeClaim
  3. apiVersion: v1
  4. metadata:
  5. name: stan-efs
  6. annotations:
  7. volume.beta.kubernetes.io/storage-class: "azurefile"
  8. spec:
  9. accessModes:
  10. - ReadWriteMany
  11. resources:
  12. requests:
  13. storage: 100Mi

Next create a NATS cluster using the Helm charts:

  1. helm repo add nats https://nats-io.github.io/k8s/helm/charts/
  2. helm install nats nats/nats

To create an FT setup using AzureFile you can use the following Helm chart values file:

  1. stan:
  2. image: nats-streaming:alpine
  3. replicas: 2
  4. nats:
  5. url: nats://nats:4222
  6. store:
  7. type: file
  8. ft:
  9. group: my-group
  10. file:
  11. path: /data/stan/store
  12. volume:
  13. enabled: true
  14. # Mount path for the volume.
  15. mount: /data/stan
  16. # FT mode requires a single shared ReadWriteMany PVC volume.
  17. persistentVolumeClaim:
  18. claimName: stan-efs

Now deploy with Helm:

  1. helm install stan nats/stan -f ./examples/deploy-stan-ft-file.yaml

Send a few commands to the NATS Server to which STAN/NATS Streaming is connected:

  1. kubectl port-forward nats-0 4222:4222 &
  2. stan-pub -c stan foo bar.1
  3. stan-pub -c stan foo bar.2
  4. stan-pub -c stan foo bar.3

Subscribe to get all the messages:

  1. stan-sub -c stan -all foo