OpenEBS for NuoDB

OpenEBS and Nuodb

Introduction

NuoDB’s distributed SQL database combines the elastic scale and continuous availability of the cloud with the transactional consistency and durability that databases of record demand. NuoDB is deployed usually as a StatefulSet on Kubernetes and requires persistent storage for each instance of NuoDB StorageManager instance. OpenEBS provides persistent volumes on the fly when StorageManagers are scaled up.

Advantages of using OpenEBS for NuoDB database:

  • No need to manage the local disks, they are managed by OpenEBS
  • Large size PVs can be provisioned by OpenEBS and NuoDB
  • Start with small storage and add disks as needed on the fly. Sometimes NuoDB instances are scaled up because of capacity on the nodes. With OpenEBS persistent volumes, capacity can be thin provisioned and disks can be added to OpenEBS on the fly without disruption of service
  • If required, take backup of the NuoDB data periodically and back them up to S3 or any object storage so that restoration of the same data is possible to the same or any other Kubernetes cluster

Note: NuoDB can be deployed both as deployment or as statefulset. When NuoDB deployed as statefulset, you don’t need to replicate the data again at OpenEBS level. When NuoDB is deployed as deployment, consider 3 OpenEBS replicas, choose the StorageClass accordingly.

Deployment model

OpenEBS and NuoDB

Configuration workflow

  1. Install OpenEBS

    If OpenEBS is not installed in your K8s cluster, this can done from here. If OpenEBS is already installed, go to the next step.

  2. Connect to Kubera (Optional) : Connecting the Kubernetes cluster to Kubera provides good visibility of storage resources. Kubera has various support options for enterprise customers.

  3. Configure cStor Pool

    After OpenEBS installation, cStor pool has to be configured. If cStor Pool is not configured in your OpenEBS cluster, this can be done from here. Sample YAML named openebs-config.yaml for configuring cStor Pool is provided in the Configuration details below. During cStor Pool creation, make sure that the maxPools parameter is set to >=3. If cStor pool is already configured, go to the next step.

  4. Create Storage Class

    You must configure a StorageClass to provision cStor volume on given cStor pool. StorageClass is the interface through which most of the OpenEBS storage policies are defined. In this solution we are using a StorageClass to consume the cStor Pool which is created using external disks attached on the Nodes. Since NuoDB is a statefulset application, it requires only single storage replica. So cStor volume replicaCount is =1. Sample YAML named openebs-sc-disk.yaml to consume cStor pool with cStoveVolume Replica count as 1 is provided in the configuration details below.

  5. Download the YAML spec files from OpenEBS litmus repository.

    1. wget https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/nuodb.yaml
    3. wget https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/nuodb-sm.yaml
    5. wget https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/nuodb-te.yaml

  6. Edit the above YAML spec files and update them with correct StorageClass. For example, replace openebs-cstor-sparse with openebs-cstor-disk

  7. Launch and test NuoDB

    Create a namespace called testns and apply following YAML files to deploy NuoDB application. Sample YAML files are provided in the Configuration details below.

    1. kubectl apply -f nuodb.yaml -n testns
    2. kubectl apply -f nuodb-sm.yaml -n testns
    3. kubectl apply -f nuodb-te.yaml -n testns

Reference at openebs.ci

Deployment YAML spec files for NuoDB and OpenEBS resources are found here

OpenEBS-CI dashboard of NuoDB

Live access to NuoDB dashboard

Post deployment Operations

Monitor OpenEBS Volume size

It is not seamless to increase the cStor volume size (refer to the roadmap item). Hence, it is recommended that sufficient size is allocated during the initial configuration. However, an alert can be setup for volume size threshold using Kubera.

Monitor cStor Pool size

As in most cases, cStor pool may not be dedicated to just NuoDB database alone. It is recommended to watch the pool capacity and add more disks to the pool before it hits 80% threshold. See cStorPool metrics

Configuration details

openebs-config.yaml

  1. #Use the following YAMLs to create a cStor Storage Pool.
  2. # and associated storage class.
  3. apiVersion: openebs.io/v1alpha1
  4. kind: StoragePoolClaim
  5. metadata:
  6.   name: cstor-disk
  7. spec:
  8.   name: cstor-disk
  9.   type: disk
  10.   poolSpec:
  11.     poolType: striped
  12.   # NOTE - Appropriate disks need to be fetched using `kubectl get disks`
  13.   #
  14.   # `Disk` is a custom resource supported by OpenEBS with `node-disk-manager`
  15.   # as the disk operator
  16. # Replace the following with actual disk CRs from your cluster `kubectl get disks`
  17. # Uncomment the below lines after updating the actual disk names.
  18.   disks:
  19.     diskList:
  20. # Replace the following with actual disk CRs from your cluster from `kubectl get disks`
  21. #   - disk-184d99015253054c48c4aa3f17d137b1
  22. #   - disk-2f6bced7ba9b2be230ca5138fd0b07f1
  23. #   - disk-806d3e77dd2e38f188fdaf9c46020bdc
  24. #   - disk-8b6fb58d0c4e0ff3ed74a5183556424d
  25. #   - disk-bad1863742ce905e67978d082a721d61
  26. #   - disk-d172a48ad8b0fb536b9984609b7ee653
  27. ---

openebs-sc-disk.yaml

  1. apiVersion: storage.k8s.io/v1
  2. kind: StorageClass
  3. metadata:
  4.   name: openebs-cstor-disk
  5.   annotations:
  6.     openebs.io/cas-type: cstor
  7.     cas.openebs.io/config: |
  8.       - name: StoragePoolClaim
  9.         value: "cstor-disk"
  10.       - name: ReplicaCount
  11.         value: "1"       
  12. provisioner: openebs.io/provisioner-iscsi
  13. reclaimPolicy: Delete
  14. ---

nuodb.yaml

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   annotations:
  6.     description: |
  7.       Headless service permitting us to retrieve DNS A records that point directly to the pods backing the service.
  8.     "service.alpha.kubernetes.io/tolerate-unready-endpoints": "true"
  9.   name: domain
  10.   labels:
  11.     app: nuodb
  12.     group: nuodb
  13. spec:
  14.   clusterIP: None
  15.   ports:
  16.   - { name: 8888-tcp,   port: 8888,   protocol: TCP,  targetPort: 8888  }
  17.   - { name: 48004-tcp,  port: 48004,  protocol: TCP,  targetPort: 48004 }
  18.   - { name: 48005-tcp,  port: 48005,  protocol: TCP,  targetPort: 48005 }
  19.   selector:
  20.     dns: domain
  21.   sessionAffinity: None
  22.   type: ClusterIP
  23. status:
  24.   loadBalancer: {}
  26. ---
  27. apiVersion: v1
  28. kind: Service
  29. metadata:
  30.   annotations:
  31.     description: "Service (and load-balancer) for Admin pods."
  32.   labels:
  33.     app: nuodb
  34.     group: nuodb
  35.   name: admin
  36. spec:
  37.   ports:
  38.   - { name: 8888-tcp,   port: 8888,   protocol: TCP,  targetPort: 8888  }
  39.   - { name: 48004-tcp,  port: 48004,  protocol: TCP,  targetPort: 48004 }
  40.   - { name: 48005-tcp,  port: 48005,  protocol: TCP,  targetPort: 48005 }
  41.   selector:
  42.     app: admin
  43.   sessionAffinity: None
  44.   type: LoadBalancer
  45. status:
  46.   loadBalancer: {}
  48. ---
  49. apiVersion: apps/v1beta1
  50. kind: StatefulSet
  51. metadata:
  52.   name: admin
  53.   labels:
  54.     app: nuodb
  55.     group: nuodb
  56. spec:
  57.   serviceName: domain
  58.   replicas: 1
  59.   selector:
  60.     matchLabels:
  61.       affinity: admin
  62.   template:
  63.     metadata:
  64.       labels:
  65.         dns: domain
  66.         app: admin
  67.         affinity: admin
  68.     spec:
  69.       securityContext:
  70.         runAsUser: 0
  71.         fsGroup: 0
  72.       containers:
  73.       - name: admin
  74.         image: nuodb/nuodb-ce:latest
  75.         imagePullPolicy: IfNotPresent
  76.         ports:
  77.         - { containerPort: 8888,  protocol: TCP }
  78.         - { containerPort: 48004, protocol: TCP }
  79.         - { containerPort: 48005, protocol: TCP }
  80.         resources:
  81.           requests:
  82.             memory: "512Mi"
  83.             cpu: "0.5"
  84.           limits:
  85.             memory: "1024Mi"
  86.             cpu: "1"
  87.         env:
  88.         - name: NODE_NAME
  89.           valueFrom:
  90.             fieldRef:
  91.               fieldPath: spec.nodeName
  92.         - name: POD_NAME
  93.           valueFrom:
  94.             fieldRef:
  95.               fieldPath: metadata.name
  96.         - name: NAMESPACE
  97.           valueFrom:
  98.             fieldRef:
  99.               fieldPath: metadata.namespace
  100.         - { name: NUODB_DOMAIN_ENTRYPOINT,      value: "admin-0.domain" }
  101.         - { name: NUODB_ALT_ADDRESS,            value: "$(POD_NAME).domain.testns.svc" }
  102.         - { name: NUODB_VARDIR,                 value: "/var/opt/nuodb/$(POD_NAME).testns" }
  103.         args: [ "nuoadmin" ]
  104.         livenessProbe:
  105.           initialDelaySeconds: 30
  106.           periodSeconds: 15
  107.           tcpSocket:
  108.             port: 8888
  109.         readinessProbe:
  110.           initialDelaySeconds: 30
  111.           periodSeconds: 15
  112.           exec:
  113.             command: [ "nuodocker", "check", "servers" ]
  114.         volumeMounts:
  115.         - name: raftlog
  116.           mountPath: /var/opt/nuodb
  117.         - name: log-volume
  118.           mountPath: /var/log/nuodb
  119.       volumes:
  120.         - name: log-volume
  121.           emptyDir: {}
  122.   volumeClaimTemplates:
  123.   - metadata:
  124.       name: raftlog
  125.       annotations:
  126.         volume.beta.kubernetes.io/storage-class: openebs-cstor-sparse
  127.     spec:
  128.       accessModes: [ "ReadWriteOnce" ]
  129.       resources:
  130.         requests:
  131.           storage: 5G
  132. ---
  133. apiVersion: v1
  134. kind: Service
  135. metadata:
  136.   annotations:
  137.     description: "Service for redirect."
  138.   labels:
  139.     app: insights
  140.     group: nuodb
  141.   name: insights-server
  142. spec:
  143.   ports:
  144.   - { name: 8080-tcp,   port: 8080,   protocol: TCP,  targetPort: 8080  }
  145.   selector:
  146.     app: insights
  147.     group: nuodb
  148.   sessionAffinity: None
  149.   type: LoadBalancer
  150. status:
  151.   loadBalancer: {}
  153. ---
  154. apiVersion: v1
  155. kind: ConfigMap
  156. metadata:
  157.   annotations:
  158.     description: "Config map for metrics input and insights output"
  159.   name: insights-configmap
  160.   labels:
  161.     group: nuodb
  162. data:
  163.   nuoca.yml.template: |
  164.     ---
  165.     NUOCA_LOGFILE: /var/log/nuodb/nuoca.log
  167.     INPUT_PLUGINS:
  168.     - NuoAdminNuoMon:
  169.        description : Collection from NuoDB engines
  170.        nuocaCollectionName: NuoMon
  171.        api_server: https://domain:8888
  172.        client_key:  /etc/nuodb/keys/nuocmd.pem
  173.     OUTPUT_PLUGINS:
  174.     - RestClient:
  175.         url: ${INSIGHTS_INGEST_URL}
  176.   nuoinsights: |
  177.     #!/bin/bash
  178.     . ${NUODB_HOME}/etc/nuodb_setup.sh
  179.     : ${NUOCMD_API_SERVER:=https://domain:8888}
  181.     SCRIPT=$(cat <<EOF
  182.     from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
  183.     from nuoca import NuoCA as nuoca
  184.     import nuodb_mgmt
  186.     PORT_NUMBER = 8080
  188.     def get_insights_dashboard():
  189.       conn = nuodb_mgmt.AdminConnection('${NUOCMD_API_SERVER}', '/etc/nuodb/keys/nuocmd.pem')
  190.       metadata = nuoca.get_insights_metadata(conn)
  191.       return metadata.get(nuoca.INSIGHTS_DASHBOARD_KEY,None)
  193.       #ingest_url    = metadata[nuoca.INSIGHTS_URL_KEY]
  194.       #subscriber_id = metadata[nuoca.INSIGHTS_ID_KEY]
  195.       #return ingest_url[:-6] + subscriber_id
  197.     class myHandler(BaseHTTPRequestHandler):
  198.       def do_GET(self):
  199.         new_path = get_insights_dashboard()
  200.         if new_path and new_path != "None":
  201.           self.send_response(307)
  202.           self.send_header('Location', new_path)
  203.           self.end_headers()
  204.         else:
  205.           self.send_response(404)
  206.         self.end_headers()
  207.         return
  209.     try:
  210.       server = HTTPServer(('', PORT_NUMBER), myHandler)
  211.       print 'Started httpserver on port ' , PORT_NUMBER
  213.       #Wait forever for incoming http requests
  214.       server.serve_forever()
  216.     except KeyboardInterrupt:
  217.       print '^C received, shutting down the web server'
  218.       server.socket.close()
  220.     EOF
  221.     )
  223.     export PYTHONPATH=/opt/nuodb/drivers/pynuoadmin
  224.     exec /opt/nuodb/etc/python/nuopython -c "${SCRIPT}"
  226. ---
  227. apiVersion: v1
  228. kind: Pod
  229. metadata:
  230.   name: nuodb-insights
  231.   labels:
  232.     app: insights
  233.     group: nuodb
  234. spec:
  235.   initContainers:
  236.   - name: optin
  237.     image: nuodb/nuodb-ce:latest
  238.     imagePullPolicy: IfNotPresent
  239.     args:
  240.     - "/bin/sh"
  241.     - "-c"
  242.     - "[ \"${OPT_IN}\" == \"true\" ] && nuoca enable insights --connect-timeout 300 || nuoca disable insights --connect-timeout 300"
  243.     env:
  244.     - { name: NUOCMD_API_SERVER , value: "https://domain:8888" }
  245.   containers:
  246.   - name: insights
  247.     image: nuodb/nuodb-ce:latest
  248.     imagePullPolicy: IfNotPresent
  249.     args: [ "nuoca", "start" , "nuoca", "--insights" ]
  250.     env:
  251.     - { name: NUOCMD_API_SERVER , value: "https://domain:8888" }
  252.     volumeMounts:
  253.     - name: log-volume
  254.       mountPath: /var/log/nuodb
  255.     - name: config-insights
  256.       mountPath: /etc/nuodb/nuoca.yml.template
  257.       subPath: nuoca.yml.template
  258.   - name: insights-server
  259.     image: nuodb/nuodb-ce:latest
  260.     imagePullPolicy: IfNotPresent
  261.     args: [ "nuoinsights" ]
  262.     ports:
  263.     - { name: 8080-tcp,  containerPort: 8080,  protocol: TCP }
  264.     env:
  265.     - { name: NUOCMD_API_SERVER , value: "https://domain:8888" }
  266.     volumeMounts:
  267.       - name: log-volume
  268.         mountPath: /var/log/nuodb
  269.       - name: nuoinsights
  270.         mountPath: /usr/local/bin/nuoinsights
  271.         subPath: nuoinsights
  272.   volumes:
  273.   - name: log-volume
  274.     emptyDir: {}
  275.   - name: config-insights
  276.     configMap:
  277.       name: insights-configmap
  278.   - name: nuoinsights
  279.     configMap:
  280.       name: insights-configmap
  281.       defaultMode: 0754

nuodb-sm.yaml

  1. ---
  2. apiVersion: apps/v1beta1
  3. kind: StatefulSet
  4. metadata:
  5.   name: sm
  6.   labels:
  7.     app: nuodb
  8.     group: nuodb
  9. spec:
  10.   serviceName: sm
  11.   replicas: 1
  12.   selector:
  13.     matchLabels:
  14.       nodetype: sm
  15.       database: nuodb
  16.       app: sm
  17.       group: nuodb
  18.   podManagementPolicy: Parallel
  19.   template:
  20.     metadata:
  21.       labels:
  22.         nodetype: sm
  23.         database: nuodb
  24.         app: sm
  25.         group: nuodb
  26.     spec:
  27.       securityContext:
  28.         runAsUser: 0
  29.         fsGroup: 0
  30.       containers:
  31.       - name: sm
  32.         image: nuodb/nuodb-ce:latest
  33.         imagePullPolicy: IfNotPresent
  34.         args: [ "nuosm", "--servers-ready-timeout", "300" ]
  35.         env:
  36.         - name: NODE_NAME
  37.           valueFrom:
  38.             fieldRef:
  39.               fieldPath: spec.nodeName
  40.         - name: POD_NAME
  41.           valueFrom:
  42.             fieldRef:
  43.               fieldPath: metadata.name
  44.         - name: NAMESPACE
  45.           valueFrom:
  46.             fieldRef:
  47.               fieldPath: metadata.namespace
  48.         - { name: DB_NAME, value: "nuodb"}
  49.         - { name: DB_USER, value: "dba"}
  50.         - { name: DB_PASSWORD, value: "goalie"}
  51.         - { name: NUOCMD_API_SERVER,   value: "admin.testns.svc:8888"}
  52.         - { name: PEER_ADDRESS,        value: "admin.testns.svc" }
  53.         - { name: NUODB_OPTIONS,       value: "mem 1g" }
  54.         ports:
  55.         - containerPort: 48006
  56.           protocol: TCP
  57.         resources:
  58.           requests:
  59.             memory: "1Gi"
  60.             cpu: "200m"
  61.         terminationMessagePath: /dev/termination-log
  62.         terminationMessagePolicy: File
  63.         volumeMounts:
  64.         - mountPath: /var/opt/nuodb/archive
  65.           name: archive
  66.   volumeClaimTemplates:
  67.   - metadata:
  68.       name: archive
  69.       annotations:
  70.         volume.beta.kubernetes.io/storage-class: openebs-cstor-sparse
  71.     spec:
  72.       accessModes: [ "ReadWriteOnce" ]
  73.       resources:
  74.         requests:
  75.           storage: 5G

nuodb-te.yaml

  1. ---
  2. apiVersion: apps/v1beta1
  3. kind: Deployment
  4. metadata:
  5.   name: te
  6.   labels:
  7.     app: nuodb
  8.     group: nuodb
  9.     deploymentconfig: te
  10. spec:
  11.   replicas: 1
  12.   template:
  13.     metadata:
  14.       labels:
  15.         app: te
  16.         deploymentconfig: te
  17.         database: nuodb
  18.         nodetype: te
  19.     spec:
  20.       containers:
  21.       -
  22.         name: te
  23.         image: nuodb/nuodb-ce:latest
  24.         imagePullPolicy: IfNotPresent
  25.         env:
  26.         - name: NODE_NAME
  27.           valueFrom:
  28.             fieldRef:
  29.               fieldPath: spec.nodeName
  30.         - name: POD_NAME
  31.           valueFrom:
  32.             fieldRef:
  33.               fieldPath: metadata.name
  34.         - name: NAMESPACE
  35.           valueFrom:
  36.             fieldRef:
  37.               fieldPath: metadata.namespace
  38.         - { name: DB_NAME,             value: "nuodb" }
  39.         - { name: NUOCMD_API_SERVER,   value: "admin.testns.svc:8888"}
  40.         - { name: PEER_ADDRESS,        value: "admin.testns.svc" }
  41.         - { name: NUODB_LABELS,        value: "" }
  42.         - { name: NUODB_OPTIONS,       value: "mem 1g" }
  43.         args: [ "nuote", "--servers-ready-timeout", "300", "--database-created-timeout", "300" ]
  44.         ports:
  45.         - containerPort: 48006
  46.           protocol: TCP
  47.         resources:
  48.           requests:
  49.             memory: "1Gi"
  50.             cpu: "200m"
  51.         terminationMessagePath: /dev/termination-log
  52.         terminationMessagePolicy: File
  53.         volumeMounts:
  54.         - name: logdir
  55.           mountPath: /var/log/nuodb
  56.       volumes:
  57.         - name: logdir
  58.           emptyDir: {}

See Also:

OpenEBS architecture

OpenEBS use cases

cStor pools overview