OpenEBS for Elasticsearch

OpenEBS and Elasticsearch

Introduction

EFK is the most popular cloud native logging solution on Kubernetes for On-Premise as well as cloud platforms. In the EFK stack, Elasticsearch is a stateful application that needs persistent storage. Logs of production applications need to be stored for a long time which requires reliable and highly available storage. OpenEBS and EFK together provides a complete logging solution.

Advantages of using OpenEBS for Elasticsearch database:

  • All the logs data is stored locally and managed natively to Kubernetes
  • Start with small storage and add disks as needed on the fly
  • Logs are highly available. When a node fails or rebooted during upgrades, the persistent volumes from OpenEBS continue to be highly available.
  • If required, take backup of the Elasticsearch database periodically and back them up to S3 or any object storage so that restoration of the same logs is possible to the same or any other Kubernetes cluster

Note: Elasticsearch can be deployed both as deployment or as statefulset. When Elasticsearch deployed as statefulset, you don’t need to replicate the data again at OpenEBS level. When Elasticsearch is deployed as deployment, consider 3 OpenEBS replicas, choose the StorageClass accordingly.

Deployment model

OpenEBS and Elasticsearch

Configuration workflow

  1. Install OpenEBS

    If OpenEBS is not installed in your K8s cluster, this can done from here. If OpenEBS is already installed, go to the next step.

  2. Configure cStor Pool

    After OpenEBS installation, cStor pool has to be configured. If cStor Pool is not configured in your OpenEBS cluster, this can be done from here. During cStor Pool creation, make sure that the maxPools parameter is set to >=3. Sample YAML named openebs-config.yaml for configuring cStor Pool is provided in the Configuration details below. If cStor pool is already configured, go to the next step.

  3. Create Storage Class

    You must configure a StorageClass to provision cStor volume on given cStor pool. StorageClass is the interface through which most of the OpenEBS storage policies are defined. In this solution we are using a StorageClass to consume the cStor Pool which is created using external disks attached on the Nodes. Since Elasticsearch is a StatefulSet, it requires only single storage replica. So cStor volume replicaCount is >=1. Sample YAML named openebs-sc-disk.yamlto consume cStor pool with cStor volume replica count as 1 is provided in the configuration details below.

  4. Launch and test Elasticsearch

    Use latest Elasticsearch chart with helm to deploy Elasticsearch in your cluster using the following command. In the following command, it will create PVC with 30G size.

    1. helm install --name es-test --set volumeClaimTemplate.storageClassName=openebs-cstor-disk elastic/Elasticsearch --version 6.6.0-alpha1

    For more information on installation, see Elasticsearch documentation.

Reference at openebs.ci

A live deployment of Elasticsearch with Kibana using OpenEBS volumes can be seen at the website www.openebs.ci

Deployment YAML spec files for Elasticsearch and OpenEBS resources are found here

OpenEBS-CI dashboard of Elasticsearch

Live access to Elasticsearch dashboard

Post deployment Operations

Monitor OpenEBS Volume size

It is not seamless to increase the cStor volume size (refer to the roadmap item). Hence, it is recommended that sufficient size is allocated during the initial configuration.

Monitor cStor Pool size

As in most cases, cStor pool may not be dedicated to just Elasticsearch database alone. It is recommended to watch the pool capacity and add more disks to the pool before it hits 80% threshold. See cStorPool metrics.

Configuration details

openebs-config.yaml

  1. #Use the following YAMLs to create a cStor Storage Pool.
  2. # and associated storage class.
  3. apiVersion: openebs.io/v1alpha1
  4. kind: StoragePoolClaim
  5. metadata:
  6.   name: cstor-disk
  7. spec:
  8.   name: cstor-disk
  9.   type: disk
  10.   poolSpec:
  11.     poolType: striped
  12.   # NOTE - Appropriate disks need to be fetched using `kubectl get blockdevices -n openebs`
  13.   #
  14.   # `Block devices` is a custom resource supported by OpenEBS with `node-disk-manager`
  15.   # as the disk operator
  16. # Replace the following with actual disk CRs from your cluster `kubectl get blockdevices -n openebs`
  17. # Uncomment the below lines after updating the actual disk names.
  18.   blockDevices:
  19.     blockDeviceList:
  20. # Replace the following with actual disk CRs from your cluster from `kubectl get blockdevices -n openebs`
  21. #   - blockdevice-69cdfd958dcce3025ed1ff02b936d9b4
  22. #   - blockdevice-891ad1b581591ae6b54a36b5526550a2
  23. #   - blockdevice-ceaab442d802ca6aae20c36d20859a0b
  24. ---

openebs-sc-disk.yaml

  1. apiVersion: storage.k8s.io/v1
  2. kind: StorageClass
  3. metadata:
  4.   name: openebs-cstor-disk
  5.   annotations:
  6.     openebs.io/cas-type: cstor
  7.     cas.openebs.io/config: |
  8.       - name: StoragePoolClaim
  9.         value: "cstor-disk"
  10.       - name: ReplicaCount
  11.         value: "1"       
  12. provisioner: openebs.io/provisioner-iscsi
  13. reclaimPolicy: Delete
  14. ---

See Also:

OpenEBS architecture

OpenEBS use cases

cStor pools overview