Consider this an EXPERIMENTAL feature mostly because it has not been tested yet on a wide variety of long running Druid clusters.

Apache Druid Extension to enable using Kubernetes API Server for node discovery and leader election. This extension allows Druid cluster deployment on Kubernetes without Zookeeper. It allows running multiple Druid clusters within same Kubernetes Cluster, See clusterIdentifier config below.

Configuration

To use this extension please make sure to include druid-kubernetes-extensions in the extensions load list.

This extension works together with HTTP based segment and task management in Druid. Consequently, following configurations must be set on all Druid nodes.

druid.zk.service.enabled=false druid.serverview.type=http druid.coordinator.loadqueuepeon.type=http druid.indexer.runner.type=httpRemote druid.discovery.type=k8s

For Node Discovery, Each Druid process running inside a pod “announces” itself by adding few “labels” and “annotations” in the pod spec. Druid process needs to be aware of pod name and namespace which it reads from environment variables POD_NAME and POD_NAMESPACE. These variable names can be changed, see configuration below. But in the end, each pod needs to have self pod name and namespace added as environment variables.

Additionally, this extension has following configuration.

Properties

PropertyPossible ValuesDescriptionDefaultrequired
druid.discovery.k8s.clusterIdentifierstring that matches [a-z0-9][a-z0-9-]*[a-z0-9]Unique identifier for this Druid cluster in Kubernetes e.g. us-west-prod-druid.NoneYes
druid.discovery.k8s.podNameEnvKeyPod Env VariablePod Env variable whose value is that pod’s name.POD_NAMENo
druid.discovery.k8s.podNamespaceEnvKeyPod Env VariablePod Env variable whose value is that pod’s kubernetes namespace.POD_NAMESPACENo
druid.discovery.k8s.leaseDurationDurationLease duration used by Leader Election algorithm. Candidates wait for this time before taking over previous Leader.PT60SNo
druid.discovery.k8s.renewDeadlineDurationLease renewal period used by Leader.PT17SNo
druid.discovery.k8s.retryPeriodDurationRetry wait used by Leader Election algorithm on failed operations.PT5SNo

Gotchas

  • Label/Annotation path in each pod spec MUST EXIST, which is easily satisfied if there is at least one label/annotation in the pod spec already. This limitation may be removed in future.
  • All Druid Pods belonging to one Druid cluster must be inside same kubernetes namespace.
  • All Druid Pods need permissions to be able to add labels to self-pod, List and Watch other Pods, create and read ConfigMap for leader election. Assuming, “default” service account is used by Druid pods, you might need to add following or something similar Kubernetes Role and Role Binding.
  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: Role
  3. metadata:
  4.   name: druid-cluster
  5. rules:
  6. - apiGroups:
  7.   - ""
  8.   resources:
  9.   - pods
  10.   - configmaps
  11.   verbs:
  12.   - '*'
  13. ---
  14. kind: RoleBinding
  15. apiVersion: rbac.authorization.k8s.io/v1
  16. metadata:
  17.   name: druid-cluster
  18. subjects:
  19. - kind: ServiceAccount
  20.   name: default
  21. roleRef:
  22.   kind: Role
  23.   name: druid-cluster
  24.   apiGroup: rbac.authorization.k8s.io