QuickStart in Kubernetes

Kubernetes deployment is deploy DolphinScheduler in a Kubernetes cluster, which can schedule a large number of tasks and can be used in production.

If you are a green hand and want to experience DolphinScheduler, we recommended you install follow Standalone. If you want to experience more complete functions or schedule large tasks number, we recommended you install follow pseudo-cluster deployment. If you want to using DolphinScheduler in production, we recommended you follow cluster deployment or kubernetes

Prerequisites

• Helm 3.1.0+
• Kubernetes 1.12+
• PV provisioner support in the underlying infrastructure

Installing the Chart

NOTE: You need to change your Chart.yaml locally to make it work. Due to a change in the Bitnami repo, https://charts.bitnami.com/bitnami was truncated only containing entries for the latest 6 months (from January 2022 on). This URL: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami contains the full ‘index.yaml’.

See detail here: https://github.com/bitnami/charts/issues/10833.

Change you Chart.yaml in path apache-dolphinscheduler-2.0.6-src/docker/kubernetes/dolphinscheduler after you download the source code and replace two places from repository: https://charts.bitnami.com/bitnami to repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami

Please download the source code package apache-dolphinscheduler-2.0.6-src.tar.gz, download address: download

To install the chart with the release name dolphinscheduler, please execute the following commands:

$ tar -zxvf apache-dolphinscheduler-2.0.6-src.tar.gz
$ cd apache-dolphinscheduler-2.0.6-src/docker/kubernetes/dolphinscheduler
$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm dependency update .
$ helm install dolphinscheduler . --set image.tag=2.0.6

To install the chart with a namespace named test:

$ helm install dolphinscheduler . -n test

Tip: If a namespace named test is used, the option -n test needs to be added to the helm and kubectl command

These commands deploy DolphinScheduler on the Kubernetes cluster in the default configuration. The Appendix-Configuration section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

The PostgreSQL (with username root, password root and database dolphinscheduler) and ZooKeeper services will start by default

Access DolphinScheduler UI

If ingress.enabled in values.yaml is set to true, you just access http://${ingress.host}/dolphinscheduler in browser.

Tip: If there is a problem with ingress access, please contact the Kubernetes administrator and refer to the Ingress

Otherwise, when api.service.type=ClusterIP you need to execute port-forward command like:

$ kubectl port-forward --address 0.0.0.0 svc/dolphinscheduler-api 12345:12345
$ kubectl port-forward --address 0.0.0.0 -n test svc/dolphinscheduler-api 12345:12345 # with test namespace

Tip: If the error of unable to do port forwarding: socat not found appears, you need to install socat at first

And then access the web: http://localhost:12345/dolphinscheduler (The local address is http://localhost:12345/dolphinscheduler)

Or when api.service.type=NodePort you need to execute the command:

NODE_IP=$(kubectl get no -n {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
NODE_PORT=$(kubectl get svc {{ template "dolphinscheduler.fullname" . }}-api -n {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}")
echo http://$NODE_IP:$NODE_PORT/dolphinscheduler

And then access the web: http://NODEIP:NODE\_IP:NODE​I​​P:NODE\_PORT/dolphinscheduler

The default username is admin and the default password is dolphinscheduler123

Please refer to the Quick Start in the chapter Quick Start to explore how to use DolphinScheduler

Uninstalling the Chart

To uninstall/delete the dolphinscheduler deployment:

$ helm uninstall dolphinscheduler

The command removes all the Kubernetes components but PVC’s associated with the chart and deletes the release.

To delete the PVC’s associated with dolphinscheduler:

$ kubectl delete pvc -l app.kubernetes.io/instance=dolphinscheduler

Note: Deleting the PVC’s will delete all data as well. Please be cautious before doing it.

Configuration

The configuration file is values.yaml, and the Appendix-Configuration tables lists the configurable parameters of the DolphinScheduler and their default values.

Support Matrix

FAQ

How to view the logs of a pod container?

List all pods (aka po):

kubectl get po
kubectl get po -n test # with test namespace

View the logs of a pod container named dolphinscheduler-master-0:

kubectl logs dolphinscheduler-master-0
kubectl logs -f dolphinscheduler-master-0 # follow log output
kubectl logs --tail 10 dolphinscheduler-master-0 -n test # show last 10 lines from the end of the logs

How to scale api, master and worker on Kubernetes?

List all deployments (aka deploy):

kubectl get deploy
kubectl get deploy -n test # with test namespace

Scale api to 3 replicas:

kubectl scale --replicas=3 deploy dolphinscheduler-api
kubectl scale --replicas=3 deploy dolphinscheduler-api -n test # with test namespace

List all stateful sets (aka sts):

kubectl get sts
kubectl get sts -n test # with test namespace

Scale master to 2 replicas:

kubectl scale --replicas=2 sts dolphinscheduler-master
kubectl scale --replicas=2 sts dolphinscheduler-master -n test # with test namespace

Scale worker to 6 replicas:

kubectl scale --replicas=6 sts dolphinscheduler-worker
kubectl scale --replicas=6 sts dolphinscheduler-worker -n test # with test namespace

How to use MySQL as the DolphinScheduler’s database instead of PostgreSQL?

Because of the commercial license, we cannot directly use the driver of MySQL.

If you want to use MySQL, you can build a new image based on the apache/dolphinscheduler image as follows.

1. Download the MySQL driver mysql-connector-java-8.0.16.jar

2. Create a new Dockerfile to add MySQL driver:

FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler:2.0.6
COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/lib

1. Build a new docker image including MySQL driver:

docker build -t apache/dolphinscheduler:mysql-driver .

1. Push the docker image apache/dolphinscheduler:mysql-driver to a docker registry

2. Modify image repository and update tag to mysql-driver in values.yaml

3. Modify postgresql enabled to false in values.yaml

4. Modify externalDatabase (especially modify host, username and password) in values.yaml:

externalDatabase:
  type: "mysql"
  driver: "com.mysql.jdbc.Driver"
  host: "localhost"
  port: "3306"
  username: "root"
  password: "root"
  database: "dolphinscheduler"
  params: "useUnicode=true&characterEncoding=UTF-8"

1. Run a DolphinScheduler release in Kubernetes (See Installing the Chart)

How to support MySQL datasource in Datasource manage?

Because of the commercial license, we cannot directly use the driver of MySQL.

If you want to add MySQL datasource, you can build a new image based on the apache/dolphinscheduler image as follows.

1. Download the MySQL driver mysql-connector-java-8.0.16.jar

2. Create a new Dockerfile to add MySQL driver:

FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler:2.0.6
COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/lib

1. Build a new docker image including MySQL driver:

docker build -t apache/dolphinscheduler:mysql-driver .

1. Push the docker image apache/dolphinscheduler:mysql-driver to a docker registry

2. Modify image repository and update tag to mysql-driver in values.yaml

3. Run a DolphinScheduler release in Kubernetes (See Installing the Chart)

4. Add a MySQL datasource in Datasource manage

How to support Oracle datasource in Datasource manage?

Because of the commercial license, we cannot directly use the driver of Oracle.

If you want to add Oracle datasource, you can build a new image based on the apache/dolphinscheduler image as follows.

1. Download the Oracle driver ojdbc8.jar (such as ojdbc8-19.9.0.0.jar)

2. Create a new Dockerfile to add Oracle driver:

FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler:2.0.6
COPY ojdbc8-19.9.0.0.jar /opt/dolphinscheduler/lib

1. Build a new docker image including Oracle driver:

docker build -t apache/dolphinscheduler:oracle-driver .

1. Push the docker image apache/dolphinscheduler:oracle-driver to a docker registry

2. Modify image repository and update tag to oracle-driver in values.yaml

3. Run a DolphinScheduler release in Kubernetes (See Installing the Chart)

4. Add an Oracle datasource in Datasource manage

How to support Python 2 pip and custom requirements.txt?

1. Create a new Dockerfile to install pip:

FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler:2.0.6
COPY requirements.txt /tmp
RUN apt-get update && \
    apt-get install -y --no-install-recommends python-pip && \
    pip install --no-cache-dir -r /tmp/requirements.txt && \
    rm -rf /var/lib/apt/lists/*

The command will install the default pip 18.1. If you upgrade the pip, just add one line

    pip install --no-cache-dir -U pip && \

1. Build a new docker image including pip:

docker build -t apache/dolphinscheduler:pip .

1. Push the docker image apache/dolphinscheduler:pip to a docker registry

2. Modify image repository and update tag to pip in values.yaml

3. Run a DolphinScheduler release in Kubernetes (See Installing the Chart)

4. Verify pip under a new Python task

How to support Python 3?

1. Create a new Dockerfile to install Python 3:

FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler:2.0.6
RUN apt-get update && \
    apt-get install -y --no-install-recommends python3 && \
    rm -rf /var/lib/apt/lists/*

The command will install the default Python 3.7.3. If you also want to install pip3, just replace python3 with python3-pip like

    apt-get install -y --no-install-recommends python3-pip && \

1. Build a new docker image including Python 3:

docker build -t apache/dolphinscheduler:python3 .

1. Push the docker image apache/dolphinscheduler:python3 to a docker registry

2. Modify image repository and update tag to python3 in values.yaml

3. Modify PYTHON_HOME to /usr/bin/python3 in values.yaml

4. Run a DolphinScheduler release in Kubernetes (See Installing the Chart)

5. Verify Python 3 under a new Python task

How to support Hadoop, Spark, Flink, Hive or DataX?

Take Spark 2.4.7 as an example:

1. Download the Spark 2.4.7 release binary spark-2.4.7-bin-hadoop2.7.tgz

2. Ensure that common.sharedStoragePersistence.enabled is turned on

3. Run a DolphinScheduler release in Kubernetes (See Installing the Chart)

4. Copy the Spark 2.4.7 release binary into the Docker container

kubectl cp spark-2.4.7-bin-hadoop2.7.tgz dolphinscheduler-worker-0:/opt/soft
kubectl cp -n test spark-2.4.7-bin-hadoop2.7.tgz dolphinscheduler-worker-0:/opt/soft # with test namespace

Because the volume sharedStoragePersistence is mounted on /opt/soft, all files in /opt/soft will not be lost

1. Attach the container and ensure that SPARK_HOME2 exists

kubectl exec -it dolphinscheduler-worker-0 bash
kubectl exec -n test -it dolphinscheduler-worker-0 bash # with test namespace
cd /opt/soft
tar zxf spark-2.4.7-bin-hadoop2.7.tgz
rm -f spark-2.4.7-bin-hadoop2.7.tgz
ln -s spark-2.4.7-bin-hadoop2.7 spark2 # or just mv
$SPARK_HOME2/bin/spark-submit --version

The last command will print the Spark version if everything goes well

1. Verify Spark under a Shell task

$SPARK_HOME2/bin/spark-submit --class org.apache.spark.examples.SparkPi $SPARK_HOME2/examples/jars/spark-examples_2.11-2.4.7.jar

Check whether the task log contains the output like Pi is roughly 3.146015

1. Verify Spark under a Spark task

The file spark-examples_2.11-2.4.7.jar needs to be uploaded to the resources first, and then create a Spark task with:

• Spark Version: SPARK2
• Main Class: org.apache.spark.examples.SparkPi
• Main Package: spark-examples_2.11-2.4.7.jar
• Deploy Mode: local

Similarly, check whether the task log contains the output like Pi is roughly 3.146015

1. Verify Spark on YARN

Spark on YARN (Deploy Mode is cluster or client) requires Hadoop support. Similar to Spark support, the operation of supporting Hadoop is almost the same as the previous steps

Ensure that $HADOOP_HOME and $HADOOP_CONF_DIR exists

How to support Spark 3?

In fact, the way to submit applications with spark-submit is the same, regardless of Spark 1, 2 or 3. In other words, the semantics of SPARK_HOME2 is the second SPARK_HOME instead of SPARK2‘s HOME, so just set SPARK_HOME2=/path/to/spark3

Take Spark 3.1.1 as an example:

1. Download the Spark 3.1.1 release binary spark-3.1.1-bin-hadoop2.7.tgz

2. Ensure that common.sharedStoragePersistence.enabled is turned on

3. Run a DolphinScheduler release in Kubernetes (See Installing the Chart)

4. Copy the Spark 3.1.1 release binary into the Docker container

kubectl cp spark-3.1.1-bin-hadoop2.7.tgz dolphinscheduler-worker-0:/opt/soft
kubectl cp -n test spark-3.1.1-bin-hadoop2.7.tgz dolphinscheduler-worker-0:/opt/soft # with test namespace

1. Attach the container and ensure that SPARK_HOME2 exists

kubectl exec -it dolphinscheduler-worker-0 bash
kubectl exec -n test -it dolphinscheduler-worker-0 bash # with test namespace
cd /opt/soft
tar zxf spark-3.1.1-bin-hadoop2.7.tgz
rm -f spark-3.1.1-bin-hadoop2.7.tgz
ln -s spark-3.1.1-bin-hadoop2.7 spark2 # or just mv
$SPARK_HOME2/bin/spark-submit --version

The last command will print the Spark version if everything goes well

1. Verify Spark under a Shell task

$SPARK_HOME2/bin/spark-submit --class org.apache.spark.examples.SparkPi $SPARK_HOME2/examples/jars/spark-examples_2.12-3.1.1.jar

Check whether the task log contains the output like Pi is roughly 3.146015

How to support shared storage between Master, Worker and Api server?

For example, Master, Worker and API server may use Hadoop at the same time

1. Modify the following configurations in values.yaml

common:
  sharedStoragePersistence:
    enabled: false
    mountPath: "/opt/soft"
    accessModes:
    - "ReadWriteMany"
    storageClassName: "-"
    storage: "20Gi"

storageClassName and storage need to be modified to actual values

Note: storageClassName must support the access mode: ReadWriteMany

1. Copy the Hadoop into the directory /opt/soft

2. Ensure that $HADOOP_HOME and $HADOOP_CONF_DIR are correct

How to support local file resource storage instead of HDFS and S3?

Modify the following configurations in values.yaml

common:
  configmap:
    RESOURCE_STORAGE_TYPE: "HDFS"
    RESOURCE_UPLOAD_PATH: "/dolphinscheduler"
    FS_DEFAULT_FS: "file:///"
  fsFileResourcePersistence:
    enabled: true
    accessModes:
    - "ReadWriteMany"
    storageClassName: "-"
    storage: "20Gi"

storageClassName and storage need to be modified to actual values

Note: storageClassName must support the access mode: ReadWriteMany

How to support S3 resource storage like MinIO?

Take MinIO as an example: Modify the following configurations in values.yaml

common:
  configmap:
    RESOURCE_STORAGE_TYPE: "S3"
    RESOURCE_UPLOAD_PATH: "/dolphinscheduler"
    FS_DEFAULT_FS: "s3a://BUCKET_NAME"
    FS_S3A_ENDPOINT: "http://MINIO_IP:9000"
    FS_S3A_ACCESS_KEY: "MINIO_ACCESS_KEY"
    FS_S3A_SECRET_KEY: "MINIO_SECRET_KEY"

BUCKET_NAME, MINIO_IP, MINIO_ACCESS_KEY and MINIO_SECRET_KEY need to be modified to actual values

Note: MINIO_IP can only use IP instead of domain name, because DolphinScheduler currently doesn’t support S3 path style access

How to configure SkyWalking?

Modify SKYWALKING configurations in values.yaml:

common:
  configmap:
    SKYWALKING_ENABLE: "true"
    SW_AGENT_COLLECTOR_BACKEND_SERVICES: "127.0.0.1:11800"
    SW_GRPC_LOG_SERVER_HOST: "127.0.0.1"
    SW_GRPC_LOG_SERVER_PORT: "11800"

Appendix-Configuration