Pseudo-Cluster Deployment

The purpose of the pseudo-cluster deployment is to deploy the DolphinScheduler service on a single machine. In this mode, DolphinScheduler’s master, worker, API server, are all on the same machine.

If you are a new hand and want to experience DolphinScheduler functions, we recommend you install follow Standalone deployment. If you want to experience more complete functions and schedule massive tasks, we recommend you install follow[pseudo-cluster deployment. If you want to deploy DolphinScheduler in production, we recommend you follow cluster deployment or Kubernetes deployment.

Preparation

Pseudo-cluster deployment of DolphinScheduler requires external software support:

  • JDK：download JDK (1.8+), install and configure environment variable JAVA_HOME and append bin dir (included in JAVA_HOME) to PATH variable. You can skip this step if it already exists in your environment.
  • Binary package: Download the DolphinScheduler binary package at download page
  • Database: PostgreSQL (8.2.15+) or MySQL (5.7+), you can choose one of the two, such as MySQL requires JDBC Driver 8.0.16
  • Registry Center: ZooKeeper (3.4.6+)，download link
  • Process tree analysis
    • pstree for macOS
    • psmisc for Fedora/Red/Hat/CentOS/Ubuntu/Debian

Note: DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but if you need to run tasks that depend on them, you need to have the corresponding environment support.

DolphinScheduler Startup Environment

Configure User Exemption and Permissions

Create a deployment user, and make sure to configure sudo without password. Here make an example to create user dolphinscheduler:

  1. # To create a user, login as root
  2. useradd dolphinscheduler
  4. # Add password
  5. echo "dolphinscheduler" | passwd --stdin dolphinscheduler
  7. # Configure sudo without password
  8. sed -i '$adolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' /etc/sudoers
  9. sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers
  11. # Modify directory permissions and grant permissions for user you created above
  12. chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin

NOTICE:

  • Due to DolphinScheduler’s multi-tenant task switch user using command sudo -u {linux-user}, the deployment user needs to have sudo privileges and be password-free. If novice learners don’t understand, you can ignore this point for now.
  • If you find the line “Defaults requirett” in the /etc/sudoers file, please comment the content.

Configure Machine SSH Password-Free Login

Since resources need to be sent to different machines during installation, SSH password-free login is required between each machine. The following shows the steps to configure password-free login:

  1. su dolphinscheduler
  3. ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
  4. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  5. chmod 600 ~/.ssh/authorized_keys

Notice: After the configuration is complete, you can run the command ssh localhost to test works or not. If you can login with ssh without password stands for successful.

Start ZooKeeper

Go to the ZooKeeper installation directory, copy configure file zoo_sample.cfg to conf/zoo.cfg, and change value of dataDir in conf/zoo.cfg to dataDir=./tmp/zookeeper.

  1. # Start ZooKeeper
  2. ./bin/zkServer.sh start

Modify Configuration

After completing the preparation of the basic environment, you need to modify the configuration file according to the environment you used. Change the environment configurations via export <ENV_NAME>=<VALUE>. The configuration files are located in directory bin/env as install_env.sh and dolphinscheduler_env.sh.

Modify install_env.sh

File install_env.sh describes which machines will be installed DolphinScheduler and what server will be installed on each machine. You could find this file in the path bin/env/install_env.sh and the detail of the configuration as below.

  1. # ---------------------------------------------------------
  2. # INSTALL MACHINE
  3. # ---------------------------------------------------------
  4. # Due to the master, worker, and API server being deployed on a single node, the IP of the server is the machine IP or localhost
  5. ips="localhost"
  6. sshPort="22"
  7. masters="localhost"
  8. workers="localhost:default"
  9. alertServer="localhost"
  10. apiServers="localhost"
  12. # DolphinScheduler installation path, it will auto-create if not exists
  13. installPath=~/dolphinscheduler
  15. # Deploy user, use the user you create in section **Configure machine SSH password-free login**
  16. deployUser="dolphinscheduler"

Modify dolphinscheduler_env.sh

File ./bin/env/dolphinscheduler_env.sh describes the following configurations:

  • Database configuration of DolphinScheduler, see Initialize the Database for detailed instructions.
  • Some tasks which need external dependencies or libraries such as JAVA_HOME and SPARK_HOME.
  • Registry center zookeeper.
  • Server related configuration, such as cache type, timezone, etc.

You could ignore the task external dependencies if you do not use those tasks, but you have to change JAVA_HOME, registry center and database related configurations based on your environment.

  1. # JAVA_HOME, will use it to start DolphinScheduler server
  2. export JAVA_HOME=${JAVA_HOME:-/opt/soft/java}
  4. # Database related configuration, set database type, username and password
  5. export DATABASE=${DATABASE:-postgresql}
  6. export SPRING_PROFILES_ACTIVE=${DATABASE}
  7. export SPRING_DATASOURCE_URL="jdbc:postgresql://127.0.0.1:5432/dolphinscheduler"
  8. export SPRING_DATASOURCE_USERNAME={user}
  9. export SPRING_DATASOURCE_PASSWORD={password}
  11. # DolphinScheduler server related configuration
  12. export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
  13. export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
  14. export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
  16. # Registry center configuration, determines the type and link of the registry center
  17. export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
  18. export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
  20. # Tasks related configurations, need to change the configuration if you use the related tasks.
  21. export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
  22. export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
  23. export SPARK_HOME1=${SPARK_HOME1:-/opt/soft/spark1}
  24. export SPARK_HOME2=${SPARK_HOME2:-/opt/soft/spark2}
  25. export PYTHON_HOME=${PYTHON_HOME:-/opt/soft/python}
  26. export HIVE_HOME=${HIVE_HOME:-/opt/soft/hive}
  27. export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
  28. export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}
  30. export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH

Initialize the Database

Follow the instructions in datasource-setting Pseudo-Cluster/Cluster Initialize the Database section to create and initialize database

Start DolphinScheduler

Use deployment user you created above, running the following command to complete the deployment, and the server log will be stored in the logs folder.

  1. bash ./bin/install.sh

Note: For the first time deployment, there maybe occur five times of sh: bin/dolphinscheduler-daemon.sh: No such file or directory in the terminal, this is non-important information that you can ignore.

Login DolphinScheduler

Access address http://localhost:12345/dolphinscheduler/ui and login DolphinScheduler UI. The default username and password are admin/dolphinscheduler123

Start or Stop Server

  1. # Stop all DolphinScheduler server
  2. bash ./bin/stop-all.sh
  4. # Start all DolphinScheduler server
  5. bash ./bin/start-all.sh
  7. # Start or stop DolphinScheduler Master
  8. bash ./bin/dolphinscheduler-daemon.sh stop master-server
  9. bash ./bin/dolphinscheduler-daemon.sh start master-server
  11. # Start or stop DolphinScheduler Worker
  12. bash ./bin/dolphinscheduler-daemon.sh start worker-server
  13. bash ./bin/dolphinscheduler-daemon.sh stop worker-server
  15. # Start or stop DolphinScheduler Api
  16. bash ./bin/dolphinscheduler-daemon.sh start api-server
  17. bash ./bin/dolphinscheduler-daemon.sh stop api-server
  19. # Start or stop Alert
  20. bash ./bin/dolphinscheduler-daemon.sh start alert-server
  21. bash ./bin/dolphinscheduler-daemon.sh stop alert-server

Note1:: Each server have dolphinscheduler_env.sh file in path <service>/conf/dolphinscheduler_env.sh which for micro-services need. It means that you could start all servers by command <service>/bin/start.sh with different environment variable from <service>/conf/dolphinscheduler_env.sh. But it will use file bin/env/dolphinscheduler_env.sh overwrite <service>/conf/dolphinscheduler_env.sh if you start server with command /bin/dolphinscheduler-daemon.sh start <service>.

Note2:: Please refer to the section of “System Architecture Design” for service usage. Python gateway service is started along with the api-server, and if you do not want to start Python gateway service please disabled it by changing the yaml config python-gateway.enabled : false in api-server’s configuration path api-server/conf/application.yaml