Automatic service start

This document describes how to configure the automatic pull-up of the Doris cluster to ensure that services are not pulled up in time after service breaks down due to special circumstances in the production environment.

The automatic pull-up service of FE and BE must be configured after the Doris cluster is completely set up.

Systemd Configures the Doris service

For details about systemd usage and parameter parsing, see here

sudo permission control

sudo permissions are required to control the doris service using systemd. To ensure the minimum granularity of sudo permission assignment, you can assign the systemd control permission of doris-fe and doris-be services to specified non-root users. Configure the systemctl management permission for doris-fe and doris-be in visudo.

  1. Cmnd_Alias DORISCTL=/usr/bin/systemctl start doris-fe,/usr/bin/systemctl stop doris-fe,/usr/bin/systemctl start doris-be,/usr/bin/systemctl stop doris-be
  3. ## Allow root to run any commands anywhere
  4. root    ALL=(ALL)       ALL
  5. doris   ALL=(ALL)       NOPASSWD:DORISCTL

Configuration procedure

  1. You should config the “JAVA_HOME” variable in the config file, both fe.conf and be.conf, or you can’t use the command “systemctl start” to start doris

    1. echo "JAVA_HOME=your_java_home" >> /home/doris/fe/conf/fe.conf
    2. echo "JAVA_HOME=your_java_home" >> /home/doris/be/conf/be.conf

  2. Download the doris-fe.service file: doris-fe.service

  3. The details of doris-fe.service are as follows:

    1. # Licensed to the Apache Software Foundation (ASF) under one
    2. # or more contributor license agreements.  See the NOTICE file
    3. # distributed with this work for additional information
    4. # regarding copyright ownership.  The ASF licenses this file
    5. # to you under the Apache License, Version 2.0 (the
    6. # "License"); you may not use this file except in compliance
    7. # with the License.  You may obtain a copy of the License at
    8. #
    9. #   http://www.apache.org/licenses/LICENSE-2.0
    10. #
    11. # Unless required by applicable law or agreed to in writing,
    12. # software distributed under the License is distributed on an
    13. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    14. # KIND, either express or implied.  See the License for the
    15. # specific language governing permissions and limitations
    16. # under the License.
    18. [Unit]
    19. Description=Doris FE
    20. After=network-online.target
    21. Wants=network-online.target
    23. [Service]
    24. Type=forking
    25. User=root
    26. Group=root
    27. LimitCORE=infinity
    28. LimitNOFILE=200000
    29. Restart=on-failure
    30. RestartSec=30
    31. StartLimitInterval=120
    32. StartLimitBurst=3
    33. KillMode=none
    34. ExecStart=/home/doris/fe/bin/start_fe.sh --daemon 
    35. ExecStop=/home/doris/fe/bin/stop_fe.sh
    37. [Install]
    38. WantedBy=multi-user.target

Matters needing attention

  • ExecStart and ExecStop are configured based on actual fe paths

  1. Download the doris-be.service file : doris-be.service

  2. The details of doris-be.service are as follows:

    1. # Licensed to the Apache Software Foundation (ASF) under one
    2. # or more contributor license agreements.  See the NOTICE file
    3. # distributed with this work for additional information
    4. # regarding copyright ownership.  The ASF licenses this file
    5. # to you under the Apache License, Version 2.0 (the
    6. # "License"); you may not use this file except in compliance
    7. # with the License.  You may obtain a copy of the License at
    8. #
    9. #   http://www.apache.org/licenses/LICENSE-2.0
    10. #
    11. # Unless required by applicable law or agreed to in writing,
    12. # software distributed under the License is distributed on an
    13. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    14. # KIND, either express or implied.  See the License for the
    15. # specific language governing permissions and limitations
    16. # under the License.
    18. [Unit]
    19. Description=Doris BE
    20. After=network-online.target
    21. Wants=network-online.target
    23. [Service]
    24. Type=forking
    25. User=root
    26. Group=root
    27. LimitCORE=infinity
    28. LimitNOFILE=200000
    29. Restart=on-failure
    30. RestartSec=30
    31. StartLimitInterval=120
    32. StartLimitBurst=3
    33. KillMode=none
    34. ExecStart=/home/doris/be/bin/start_be.sh --daemon
    35. ExecStop=/home/doris/be/bin/stop_be.sh
    37. [Install]
    38. WantedBy=multi-user.target

Matters needing attention

  • ExecStart and ExecStop are configured based on actual be paths

  1. Service configuration

    Place doris-fe.service and doris-be.service in the /usr/lib/systemd/system directory

  2. Set self-start

    After you add or modify the configuration file, you need to reload it

    1. systemctl daemon-reload

    Set the start, the essence is in the/etc/systemd/system/multi - user. Target. Wants/add service file link

    1. systemctl enable doris-fe
    2. systemctl enable doris-be

  3. Service initiation

    1. systemctl start doris-fe
    2. systemctl start doris-be

Supervisor configures the Doris service

Supervisor Specific use and parameter analysis can be referred to here

Supervisor configuration automatically pulls up the supervisor configuration. You can install the supervisor directly using the yum command or manually using pip. The pip manual installation process is complicated, and only the yum deployment mode is displayed.Manual deployment refer to [here] (http://supervisord.org/installing.html) for installation deployment.

Configuration procedure

  1. yum Install supervisor

    1. yum install epel-release
    2. yum install -y supervisor

  2. Start the service and view the status

    1. systemctl enable supervisord # bootstrap
    2. systemctl start supervisord # Start the supervisord service
    3. systemctl status supervisord # Check the supervisord service status
    4. ps -ef|grep supervisord # Check whether the supervisord process exists

  3. Configure BE process management

    1. Modify the start_be.sh script remove the last symbol &
    3. vim /path/doris/be/bin/start_be.sh
    4. Take this code : nohup $LIMIT ${DORIS_HOME}/lib/palo_be "$@" >> $LOG_DIR/be.out 2>&1 </dev/null &
    5. Be changed to : nohup $LIMIT ${DORIS_HOME}/lib/palo_be "$@" >> $LOG_DIR/be.out 2>&1 </dev/null

    Create a supervisor process management configuration file for the BE

    1. vim /etc/supervisord.d/doris-be.ini
    3. [program:doris_be]      
    4. process_name=%(program_name)s      
    5. directory=/path/doris/be/be
    6. command=sh /path/doris/be/bin/start_be.sh
    7. autostart=true
    8. autorestart=true
    9. user=root
    10. numprocs=1
    11. startretries=3
    12. stopasgroup=true
    13. killasgroup=true
    14. startsecs=5
    15. #redirect_stderr = true
    16. #stdout_logfile_maxbytes = 20MB
    17. #stdout_logfile_backups = 10
    18. #stdout_logfile=/var/log/supervisor-palo_be.log

  4. Configure FE process management

    1. Modify the start_fe.sh script remove the last symbol &
    3. vim /path/doris/fe/bin/start_fe.sh 
    4. Take this code : nohup $LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe ${HELPER} "$@" >> $LOG_DIR/fe.out 2>&1 </dev/null &
    5. Be changed to : nohup $LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe ${HELPER} "$@" >> $LOG_DIR/fe.out 2>&1 </dev/null

    Create a supervisor process management configuration file for FE

    1. vim /etc/supervisord.d/doris-fe.ini
    3. [program:PaloFe]
    4. environment = JAVA_HOME="/usr/local/java"
    5. process_name=PaloFe
    6. directory=/path/doris/fe
    7. command=sh /path/doris/fe/bin/start_fe.sh
    8. autostart=true
    9. autorestart=true
    10. user=root
    11. numprocs=1
    12. startretries=3
    13. stopasgroup=true
    14. killasgroup=true
    15. startsecs=10
    16. #redirect_stderr=true
    17. #stdout_logfile_maxbytes=20MB
    18. #stdout_logfile_backups=10
    19. #stdout_logfile=/var/log/supervisor-PaloFe.log

  5. Configure Broker process management

    1. Modify the start_broker.sh script remove the last symbol &
    3. vim /path/apache_hdfs_broker/bin/start_broker.sh
    4. Take this code : nohup $LIMIT $JAVA $JAVA_OPTS org.apache.doris.broker.hdfs.BrokerBootstrap "$@" >> $BROKER_LOG_DIR/apache_hdfs_broker.out 2>&1 </dev/null &
    5. Be changed to : nohup $LIMIT $JAVA $JAVA_OPTS org.apache.doris.broker.hdfs.BrokerBootstrap "$@" >> $BROKER_LOG_DIR/apache_hdfs_broker.out 2>&1 </dev/null

    Create the supervisor process management profile for the Broker

    1. vim /etc/supervisord.d/doris-broker.ini
    3. [program:BrokerBootstrap]
    4. environment = JAVA_HOME="/usr/local/java"
    5. process_name=%(program_name)s
    6. directory=/path/apache_hdfs_broker
    7. command=sh /path/apache_hdfs_broker/bin/start_broker.sh
    8. autostart=true
    9. autorestart=true
    10. user=root
    11. numprocs=1
    12. startretries=3
    13. stopasgroup=true
    14. killasgroup=true
    15. startsecs=5
    16. #redirect_stderr=true
    17. #stdout_logfile_maxbytes=20MB
    18. #stdout_logfile_backups=10
    19. #stdout_logfile=/var/log/supervisor-BrokerBootstrap.log

  6. First determine whether the Doris service is stopped, then use supervisor to automatically pull up Doris, and then determine whether the process starts normally

    1. supervisorctl reload # Reload all the Supervisor configuration files
    2. supervisorctl status # Check the supervisor status and verify that the Doris service process starts normally
    4. 其他命令 : 
    5. supervisorctl start all # supervisorctl start It is capable of opening processes
    6. supervisorctl stop doris-be # The process is supervisorctl stop

Matters needing attention:

  • If the supervisor installed using yum starts, an error occurs: pkg_resources.DistributionNotFound: The ‘supervisor==3.4.0’ distribution was not found
  1. supervisor installed directly using the yum command only supports python2,Therefore, the file contents in /usr/bin/supervisorctl and /usr/bin/supervisorctl should be changed at the beginning Change #! /usr/bin/python to #! /usr/bin/python2, python2 must be installed
  • If the supervisor is configured to automatically pull up the Doris process, if the BE node breaks down due to abnormal factors on Doris, the error stack information that should be output to be.out will be intercepted by the supervisor. We need to look it up in supervisor’s log for further analysis.