Deploy without HDFS

This article describes how to deploy Linkis services in an environment where HDFS is not deployed to facilitate more lightweight learning, use and debugging.

For the overall deployment process, please refer to “Quick Deployment”, and you need to modify the following content.

Modify the linkis-env.sh file to modify the following:

  1. #Use path mode [file://path]  instead of [hdfs://] 
  2. WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis/
  3. HDFS_USER_ROOT_PATH=file:///tmp/linkis
  4. RESULT_SET_ROOT_PATH=file:///tmp/linkis
  6. export ENABLE_HDFS=false
  7. export ENABLE_HIVE=false
  8. export ENABLE_SPARK=false

After changing the above configuration to false, there is no need to configure HDFS/HIVE/SPARK environments separately.

Because the mysql-connector-java driver follows the GPL2.0 protocol and does not meet the license policy of the Apache open source protocol, we need to manually copy it to the following two directories.

  1. ${LINKIS_HOME}/lib/linkis-commons/public-module/
  2. ${LINKIS_HOME}/lib/linkis-spring-cloud-services/linkis-mg-gateway/

It can be downloaded directly from the maven repository, such as https://mvnrepository.com/artifact/mysql/mysql-connector-java/5.1.49

Refer to the “Quick Deployment” section to complete the deployment by executing the sh bin/install.sh command. Refering to the “Quick Deployment” section, start the Linkis services by executing the ${LINKIS_HOME}/sbin/linkis-start-all.sh command.

Currently, version 1.1.2 only supports shell jobs to run in non-HDFS environments. The execution commands are as follows.

  1. $ cd ./bin
  2. $ chmod +x linkis-cli
  3. $ ./linkis-cli -engineType shell-1 -codeType shell -code "echo \"hello\" "  -submitUser <submitUser> -proxyUser <proxyUser>

The following output is expected.

  1. =====Java Start Command=====
  2. exec /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-2.el8_5.x86_64/jre/bin/java -server -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Linkis/linkis03/logs/linkis-cli -XX:ErrorFile=/Linkis/linkis03/logs/linkis-cli/ps_err_pid%p.log -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC    -classpath /Linkis/linkis03/conf/linkis-cli:/Linkis/linkis03/lib/linkis-computation-governance/linkis-client/linkis-cli/*:/Linkis/linkis03/lib/linkis-commons/public-module/*: -Dconf.root=/Linkis/linkis03/conf/linkis-cli -Dconf.file=linkis-cli.properties -Dlog.path=/Linkis/linkis03/logs/linkis-cli -Dlog.file=linkis-client.root.log.20220418221952287912946  org.apache.linkis.cli.application.LinkisClientApplication '-engineType shell-1 -codeType shell -code echo "hello"  -submitUser test -proxyUser test'
  3. [INFO] LogFile path: /Linkis/linkis03/logs/linkis-cli/linkis-client.root.log.20220418221952287912946
  4. [INFO] User does not provide usr-configuration file. Will use default config
  5. [INFO] connecting to linkis gateway:http://127.0.0.1:9001
  6. JobId:6
  7. TaskId:6
  8. ExecId:exec_id018028linkis-cg-entranceiZbp19q51jb8p984yk2jxdZ:9104LINKISCLI_test_shell_1
  9. [INFO] Job is successfully submitted!
  11. 2022-04-18 22:19:53.019 INFO Program is substituting variables for you
  12. 2022-04-18 22:19:53.019 INFO Variables substitution ended successfully
  13. 2022-04-18 22:19:53.019 WARN The code you submit will not be limited by the limit
  14. Job with jobId : LINKISCLI_test_shell_1 and execID : LINKISCLI_test_shell_1 submitted 
  15. 2022-04-18 22:19:53.019 INFO You have submitted a new job, script code (after variable substitution) is
  16. ************************************SCRIPT CODE************************************
  17. echo "hello"
  18. ************************************SCRIPT CODE************************************
  19. 2022-04-18 22:19:53.019 INFO Your job is accepted,  jobID is LINKISCLI_test_shell_1 and jobReqId is 6 in ServiceInstance(linkis-cg-entrance, iZbp19q51jb8p984yk2jxdZ:9104). Please wait it to be scheduled
  20. job is scheduled.
  21. 2022-04-18 22:19:53.019 INFO Your job is Scheduled. Please wait it to run.
  22. Your job is being scheduled by orchestrator.
  23. 2022-04-18 22:19:53.019 INFO job is running.
  24. 2022-04-18 22:19:53.019 INFO Your job is Running now. Please wait it to complete.
  25. 2022-04-18 22:19:53.019 INFO Job with jobGroupId : 6 and subJobId : 5 was submitted to Orchestrator.
  26. 2022-04-18 22:19:53.019 INFO Background is starting a new engine for you,execId astJob_4_codeExec_4 mark id is mark_4, it may take several seconds, please wait
  27. 2022-04-18 22:20:01.020 INFO Task submit to ec: ServiceInstance(linkis-cg-engineconn, iZbp19q51jb8p984yk2jxdZ:43213) get engineConnExecId is: 1
  28. 2022-04-18 22:20:01.020 INFO EngineConn local log path: ServiceInstance(linkis-cg-engineconn, iZbp19q51jb8p984yk2jxdZ:43213) /appcom1/tmp/test/20220418/shell/cc21fbb5-3a33-471b-a565-8407ff8ebd80/logs
  29. iZbp19q51jb8p984yk2jxdZ:43213_0 >> echo "hello"
  30. 2022-04-18 22:20:01.438 WARN  [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconn.computation.executor.hook.executor.ExecuteOnceHook 50 warn - execute once become effective, register lock listener
  31. hello
  32. 2022-04-18 22:20:01.020 INFO Your subjob : 5 execue with state succeed, has 1 resultsets.
  33. 2022-04-18 22:20:01.020 INFO Congratuaions! Your job : LINKISCLI_test_shell_1 executed with status succeed and 0 results.
  34. 2022-04-18 22:20:01.020 INFO job is completed.
  35. 2022-04-18 22:20:01.020 INFO Task creation time(任务创建时间): 2022-04-18 22:19:53, Task scheduling time(任务调度时间): 2022-04-18 22:19:53, Task start time(任务开始时间): 2022-04-18 22: Mission end time(任务结束时间): 2022-04-18 22:20:01
  36. 2022-04-18 22:20:01.020 INFO Your mission(您的任务) 6 The total time spent is(总耗时时间为): 8.3 秒
  37. 2022-04-18 22:20:01.020 INFO Congratulations. Your job completed with status Success.
  39. [INFO] Job execute successfully! Will try get execute result
  40. ============Result:================
  41. TaskId:6
  42. ExecId: exec_id018028linkis-cg-entranceiZbp19q51jb8p984yk2jxdZ:9104LINKISCLI_test_shell_1
  43. User:test
  44. Current job status:SUCCEED
  45. extraMsg: 
  46. result: 
  48. [INFO] Retrieving result-set, may take time if result-set is large, please do not exit program.
  49. ============ RESULT SET 1 ============
  50. hello   
  51. ############Execute Success!!!########