Linkis 去 HDFS 部署

这篇文章介绍一下如何在没有部署 HDFS 的环境中部署 Linkis 服务，以方便更轻量化的学习使用和调试。

部署整体流程参考 “快速部署”，需要修改如下少许内容。

去HDFS模式部署不支持hive/spark/flink引擎等任务

1. 配置修改

修改 linkis-env.sh 文件，修改以下内容：

  1. #使用[file://]路径模式代替[hdfs://]模式 
  2. WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis/ 
  3. HDFS_USER_ROOT_PATH=file:///tmp/linkis 
  4. RESULT_SET_ROOT_PATH=file:///tmp/linkis 
  7. export ENABLE_HDFS=false
  8. export ENABLE_HIVE=false
  9. export ENABLE_SPARK=false

将上述的配置修改为 false 之后，就不需要再单独配置 HDFS/HIVE/SPARK 等环境了, 安装部署时也会跳过这些基础环境的检查。

2. 拷贝 jar 包

因为 mysql-connector-java 驱动是 GPL2.0 协议，不满足Apache开源协议关于license的政策，所以需要我们手动将其拷贝到如下两个目录下

  1. ${LINKIS_HOME}/lib/linkis-commons/public-module/
  2. ${LINKIS_HOME}/lib/linkis-spring-cloud-services/linkis-mg-gateway/

可以直接从 maven 仓库下载，比如 https://mvnrepository.com/artifact/mysql/mysql-connector-java/5.1.49

3. 部署和启动

参考 “快速部署” 部分，通过执行 ${LINKIS_HOME}/bin/install.sh 命令完成部署。 参考 “快速部署” 部分，通过执行 ${LINKIS_HOME}/sbin/linkis-start-all.sh 命令启动 Linkis 服务。

4. 运行作业验证

目前 1.1.2 版本只支持 shell 作业在非 HDFS 环境中运行，执行命令如下

  1. $ cd ./bin
  2. $ chmod +x linkis-cli
  3. $ ./linkis-cli -engineType shell-1 -codeType shell -code "echo \"hello\" "  -submitUser <submitUser> -proxyUser <proxyUser>

正常会输出如下类似内容

  1. =====Java Start Command=====
  2. exec /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-2.el8_5.x86_64/jre/bin/java -server -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Linkis/linkis03/logs/linkis-cli -XX:ErrorFile=/Linkis/linkis03/logs/linkis-cli/ps_err_pid%p.log -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC    -classpath /Linkis/linkis03/conf/linkis-cli:/Linkis/linkis03/lib/linkis-computation-governance/linkis-client/linkis-cli/*:/Linkis/linkis03/lib/linkis-commons/public-module/*: -Dconf.root=/Linkis/linkis03/conf/linkis-cli -Dconf.file=linkis-cli.properties -Dlog.path=/Linkis/linkis03/logs/linkis-cli -Dlog.file=linkis-client.root.log.20220418221952287912946  org.apache.linkis.cli.application.LinkisClientApplication '-engineType shell-1 -codeType shell -code echo "hello"  -submitUser test -proxyUser test'
  3. [INFO] LogFile path: /Linkis/linkis03/logs/linkis-cli/linkis-client.root.log.20220418221952287912946
  4. [INFO] User does not provide usr-configuration file. Will use default config
  5. [INFO] connecting to linkis gateway:http://127.0.0.1:9001
  6. JobId:6
  7. TaskId:6
  8. ExecId:exec_id018028linkis-cg-entranceiZbp19q51jb8p984yk2jxdZ:9104LINKISCLI_test_shell_1
  9. [INFO] Job is successfully submitted!
  12. 2022-04-18 22:19:53.019 INFO Program is substituting variables for you
  13. 2022-04-18 22:19:53.019 INFO Variables substitution ended successfully
  14. 2022-04-18 22:19:53.019 WARN The code you submit will not be limited by the limit
  15. Job with jobId : LINKISCLI_test_shell_1 and execID : LINKISCLI_test_shell_1 submitted 
  16. 2022-04-18 22:19:53.019 INFO You have submitted a new job, script code (after variable substitution) is
  17. ************************************SCRIPT CODE************************************
  18. echo "hello"
  19. ************************************SCRIPT CODE************************************
  20. 2022-04-18 22:19:53.019 INFO Your job is accepted,  jobID is LINKISCLI_test_shell_1 and jobReqId is 6 in ServiceInstance(linkis-cg-entrance, iZbp19q51jb8p984yk2jxdZ:9104). Please wait it to be scheduled
  21. job is scheduled.
  22. 2022-04-18 22:19:53.019 INFO Your job is Scheduled. Please wait it to run.
  23. Your job is being scheduled by orchestrator.
  24. 2022-04-18 22:19:53.019 INFO job is running.
  25. 2022-04-18 22:19:53.019 INFO Your job is Running now. Please wait it to complete.
  26. 2022-04-18 22:19:53.019 INFO Job with jobGroupId : 6 and subJobId : 5 was submitted to Orchestrator.
  27. 2022-04-18 22:19:53.019 INFO Background is starting a new engine for you,execId astJob_4_codeExec_4 mark id is mark_4, it may take several seconds, please wait
  28. 2022-04-18 22:20:01.020 INFO Task submit to ec: ServiceInstance(linkis-cg-engineconn, iZbp19q51jb8p984yk2jxdZ:43213) get engineConnExecId is: 1
  29. 2022-04-18 22:20:01.020 INFO EngineConn local log path: ServiceInstance(linkis-cg-engineconn, iZbp19q51jb8p984yk2jxdZ:43213) /appcom1/tmp/test/20220418/shell/cc21fbb5-3a33-471b-a565-8407ff8ebd80/logs
  30. iZbp19q51jb8p984yk2jxdZ:43213_0 >> echo "hello"
  31. 2022-04-18 22:20:01.438 WARN  [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconn.computation.executor.hook.executor.ExecuteOnceHook 50 warn - execute once become effective, register lock listener
  32. hello
  33. 2022-04-18 22:20:01.020 INFO Your subjob : 5 execue with state succeed, has 1 resultsets.
  34. 2022-04-18 22:20:01.020 INFO Congratuaions! Your job : LINKISCLI_test_shell_1 executed with status succeed and 0 results.
  35. 2022-04-18 22:20:01.020 INFO job is completed.
  36. 2022-04-18 22:20:01.020 INFO Task creation time(任务创建时间): 2022-04-18 22:19:53, Task scheduling time(任务调度时间): 2022-04-18 22:19:53, Task start time(任务开始时间): 2022-04-18 22: Mission end time(任务结束时间): 2022-04-18 22:20:01
  37. 2022-04-18 22:20:01.020 INFO Your mission(您的任务) 6 The total time spent is(总耗时时间为): 8.3 秒
  38. 2022-04-18 22:20:01.020 INFO Congratulations. Your job completed with status Success.
  41. [INFO] Job execute successfully! Will try get execute result
  42. ============Result:================
  43. TaskId:6
  44. ExecId: exec_id018028linkis-cg-entranceiZbp19q51jb8p984yk2jxdZ:9104LINKISCLI_test_shell_1
  45. User:test
  46. Current job status:SUCCEED
  47. extraMsg: 
  48. result: 
  51. [INFO] Retrieving result-set, may take time if result-set is large, please do not exit program.
  52. ============ RESULT SET 1 ============
  53. hello   
  54. ############Execute Success!!!########