无 HDFS 部署

无 HDFS 部署

这篇文章介绍一下如何在没有部署 HDFS 的环境中部署 Linkis 服务，以方便更轻量化的学习使用和调试。

部署整体流程参考 “快速部署”，需要修改如下少许内容。

去HDFS模式部署不支持hive/spark/flink引擎等任务

修改 linkis-env.sh 文件，修改以下内容：

1. #使用[file://]路径模式代替[hdfs://]模式
2. WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis/
3. HDFS_USER_ROOT_PATH=file:///tmp/linkis
4. RESULT_SET_ROOT_PATH=file:///tmp/linkis

6. export ENABLE_HDFS=false
7. export ENABLE_HIVE=false
8. export ENABLE_SPARK=false

将上述的配置修改为 false 之后，就不需要再单独配置 HDFS/HIVE/SPARK 等环境了, 安装部署时也会跳过这些基础环境的检查。

因为 mysql-connector-java 驱动是 GPL2.0 协议，不满足Apache开源协议关于license的政策，所以需要我们手动将其拷贝到如下两个目录下

1. ${LINKIS_HOME}/lib/linkis-commons/public-module/
2. ${LINKIS_HOME}/lib/linkis-spring-cloud-services/linkis-mg-gateway/

可以直接从 maven 仓库下载，比如 <https://mvnrepository.com/artifact/mysql/mysql-connector-java/5.1.49>

参考 “快速部署” 部分，通过执行 sh bin/install.sh 命令完成部署。参考 “快速部署” 部分，通过执行 ${LINKIS_HOME}/sbin/linkis-start-all.sh 命令启动 Linkis 服务。

目前 1.1.2 版本只支持 shell 作业在非 HDFS 环境中运行，执行命令如下

1. $ cd ./bin
2. $ chmod +x linkis-cli
3. $ ./linkis-cli -engineType shell-1 -codeType shell -code "echo \"hello\" "  -submitUser <submitUser> -proxyUser <proxyUser>

正常会输出如下类似内容

1. =====Java Start Command=====
2. exec /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-2.el8_5.x86_64/jre/bin/java -server -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Linkis/linkis03/logs/linkis-cli -XX:ErrorFile=/Linkis/linkis03/logs/linkis-cli/ps_err_pid%p.log -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC    -classpath /Linkis/linkis03/conf/linkis-cli:/Linkis/linkis03/lib/linkis-computation-governance/linkis-client/linkis-cli/*:/Linkis/linkis03/lib/linkis-commons/public-module/*: -Dconf.root=/Linkis/linkis03/conf/linkis-cli -Dconf.file=linkis-cli.properties -Dlog.path=/Linkis/linkis03/logs/linkis-cli -Dlog.file=linkis-client.root.log.20220418221952287912946  org.apache.linkis.cli.application.LinkisClientApplication '-engineType shell-1 -codeType shell -code echo "hello"  -submitUser test -proxyUser test'
3. [INFO] LogFile path: /Linkis/linkis03/logs/linkis-cli/linkis-client.root.log.20220418221952287912946
4. [INFO] User does not provide usr-configuration file. Will use default config
5. [INFO] connecting to linkis gateway:http://127.0.0.1:9001
6. JobId:6
7. TaskId:6
8. ExecId:exec_id018028linkis-cg-entranceiZbp19q51jb8p984yk2jxdZ:9104LINKISCLI_test_shell_1
9. [INFO] Job is successfully submitted!

11. 2022-04-18 22:19:53.019 INFO Program is substituting variables for you
12. 2022-04-18 22:19:53.019 INFO Variables substitution ended successfully
13. 2022-04-18 22:19:53.019 WARN The code you submit will not be limited by the limit
14. Job with jobId : LINKISCLI_test_shell_1 and execID : LINKISCLI_test_shell_1 submitted
15. 2022-04-18 22:19:53.019 INFO You have submitted a new job, script code (after variable substitution) is
16. ************************************SCRIPT CODE************************************
17. echo "hello"
18. ************************************SCRIPT CODE************************************
19. 2022-04-18 22:19:53.019 INFO Your job is accepted,  jobID is LINKISCLI_test_shell_1 and jobReqId is 6 in ServiceInstance(linkis-cg-entrance, iZbp19q51jb8p984yk2jxdZ:9104). Please wait it to be scheduled
20. job is scheduled.
21. 2022-04-18 22:19:53.019 INFO Your job is Scheduled. Please wait it to run.
22. Your job is being scheduled by orchestrator.
23. 2022-04-18 22:19:53.019 INFO job is running.
24. 2022-04-18 22:19:53.019 INFO Your job is Running now. Please wait it to complete.
25. 2022-04-18 22:19:53.019 INFO Job with jobGroupId : 6 and subJobId : 5 was submitted to Orchestrator.
26. 2022-04-18 22:19:53.019 INFO Background is starting a new engine for you,execId astJob_4_codeExec_4 mark id is mark_4, it may take several seconds, please wait
27. 2022-04-18 22:20:01.020 INFO Task submit to ec: ServiceInstance(linkis-cg-engineconn, iZbp19q51jb8p984yk2jxdZ:43213) get engineConnExecId is: 1
28. 2022-04-18 22:20:01.020 INFO EngineConn local log path: ServiceInstance(linkis-cg-engineconn, iZbp19q51jb8p984yk2jxdZ:43213) /appcom1/tmp/test/20220418/shell/cc21fbb5-3a33-471b-a565-8407ff8ebd80/logs
29. iZbp19q51jb8p984yk2jxdZ:43213_0 >> echo "hello"
30. 2022-04-18 22:20:01.438 WARN  [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconn.computation.executor.hook.executor.ExecuteOnceHook 50 warn - execute once become effective, register lock listener
31. hello
32. 2022-04-18 22:20:01.020 INFO Your subjob : 5 execue with state succeed, has 1 resultsets.
33. 2022-04-18 22:20:01.020 INFO Congratuaions! Your job : LINKISCLI_test_shell_1 executed with status succeed and 0 results.
34. 2022-04-18 22:20:01.020 INFO job is completed.
35. 2022-04-18 22:20:01.020 INFO Task creation time(任务创建时间): 2022-04-18 22:19:53, Task scheduling time(任务调度时间): 2022-04-18 22:19:53, Task start time(任务开始时间): 2022-04-18 22: Mission end time(任务结束时间): 2022-04-18 22:20:01
36. 2022-04-18 22:20:01.020 INFO Your mission(您的任务) 6 The total time spent is(总耗时时间为): 8.3 秒
37. 2022-04-18 22:20:01.020 INFO Congratulations. Your job completed with status Success.

39. [INFO] Job execute successfully! Will try get execute result
40. ============Result:================
41. TaskId:6
42. ExecId: exec_id018028linkis-cg-entranceiZbp19q51jb8p984yk2jxdZ:9104LINKISCLI_test_shell_1
43. User:test
44. Current job status:SUCCEED
45. extraMsg:
46. result:

48. [INFO] Retrieving result-set, may take time if result-set is large, please do not exit program.
49. ============ RESULT SET 1 ============
50. hello
51. ############Execute Success!!!########