Linkis Debug

Introduction: This article records in detail how to configure and start various microservices of Linkis in IDEA, and implement the submission and execution of scripts such as JDBC, Python, and Shell. On Mac OS, each microservice of Linkis supports local debugging. However, on Windows OS, linkis-cg-engineplugin and linkis-cg-engineconnmanager do not support local debugging for the time being. You can refer to the remote debugging documentation in Section 4 below for debugging.

Before version 1.0.3, linkis has not yet entered apache incubation. The organization still belongs to webank. The package name of the main class is `org.apache.linkis`. Pay attention to the distinction when debugging.

1. Code debugging environment

  • jdk1.8
  • maven3.5+

2. Prepare the code and compile

  1. git clone git@github.com:apache/incubator-linkis.git
  2. cd incubator-linkis
  3. git checkout dev-1.2.0

Clone the source code of Linkis to the local, and open it with IDEA. When you open the project for the first time, the dependency jar package required for the compilation of the Linkis project will be downloaded from the maven repository. When the dependent jar package is loaded, run the following compile and package command.

  1. mvn -N install
  2. mvn clean install

After the compilation command runs successfully, the compiled installation package can be found in the directory incubator-linkis/linkis-dist/target/: apache-linkis-version-incubating-bin.tar.gz

3. Configure and start the service

3.1 add mysql-connector-java to the classpath

If the mysql driver class cannot be found during the service startup, you can add mysql-connector-java-version.jar to the classpath of the corresponding service module. For details, please refer to Section 3.5.

The services currently relying on mysql are:

  • linkis-mg-gateway
  • linkis-ps-publicservice
  • linkis-cg-linkismanage

3.2 Adjust log4j2.xml configuration

Under the Linkis source code folder, in the subdirectory linkis-dist/package/conf, are some default configuration files of Linkis. First, edit the log4j2.xml file, and add the configuration of log output to the console.

log4j2.xml

Only the configuration content that needs to be added is posted here.

  1. <configuration status="error" monitorInterval="30">
  2.     <appenders>
  3.         <Console name="Console" target="SYSTEM_OUT">
  4.             <ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
  5.             <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/>
  6.         </Console>
  7.     </appenders>
  8.     <loggers>
  9.         <root level="INFO">
  10.             <appender-ref ref="Console"/>
  11.         </root>
  12.     </loggers>
  13. </configuration>

3.3 Start the eureka service

Both Linkis and DSS services depend on Eureka, so we need to start the Eureka service first. The Eureka service can be started locally or remotely. After ensuring that each service can access Eureka’s IP and port, you can start to start other microservices.

Inside Linkis, the application name and configuration file are set through the -DserviceName parameter, so -DserviceName is a VM startup parameter that must be specified.

You can use the “-Xbootclasspath/a: configuration file path” command to append the configuration file to the end of the bootstrap class path, that is, add the dependent configuration file to the classpath.

By checking Include dependencies with “Provided” scope, you can introduce provided-level dependency packages during debugging.

eureka

Parameter explanation:

  1. [service name]
  2. linkis-mg-eureka
  5. [Use classpath of module]
  6. linkis-eureka
  9. [Main Class]
  10. org.apache.linkis.eureka.SpringCloudEurekaApplication
  13. [VM Opitons]
  14. -DserviceName=linkis-mg-eureka -Xbootclasspath/a:/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-dist/package/conf
  17. [Program arguments]
  18. --spring.profiles.active=eureka --eureka.instance.preferIpAddress=true

Note that the local path involved in the debugging configuration needs to be modified to the path set by yourself; The path writing rule in Windows is: D:\yourDir\incubator-linkis\linkis-dist\package\conf (The same applies to the following microservices)

If you don’t want the default 20303 port, you can modify the port configuration:

  1. File path: conf/application-eureka.yml
  2. Modify the port:
  3. server:
  4.    port: 8080 ##Starting port

After the above settings are completed, run the Application directly. After successful startup, you can view the eureka service list through http://localhost:20303/.

eureka-web

3.4 Start linkis-mg-gateway

linkis-mg-gateway is the service gateway of Linkis, and all requests will be forwarded to the corresponding service through the gateway.

Before starting the server, you first need to edit the conf/linkis-mg-gateway.properties configuration file and add the administrator username and password. The username must be the same as the mac username you are currently logged in to.

  1. wds.linkis.admin.user=leojie
  2. wds.linkis.admin.password=123456

Set the startup Application of linkis-mg-gateway

gateway-app

Parameter explanation:

  1. [Service Name]
  2. linkis-mg-gateway
  5. [Use classpath of module]
  6. linkis-gateway-server-support
  9. [VM Opitons]
  10. -DserviceName=linkis-mg-gateway -Xbootclasspath/a:/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-dist/package/conf
  13. [main Class]
  14. org.apache.linkis.gateway.springcloud.LinkisGatewayApplication

After the above settings are completed, the Application can be run directly.

3.5 Start linkis-ps-publicservice

publicservice is a public enhancement service of Linkis, a module that provides functions such as unified configuration management, context service, material library, data source management, microservice management and historical task query for other microservice modules.

Set the startup Application of linkis-ps-publicservice

publicservice-app

Parameter explanation:

  1. [Service Name]
  2. linkis-ps-publicservice
  5. [Module Name]
  6. linkis-public-enhancements
  9. [VM Opitons]
  10. -DserviceName=linkis-ps-publicservice -Xbootclasspath/a:/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-dist/package/conf 
  13. [main Class]
  14. org.apache.linkis.filesystem.LinkisPublicServiceApp
  17. [Add provided scope to classpath]
  18. By checking Include dependencies with "Provided" scope, you can introduce provided-level dependency packages during debugging.

When starting publicservice directly, you may encounter the following errors:

publicservice-debug-error

The public-module module needs to be added to the classpath of the linkis-public-enhancements module. The detailed steps are as follows:

step-1

step-2

step-3

step-4

After completing the above configuration, restart the Application of publicservice

3.6 Start linkis-ps-cs

Before starting the ps-cs service, you need to ensure that the publicservice service starts successfully.

ps-cs-App

Parameter explanation:

  1. [Service Name]
  2. linkis-ps-cs
  5. [Use classpath of module]
  6. linkis-cs-server
  9. [VM Opitons]
  10. -DserviceName=linkis-ps-cs -Xbootclasspath/a:/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-dist/package/conf 
  13. [main Class]
  14. org.apache.linkis.cs.server.LinkisCSApplication
  17. [Add provided scope to classpath]
  18. By checking Include dependencies with "Provided" scope, you can introduce provided-level dependency packages during debugging.

3.7 Start linkis-cg-linkismanager

cg-linkismanager-APP

Parameter explanation:

  1. [Service Name]
  2. linkis-cg-linkismanager
  5. [Use classpath of module]
  6. linkis-application-manager
  9. [VM Opitons]
  10. -DserviceName=linkis-cg-linkismanager -Xbootclasspath/a:/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-dist/package/conf
  13. [main Class]
  14. org.apache.linkis.manager.am.LinkisManagerApplication
  17. [Add provided scope to classpath]
  18. By checking Include dependencies with "Provided" scope, you can introduce provided-level dependency packages during debugging.

3.8 Start linkis-cg-entrance

cg-entrance-APP

Parameter explanation:

  1. [Service Name]
  2. linkis-cg-entrance
  5. [Use classpath of module]
  6. linkis-entrance
  9. [VM Opitons]
  10. -DserviceName=linkis-cg-entrance -Xbootclasspath/a:D:\yourDir\incubator-linkis\linkis-dist\package\conf
  13. [main Class]
  14. org.apache.linkis.entrance.LinkisEntranceApplication
  17. [Add provided scope to classpath]
  18. By checking Include dependencies with "Provided" scope, you can introduce provided-level dependency packages during debugging.

3.9 Start cg-engineconnmanager

engineconnmanager-app

Parameter explanation:

  1. [Service Name]
  2. linkis-cg-engineconnmanager
  5. [Use classpath of module]
  6. linkis-engineconn-manager-server
  9. [VM Opitons]
  10. -DserviceName=linkis-cg-engineconnmanager -Xbootclasspath/a:/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-dist/package/conf -DJAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/
  13. [main Class]
  14. org.apache.linkis.ecm.server.LinkisECMApplication
  17. [Add provided scope to classpath]
  18. By checking Include dependencies with "Provided" scope, you can introduce provided-level dependency packages during debugging.

-DJAVA_HOME is to specify the path of the java command used by ecm to start the engine. If the version in the default JAVA environment variable meets your needs, this configuration can be omitted.

Debugging for linkis-cg-engineplugin module only supports Mac OS

3.10 Start linkis-cg-engineplugin

engineplugin-app

Parameter explanation:

  1. [Service Name]
  2. linkis-cg-engineplugin
  5. [Use classpath of module]
  6. linkis-engineconn-plugin-server
  9. [VM Opitons]
  10. -DserviceName=linkis-cg-engineplugin -Xbootclasspath/a:/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-dist/package/conf
  13. [main Class]
  14. org.apache.linkis.engineplugin.server.LinkisEngineConnPluginServer
  17. [Add provided scope to classpath]
  18. By checking Include dependencies with "Provided" scope, you can introduce provided-level dependency packages during debugging.

When starting engineplugin, you may encounter the following error:

engineplugin-debug-error

The solution is to add the public-module module to the classpath of the linkis-engineconn-plugin-server module, refer to section 3.5

Debugging for linkis-cg-engineplugin module only supports Mac OS

3.11 Key Configuration Modifications

The above operation only completes the configuration of the application startup of each Linkis microservice. In addition, in the configuration file loaded when the Linkis service starts, some key configurations also need to be modified in a targeted manner, otherwise the process of starting the service or script execution Some errors will be encountered. The key configuration modifications are summarized as follows:

3.11.1 conf/linkis.properties

  1. # linkis underlying database connection parameter configuration
  2. wds.linkis.server.mybatis.datasource.url=jdbc:mysql://yourip:3306/linkis?characterEncoding=UTF-8
  3. wds.linkis.server.mybatis.datasource.username=your username
  4. wds.linkis.server.mybatis.datasource.password=your password
  7. # Set the bml material storage path to not hdfs
  8. wds.linkis.bml.is.hdfs=false
  9. wds.linkis.bml.local.prefix=/Users/leojie/software/linkis/data/bml
  12. wds.linkis.home=/Users/leojie/software/linkis
  15. # Set the administrator username, your local username
  16. wds.linkis.governance.station.admin=leojie

Before configuring linkis underlying database connection parameters, please create linkis database and run linkis-dist/package/db/linkis_ddl.sql and linkis-dist/package/db/linkis_dml.sql to initialize all tables and data.

The directory structure of wds.linkis.home=/Users/leojie/software/linkis is as follows, only the lib directory and the conf directory are placed in it. When the engine process starts, the conf and lib paths in wds.linkis.home will be added to the classpath. If wds.linkis.home is not specified, an exception that the directory cannot be found may be encountered.

linkis-home

3.11.2 conf/linkis-cg-entrance.properties

  1. # The log directory of the entrance service execution task
  2. wds.linkis.entrance.config.log.path=file:///Users/leojie/software/linkis/data/entranceConfigLog
  5. # The result set is saved in the directory, the local user needs read and write permissions
  6. wds.linkis.resultSet.store.path=file:///Users/leojie/software/linkis/data/resultSetDir

3.11.3 conf/linkis-cg-engineconnmanager.properties

  1. wds.linkis.engineconn.root.dir=/Users/leojie/software/linkis/data/engineconnRootDir

If you do not modify it, you may encounter an exception that the path does not exist.

3.11.4 conf/linkis-cg-engineplugin.properties

  1. wds.linkis.engineconn.home=/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-engineconn-plugins/shell/target/out
  4. wds.linkis.engineconn.plugin.loader.store.path=/Users/leojie/other_project/apache/linkis/incubator-linkis/linkis-engineconn-plugins/shell/target/out

The two configurations here are mainly to specify the root directory of the engine storage, and the main purpose of specifying it as target/out is that after the engine-related code or configuration changes, the engineplugin service can be restarted directly to take effect.

3.12 Set sudo password-free for the current user

When the engine is started, sudo needs to be used to execute the shell command to start the engine process. The current user on the mac generally needs to enter a password when using sudo. Therefore, it is necessary to set sudo password-free for the current user. The setting method is as follows:

  1. sudo chmod u-w /etc/sudoers
  2. sudo visudo
  3. Replace #%admin ALL=(ALL) AL with %admin ALL=(ALL) NOPASSWD: ALL
  4. save file exit

3.13 Service Testing

Make sure that the above services are all successfully started, and then test and submit the shell script job in postman.

First visit the login interface to generate a cookie:

login

Then submit the shell code for execution

POST: http://127.0.0.1:9001/api/rest_j/v1/entrance/submit

body parameter:

  1. {
  2.   "executionContent": {
  3.     "code": "echo 'hello'",
  4.     "runType": "shell"
  5.   },
  6.   "params": {
  7.     "variable": {
  8.       "testvar": "hello"
  9.     },
  10.     "configuration": {
  11.       "runtime": {},
  12.       "startup": {}
  13.     }
  14.   },
  15.   "source": {
  16.     "scriptPath": "file:///tmp/hadoop/test.sql"
  17.   },
  18.   "labels": {
  19.     "engineType": "shell-1",
  20.     "userCreator": "leojie-IDE"
  21.   }
  22. }

Results of the:

  1. {
  2.     "method": "/api/entrance/submit",
  3.     "status": 0,
  4.     "message": "OK",
  5.     "data": {
  6.         "taskID": 1,
  7.         "execID": "exec_id018017linkis-cg-entrance192.168.3.13:9104IDE_leojie_shell_0"
  8.     }
  9. }

Finally, check the running status of the task and get the running result set:

GET http://127.0.0.1:9001/api/rest_j/v1/entrance/exec_id018017linkis-cg-entrance192.168.3.13:9104IDE_leojie_shell_0/progress

  1. {
  2.     "method": "/api/entrance/exec_id018017linkis-cg-entrance192.168.3.13:9104IDE_leojie_shell_0/progress",
  3.     "status": 0,
  4.     "message": "OK",
  5.     "data": {
  6.         "progress": 1,
  7.         "progressInfo": [],
  8.         "execID": "exec_id018017linkis-cg-entrance192.168.3.13:9104IDE_leojie_shell_0"
  9.     }
  10. }

GET http://127.0.0.1:9001/api/rest_j/v1/jobhistory/1/get

GET http://127.0.0.1:9001/api/rest_j/v1/filesystem/openFile?path=file:///Users/leojie/software/linkis/data/resultSetDir/leojie/linkis/2022-07-16/214859/IDE/1/1_0.dolphin

  1. {
  2.     "method": "/api/filesystem/openFile",
  3.     "status": 0,
  4.     "message": "OK",
  5.     "data": {
  6.         "metadata": "NULL",
  7.         "totalPage": 0,
  8.         "totalLine": 1,
  9.         "page": 1,
  10.         "type": "1",
  11.         "fileContent": [
  12.             [
  13.                 "hello"
  14.             ]
  15.         ]
  16.     }
  17. }

4. Remote debugging service guide

4.1 Open the remote debugging port

Identify the service where the package that needs to be debugged is located, and determine the service to which it belongs according to the location of the code to be debugged

4.2 Enter {LINKIS_HOME}/sbin/ext, modify the module configuration file to open the remote call port

c-port

4.3 Restart the service that needs to be debugged

  1. sh linkis-daemon.sh restart ps-publicservice

(If you are not sure about the service name, check in {LINKIS_HOME}/sbin/linkis-start-all.sh )

4.4 Compiler configuration remote debugging

Open the window as shown below and configure the remote debugging port, service, and module c-debug

4.5 Start debugging

Click the debug button, and the following information appears, indicating that you can start debugging Enterprise WeChat screenshot_16500167527083