OpenLooKeng Engine

This article mainly introduces the installation, usage and configuration of the openLooKeng engine plugin in Linkis `.

If you wish to deploy the openLooKeng engine, you need to prepare a working openLooKeng environment.

  1. # Prepare hetu-cli
  2. wget https://download.openlookeng.io/1.5.0/hetu-cli-1.5.0-executable.jar
  3. mv hetu-cli-1.5.0-executable.jar hetu-cli
  4. chmod +x hetu-cli
  6. # link service
  7. ./hetu-cli --server 127.0.0.1:9090 --catalog tpcds --schema default
  9. # Execute query statement
  10. lk:default> select d_date_sk, d_date_id, d_date, d_month_seq from tpcds.sf1.date_dim order by d_date limit 5;
  12. # Get the following output to represent the service is available
  13.  d_date_sk |    d_date_id     |   d_date   | d_month_seq
  14. -------------+--------+------------+------ -------
  15.    2415022 | AAAAAAAAOKJNECAA | 1900-01-02 | 0
  16.    2415023 | AAAAAAAAPKJNECAA | 1900-01-03 | 0
  17.    2415024 | AAAAAAAAALJNECAA | 1900-01-04 | 0
  18.    2415025 | AAAAAAAABLJNECAA | 1900-01-05 | 0
  19.    2415026 | AAAAAAAACLJNECAA | 1900-01-06 | 0
  20. (5 rows)
  22. Query 20221110_043803_00011_m9gmv, FINISHED, 1 node
  23. Splits: 33 total, 33 done (100.00%)
  24. 0:00 [73K rows, 0B] [86.8K rows/s, 0B/s]

Method 1: Download the engine plug-in package directly

Linkis Engine Plugin Download

Method 2: Compile the engine plug-in separately (requires a maven environment)

  1. # compile
  2. ${linkis_code_dir}/linkis-enginepconn-pugins/engineconn-plugins/openlookeng/
  3. mvn clean install
  4. # The compiled engine plug-in package is located in the following directory
  5. ${linkis_code_dir}/linkis-engineconn-plugins/openlookeng/target/out/

EngineConnPlugin Engine Plugin Installation

Upload the engine plug-in package in 2.1 to the engine directory of the server

  1. ${LINKIS_HOME}/lib/linkis-engineconn-plugins

The directory structure after uploading is as follows

  1. linkis-engineconn-plugins/
  2. ├── openlookeng
  3. │   ├── dist
  4. │ │ └── 1.5.0
  5. │   │       ├── conf
  6. │ │ └── lib
  7. │   └── plugin
  8. │ └── 1.5.0

Refresh the engine by restarting the linkis-cg-linkismanager service

  1. cd ${LINKIS_HOME}/sbin
  2. sh linkis-daemon.sh restart cg-linkismanager

You can check whether the last_update_time of the linkis_engine_conn_plugin_bml_resources table in the database is the time to trigger the refresh.

  1. #Login to the `linkis` database
  2. select * from linkis_cg_engine_conn_plugin_bml_resources;
  1. sh ./bin/linkis-cli -engineType openlookeng-1.5.0 \
  2. -codeType sql -code 'select * from tpcds.sf1.date_dim;' \
  3. -submitUser hadoop -proxyUser hadoop \
  4. -runtimeMap linkis.openlookeng.url=http://127.0.0.1:8080

More Linkis-Cli command parameter reference: Linkis-Cli usage

Linkis provides SDK of Java and Scala to submit tasks to Linkis server. For details, please refer to JAVA SDK Manual. For JDBC tasks you only need to modify the EngineConnType and CodeType parameters in Demo:

  1. Map<String, Object> labels = new HashMap<String, Object>();
  2. labels.put(LabelKeyConstant.ENGINE_TYPE_KEY, "openlookeng-1.5.0"); // required engineType Label
  3. labels.put(LabelKeyConstant.USER_CREATOR_TYPE_KEY, "hadoop-IDE");// required execute user and creator
  4. labels.put(LabelKeyConstant.CODE_TYPE_KEY, "sql"); // required codeType
ConfigurationDefaultRequiredDescription
linkis.openlookeng.urlhttp://127.0.0.1:8080yeslink address
linkis.openlookeng.catalogsystemyescatalog
linkis.openlookeng.sourceglobalnosource

If the default parameters are not satisfied, there are the following ways to configure some basic parameters

OpenLooKeng Engine - 图1

Note: After modifying the configuration under the IDE label, you need to specify -creator IDE to take effect (other labels are similar), such as:

  1. sh ./bin/linkis-cli -creator IDE \
  2. -engineType openlookeng-1.5.0 -codeType sql \
  3. -code 'select * from tpcds.sf1.date_dim;' \
  4. -submitUser hadoop -proxyUser hadoop

Submit the task interface, configure it through the parameter params.configuration.runtime

  1. Example of http request parameters
  2. {
  3.     "executionContent": {"code": "select * from tpcds.sf1.date_dim;", "runType":  "sql"},
  4.     "params": {
  5.                     "variable": {},
  6.                     "configuration": {
  7.                             "runtime": {
  8.                                 "linkis.openlookeng.url":"http://127.0.0.1:9090"
  9.                                 }
  10.                             }
  11.                     },
  12.     "labels": {
  13.         "engineType": "openlookeng-1.5.0",
  14.         "userCreator": "hadoop-IDE"
  15.     }
  16. }

Linkis is managed through the engine tag, and the data table information involved is shown below.

  1. linkis_ps_configuration_config_key: key and default values ​​of configuration parameters inserted into the engine
  2. linkis_cg_manager_label: insert engine label such as: openlookeng-1.5.0
  3. linkis_ps_configuration_category: Insert the directory association of the engine
  4. linkis_ps_configuration_config_value: Insert the configuration that the engine needs to display
  5. linkis_ps_configuration_key_engine_relation: The relationship between the configuration item and the engine

The initial data related to the engine in the table is as follows

  1. -- set variable
  2. SET @OPENLOOKENG_LABEL="openlookeng-1.5.0";
  3. SET @OPENLOOKENG_ALL=CONCAT('*-*,',@OPENLOOKENG_LABEL);
  4. SET @OPENLOOKENG_IDE=CONCAT('*-IDE,',@OPENLOOKENG_LABEL);
  6. -- engine label
  7. insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType', @OPENLOOKENG_ALL, 'OPTIONAL', 2, now(), now());
  8. insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType', @OPENLOOKENG_IDE, 'OPTIONAL', 2, now(), now());
  10. select @label_id := id from linkis_cg_manager_label where `label_value` = @OPENLOOKENG_IDE;
  11. insert into linkis_ps_configuration_category (`label_id`, `level`) VALUES (@label_id, 2);
  13. -- configuration key
  14. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.openlookeng.url', 'eg: http://127.0.0.1:8080', 'connection address', 'http://127.0.0.1:8080', 'Regex', '^\\s *http://([^:]+)(:\\d+)(/[^\\?]+)?(\\?\\S*)?$', 'openlookeng', 0, 0, 1, 'data source conf');
  15. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.openlookeng.catalog', 'catalog', 'catalog', 'system', 'None', '', 'openlookeng', 0, 0, 1, 'data source conf');
  16. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.openlookeng.source', 'source', 'source', 'global', 'None', '', 'openlookeng', 0, 0, 1, 'data source conf');
  18. -- key engine relation
  19. insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`)
  20. (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config
  21. INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'openlookeng' and label_value = @OPENLOOKENG_ALL);
  23. -- openlookeng default configuration
  24. insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`)
  25. (select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation
  26. INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @OPENLOOKENG_ALL);