Python Engine

This article mainly introduces the installation, use and configuration of the Python engine plugin in Linkis.

If you want to use the python engine on your server, you need to ensure that the user’s PATH has the python execution directory and execution permissions.

  1. python --version

Normal output of Python version information means Python environment is available

  1. Python 3.6.0

The binary installation package released by linkis includes the Python engine plug-in by default, and users do not need to install it additionally.

EngineConnPlugin Engine Plugin Installation

  1. sh ./bin/linkis-cli -engineType python-python2 \
  2. -codeType python -code "print(\"hello\")"  \
  3. -submitUser hadoop -proxyUser hadoop

More Linkis-Cli command parameter reference: Linkis-Cli usage

Linkis provides SDK of Java and Scala to submit tasks to Linkis server. For details, please refer to JAVA SDK Manual. For For Python tasks, you only need to modify EngineConnType and CodeType parameters.

  1. Map<String, Object> labels = new HashMap<String, Object>();
  2. labels.put(LabelKeyConstant.ENGINE_TYPE_KEY, "python-python2"); // required engineType Label
  3. labels.put(LabelKeyConstant.USER_CREATOR_TYPE_KEY, "hadoop-IDE");// required execute user and creator
  4. labels.put(LabelKeyConstant.CODE_TYPE_KEY, "python"); // required codeType

The Python engine plug-in supports python2 and python3, you can simply change the configuration to complete the switch of Python version without recompiling the python engine version. The Python engine supports a variety of configuration modification methods, the specific operations are as follows.

  1. #1: Submit tasks via cli to switch versions, and set the version python.version=python3 at the end of the command (python3: the name of the file generated when creating a soft link, which can be customized)
  2. sh ./bin/linkis-cli -engineType python-python2 \
  3. -codeType python -code "print(\"hello\")"  \
  4. -submitUser hadoop -proxyUser hadoop  \
  5. -confMap  python.version=python3
  7. #2: The cli method is used to submit tasks for version switching, and the command is set to add the version path python.version=/usr/bin/python (/usr/bin/python: the path to the file generated when creating a soft link)
  8. sh ./bin/linkis-cli -engineType python-python2 \
  9. -codeType python -code "print(\"hello\")"  \
  10. -submitUser hadoop -proxyUser hadoop  \
  11. -confMap  python.version=/usr/bin/python

Python Engine - 图1

Note: After modifying the configuration under the IDE tag, you need to specify -creator IDE to take effect (other tags are similar), such as:

  1. sh ./bin/linkis-cli -creator IDE -engineType \
  2. python-python2 -codeType python -code "print(\"hello\")"  \
  3. -submitUser hadoop -proxyUser hadoop  \
  4. -confMap  python.version=python3

Submit the task interface, configure it through the parameter params.configuration.runtime

  1. Example of http request parameters
  2. {
  3.     "executionContent": {"code": "print(\"hello\")", "runType":  "python"},
  4.     "params": {
  5.                 "variable": {},
  6.                 "configuration": {
  7.                         "runtime": {
  8.                                 "python.version":"python2",
  9.                                 "wds.linkis.engineconn.max.free.time":"1h"
  10.                         }
  11.                 }
  12.         },
  13.     "labels": {
  14.         "engineType": "python-python2",
  15.         "userCreator": "IDE"
  16.     }
  17. }

Configure by modifying the linkis-engineconn.properties file in the directory ${LINKIS_HOME}/lib/linkis-engineconn-plugins/python/dist/python2/conf/, as shown below:

Python Engine - 图2

Linkis is managed through engine tags, and the data table information involved is as follows.

  1. linkis_ps_configuration_config_key: Insert the key and default values ​​​​of the configuration parameters of the engine
  2. linkis_cg_manager_label: Insert engine label such as: python-python2
  3. linkis_ps_configuration_category: Insert the directory association of the engine
  4. linkis_ps_configuration_config_value: The configuration that the insertion engine needs to display
  5. linkis_ps_configuration_key_engine_relation: The relationship between the configuration item and the engine

The initial data related to the engine in the table is as follows

  1. -- set variable
  2. SET @PYTHON_LABEL="python-python2";
  3. SET @PYTHON_ALL=CONCAT('*-*,',@PYTHON_LABEL);
  4. SET @PYTHON_IDE=CONCAT('*-IDE,',@PYTHON_LABEL);
  6. -- engine label
  7. insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType', @PYTHON_ALL, 'OPTIONAL', 2, now(), now());
  8. insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType', @PYTHON_IDE, 'OPTIONAL', 2, now(), now());
  10. select @label_id := id from linkis_cg_manager_label where `label_value` = @PYTHON_IDE;
  11. insert into linkis_ps_configuration_category (`label_id`, `level`) VALUES (@label_id, 2);
  13. -- configuration key
  14. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`) VALUES ('wds.linkis.rm.client.memory.max', 'Value range: 1-100, unit: G', 'Python driver memory upper limit', '20G', 'Regex', '^([ 1-9]\\d{0,1}|100)(G|g)$', '0', '0', '1', 'queue resource', 'python');
  15. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`) VALUES ('wds.linkis.rm.client.core.max', 'Value range: 1-128, unit: a', 'Python drive core number upper limit', '10', 'Regex', '^( ?:[1-9]\\d?|[1234]\\d{2}|128)$', '0', '0', '1', 'queue resource', 'python');
  16. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`) VALUES ('wds.linkis.rm.instance', 'Range: 1-20, unit: a', 'Python engine maximum concurrent number', '10', 'NumInterval', '[1,20]', '0 ', '0', '1', 'queue resource', 'python');
  17. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`) VALUES ('wds.linkis.engineconn.java.driver.memory', 'value range: 1-2, unit: G', 'python engine initialization memory size', '1g', 'Regex', '^([ 1-2])(G|g)$', '0', '0', '1', 'python engine settings', 'python');
  18. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`) VALUES ('python.version', 'Value range: python2,python3', 'python version','python2', 'OFT', '[\"python3\",\"python2\"]', '0' , '0', '1', 'python engine settings', 'python');
  19. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`) VALUES ('wds.linkis.engineconn.max.free.time', 'Value range: 3m,15m,30m,1h,2h', 'Engine idle exit time','1h', 'OFT', '[\ "1h\",\"2h\",\"30m\",\"15m\",\"3m\"]', '0', '0', '1', 'python engine settings', ' python');
  21. -- key engine relation
  22. insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`)
  23. (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config
  24. INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'python' and label_value = @PYTHON_ALL);
  26. -- engine default configuration
  27. insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`)
  28. (select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation
  29. INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @PYTHON_ALL);
  1. import pandas as pd
  3. data = {'name': ['aaaaaa', 'bbbbbb', 'cccccc'], 'pay': [4000, 5000, 6000]}
  4. frame = pd.DataFrame(data)
  5. show.show(frame)
  8. print('new reuslt')
  10. from matplotlib import pyplot as plt
  12. x=[4,8,10]
  13. y=[12,16,6]
  14. x2=[6,9,11]
  15. y2=[6,15,7]
  16. plt.bar(x,y,color='r',align='center')
  17. plt.bar(x2,y2,color='g',align='center')
  18. plt.show()