Impala

本文主要介绍在 Linkis 中，Impala 引擎插件的安装、使用和配置。

如果您希望在您的服务器上使用 Impala 引擎，您需要准备 Impala 服务并提供连接信息，如 Impala 集群的连接地址、SASL用户名和密码等

执行 impala-shell 命令得到如下输出代表 impala 服务可用。

  1. [root@8f43473645b1 /]# impala-shell
  2. Starting Impala Shell without Kerberos authentication
  3. Connected to 8f43473645b1:21000
  4. Server version: impalad version 2.12.0-cdh5.15.0 RELEASE (build 23f574543323301846b41fa5433690df32efe085)
  5. ***********************************************************************************
  6. Welcome to the Impala shell.
  7. (Impala Shell v2.12.0-cdh5.15.0 (23f5745) built on Thu May 24 04:07:31 PDT 2018)
  9. When pretty-printing is disabled, you can use the '--output_delimiter' flag to set
  10. the delimiter for fields in the same row. The default is ','.
  11. ***********************************************************************************
  12. [8f43473645b1:21000] >

编译 Impala 引擎之前需要进行 Linkis 项目全量编译 , Linkis 默认发布的安装部署包中默认不包含此引擎插件。

方式一：直接下载引擎插件包

Linkis 引擎插件下载

方式二：单独编译引擎插件（需要有 maven 环境）

  1. # 编译
  2. cd ${linkis_code_dir}/linkis-engineconn-plugins/impala/
  3. mvn clean install
  4. # 编译出来的引擎插件包，位于如下目录中
  5. ${linkis_code_dir}/linkis-engineconn-plugins/impala/target/out/

EngineConnPlugin 引擎插件安装

将 2.1 中的引擎包上传到服务器的引擎目录下

  1. ${LINKIS_HOME}/lib/linkis-engineconn-plugins

上传后目录结构如下所示

  1. linkis-engineconn-plugins/
  2. ├── impala
  3. │   ├── dist
  4. │   │   └── 3.4.0
  5. │   │       ├── conf
  6. │   │       └── lib
  7. │   └── plugin
  8. │       └── 3.4.0

通过重启 linkis-cg-linkismanager 服务刷新引擎

  1. cd ${LINKIS_HOME}/sbin
  2. sh linkis-daemon.sh restart cg-linkismanager

可以查看数据库中的 linkis_engine_conn_plugin_bml_resources 这张表的last_update_time 是否为触发刷新的时间。

  1. #登陆到 `linkis` 的数据库 
  2. select * from linkis_cg_engine_conn_plugin_bml_resources;
  1. sh ./bin/linkis-cli -submitUser impala \
  2. -engineType impala-3.4.0 -code 'show databases;' \
  3. -runtimeMap linkis.es.http.method=GET \
  4. -runtimeMap linkis.impala.servers=127.0.0.1:21050

更多 Linkis-Cli 命令参数参考： Linkis-Cli 使用

配置默认值说明是否必须
linkis.impala.default.limit5000是查询的结果集返回条数限制
linkis.impala.engine.user${HDFS_ROOT_USER}是默认引擎启动用户
linkis.impala.user.isolation.modefalse是以多用户模式启动引擎
linkis.impala.servers127.0.0.1:21050是Impala服务器地址，’,’分隔
linkis.impala.maxConnections10是对每台Impala服务器的连接数上限
linkis.impala.ssl.enablefalse是是否启用SSL连接
linkis.impala.ssl.keystore.typeJKS否SSL Keystore类型
linkis.impala.ssl.keystorenull否SSL Keystore路径
linkis.impala.ssl.keystore.passwordnull否SSL Keystore密码
linkis.impala.ssl.truststore.typeJKS否SSL Truststore类型
linkis.impala.ssl.truststorenull否SSL Truststore路径
linkis.impala.ssl.truststore.passwordnull否SSL Truststore密码
linkis.impala.sasl.enablefalse是是否启用SASL认证
linkis.impala.sasl.mechanismPLAIN否SASL Mechanism
linkis.impala.sasl.authorizationIdnull否SASL AuthorizationId
linkis.impala.sasl.protocolLDAP否SASL Protocol
linkis.impala.sasl.propertiesnull否SASL Properties: key1=value1,key2=value2
linkis.impala.sasl.username${impala.engine.user}否SASL Username
linkis.impala.sasl.passwordnull否SASL Password
linkis.impala.sasl.password.cmdnull否SASL Password获取命令
linkis.impala.heartbeat.seconds1是任务状态更新间隔
linkis.impala.query.timeout.seconds0否任务执行超时时间
linkis.impala.query.batchSize1000是结果集获取批次大小
linkis.impala.query.optionsnull否查询提交参数: key1=value1,key2=value2

如果默认参数不满足时，有如下几中方式可以进行一些基础参数配置

提交任务接口，通过参数 params.configuration.runtime 进行配置

  1. http 请求参数示例 
  2. {
  3.     "executionContent": {"code": "show databases;", "runType":  "sql"},
  4.     "params": {
  5.                     "variable": {},
  6.                     "configuration": {
  7.                             "runtime": {
  8.                                 "linkis.impala.servers"="127.0.0.1:21050"
  9.                             }
  10.                     }
  11.                 },
  12.     "labels": {
  13.         "engineType": "impala-3.4.0",
  14.         "userCreator": "hadoop-IDE"
  15.     }
  16. }

Linkis 是通过引擎标签来进行管理的，所涉及的数据表信息如下所示。

  1. linkis_ps_configuration_config_key:  插入引擎的配置参数的key和默认values
  2. linkis_cg_manager_label：插入引擎label如：impala-3.4.0
  3. linkis_ps_configuration_category： 插入引擎的目录关联关系
  4. linkis_ps_configuration_config_value： 插入引擎需要展示的配置
  5. linkis_ps_configuration_key_engine_relation:配置项和引擎的关联关系

表中与引擎相关的初始数据如下

  1. -- set variable
  2. SET @ENGINE_LABEL="impala-3.4.0";
  3. SET @ENGINE_IDE=CONCAT('*-IDE,',@ENGINE_LABEL);
  4. SET @ENGINE_ALL=CONCAT('*-*,',@ENGINE_LABEL);
  5. SET @ENGINE_NAME="impala";
  7. -- add impala engine to IDE
  8. insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType', @ENGINE_ALL, 'OPTIONAL', 2, now(), now());
  9. insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType', @ENGINE_IDE, 'OPTIONAL', 2, now(), now());
  10. select @label_id := id from `linkis_cg_manager_label` where label_value = @ENGINE_IDE;
  11. insert into `linkis_ps_configuration_category` (`label_id`, `level`) VALUES (@label_id, 2);
  13. -- insert configuration key
  14. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.default.limit', '查询的结果集返回条数限制', '结果集条数限制', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  15. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.engine.user', '默认引擎启动用户', '默认启动用户', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  16. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.user.isolation.mode', '以多用户模式启动引擎', '多用户模式', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  17. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.servers', 'Impala服务器地址', '服务地址', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  18. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.maxConnections ', '对每台Impala服务器的连接数上限', '最大连接数', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  19. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.ssl.enable', '是否启用SSL连接', '启用SSL', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  20. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.ssl.keystore.type', 'SSL Keystore类型', 'SSL Keystore类型', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  21. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.ssl.keystore', 'SSL Keystore路径', 'SSL Keystore路径', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  22. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.ssl.keystore.password', 'SSL Keystore密码', 'SSL Keystore密码', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  23. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.ssl.truststore.type', 'SSL Truststore类型', 'SSL Truststore类型', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  24. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.ssl.truststore', 'SSL Truststore路径', 'SSL Truststore路径', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  25. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.ssl.truststore.password', 'SSL Truststore密码', 'SSL Truststore密码', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  26. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.sasl.enable', '是否启用SASL认证', '启用SASL', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  27. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.sasl.mechanism', 'SASL Mechanism', 'SASL Mechanism', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  28. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.sasl.authorizationId', 'SASL AuthorizationId', 'SASL AuthorizationId', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  29. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.sasl.protocol', 'SASL Protocol', 'SASL Protocol', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  30. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.sasl.properties', 'SASL Properties: key1=value1,key2=value2', 'SASL Properties', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  31. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.sasl.username', 'SASL Username', 'SASL Username', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  32. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.sasl.password', 'SASL Password', 'SASL Password', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  33. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.sasl.password.cmd', 'SASL Password获取命令', 'SASL Password获取命令', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  34. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.heartbeat.seconds', '任务状态更新间隔', '任务状态更新间隔', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  35. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.query.timeout.seconds', '任务执行超时时间', '任务执行超时时间', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  36. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.query.batchSize', '结果集获取批次大小', '结果集获取批次大小', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  37. INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.impala.query.options', '查询提交参数: key1=value1,key2=value2', '查询提交参数', 'null', 'None', '', @ENGINE_NAME, 0, 0, 1, '数据源配置');
  38. -- impala engine -*
  39. insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`)
  40. (select config.id as config_key_id, label.id AS engine_type_label_id FROM `linkis_ps_configuration_config_key` config
  41. INNER JOIN `linkis_cg_manager_label` label ON config.engine_conn_type = @ENGINE_NAME and label_value = @ENGINE_ALL);
  42. -- impala engine default configuration
  43. insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`)
  44. (select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM `linkis_ps_configuration_key_engine_relation` relation
  45. INNER JOIN `linkis_cg_manager_label` label ON relation.engine_type_label_id = label.id AND label.label_value = @ENGINE_ALL);