Hive Engine

This article mainly introduces the configuration, deployment and use of Hive engineConn in Linkis1.0.

If you want to use the hive engineConn on your linkis server, you need to ensure that the following environment variables have been set correctly and that the user who started the engineConn has these environment variables.

It is strongly recommended that you check these environment variables of the executing user before executing hive tasks.

Environment variable nameEnvironment variable contentRemarks
JAVA_HOMEJDK installation pathRequired
HADOOP_HOMEHadoop installation pathRequired
HADOOP_CONF_DIRHadoop configuration pathRequired
HIVE_CONF_DIRHive configuration pathRequired

Table 1-1 Environmental configuration list

The version of Hive supports hive1.x/hive2.x/hive3.x. The hive version supported by default is 2.3.3. If you want to modify the hive version, such as 2.3.3, you can find the linkis-engineConnplugin-hive module and change the \<hive.version> tag to 2.3 .3, then compile this module separately. The default is to support hive on MapReduce, if you want to change to Hive on Tez, You need to copy all the jars prefixed with tez-* to the directory: ${LINKIS_HOME}/lib/linkis-engineconn-plugins/hive/dist/version/lib. Other hive operating modes are similar, just copy the corresponding dependencies to the lib directory of Hive EngineConn.

If you have already compiled your hive engineConn plug-in has been compiled, then you need to put the new plug-in in the specified location to load, you can refer to the following article for details

EngineConnPlugin Installation

Linkis can configure the corresponding EngineConn parameters on the management console. If your newly added EngineConn needs this feature, you can refer to the following documents:

EngineConnPlugin Installation > 2.2 Configuration modification of management console (optional)

Hive’s MapReduce task requires yarn resources, so you need to set up the queue at the beginning

Hive Engine - 图1

Figure 3-1 Queue settings

You can also add the queue value in the StartUpMap of the submission parameter: startupMap.put("wds.linkis.rm.yarnqueue", "dws")

Linkis provides a client method to call hive tasks. The call method is through the SDK provided by LinkisClient. We provide java and scala two ways to call, the specific usage can refer to JAVA SDK Manual. If you use Hive, you only need to make the following changes:

  1.         Map<String, Object> labels = new HashMap<String, Object>();
  2.         labels.put(LabelKeyConstant.ENGINE_TYPE_KEY, "hive-2.3.3"); // required engineType Label
  3.         labels.put(LabelKeyConstant.USER_CREATOR_TYPE_KEY, "hadoop-IDE");// required execute user and creator
  4.         labels.put(LabelKeyConstant.CODE_TYPE_KEY, "hql"); // required codeType

After Linkis 1.0, you can submit tasks through cli. We only need to specify the corresponding EngineConn and CodeType tag types. The use of Hive is as follows:

  1. sh ./bin/linkis-cli -engineType jdbc-4 -codeType jdbc -code "show tables"  -submitUser hadoop -proxyUser hadoop

The specific usage can refer to Linkis CLI Manual.

The use of Scriptis is the simplest. You can directly enter Scriptis, right-click the directory and create a new hive script and write hivesql code.

The implementation of the hive engineConn is by instantiating the driver instance of hive, and then the driver submits the task, and obtains the result set and displays it.

Hive Engine - 图2

Figure 3-2 Screenshot of the execution effect of hql

In addition to the above engineConn configuration, users can also make custom settings, including the memory size of the hive Driver process, etc.

Hive Engine - 图3

Figure 4-1 User-defined configuration management console of hive

The default log interface does not display the application_id and the number of tasks completed, the user can output the log as needed The code blocks that need to be modified in the log4j2-engineconn.xml/log4j2.xml configuration file in the engine are as follows 1.Need to add under the appenders component

  1.         <Send name="SendPackage" >
  2.             <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/>
  3.         </Send>

2.Need to add under the root component

  1.         <appender-ref ref="SendPackage"/>

3.Need to add under the loggers component

  1.         <logger name="org.apache.hadoop.hive.ql.exec.StatsTask" level="info" additivity="true">
  2.             <appender-ref ref="SendPackage"/>
  3.         </logger>

After making the above relevant modifications, the log can add task progress information, which is displayed in the following style

  1. 2022-04-08 11:06:50.228 INFO  [Linkis-Default-Scheduler-Thread-3] SessionState 1111 printInfo - Status: Running (Executing on YARN cluster with App id application_1631114297082_432445)
  2. 2022-04-08 11:06:50.248 INFO  [Linkis-Default-Scheduler-Thread-3] SessionState 1111 printInfo - Map 1: -/-  Reducer 2: 0/1  
  3. 2022-04-08 11:06:52.417 INFO  [Linkis-Default-Scheduler-Thread-3] SessionState 1111 printInfo - Map 1: 0/1  Reducer 2: 0/1  
  4. 2022-04-08 11:06:55.060 INFO  [Linkis-Default-Scheduler-Thread-3] SessionState 1111 printInfo - Map 1: 0(+1)/1  Reducer 2: 0/1  
  5. 2022-04-08 11:06:57.495 INFO  [Linkis-Default-Scheduler-Thread-3] SessionState 1111 printInfo - Map 1: 1/1  Reducer 2: 0(+1)/1  
  6. 2022-04-08 11:06:57.899 INFO  [Linkis-Default-Scheduler-Thread-3] SessionState 1111 printInfo - Map 1: 1/1  Reducer 2: 1/1

An example of a complete xml configuration file is as follows:

  1. <!--
  2.   ~ Copyright 2019 WeBank
  3.   ~
  4.   ~ Licensed under the Apache License, Version 2.0 (the "License");
  5.   ~ you may not use this file except in compliance with the License.
  6.   ~ You may obtain a copy of the License at
  7.   ~
  8.   ~ http://www.apache.org/licenses/LICENSE-2.0
  9.   ~
  10.   ~ Unless required by applicable law or agreed to in writing, software
  11.   ~ distributed under the License is distributed on an "AS IS" BASIS,
  12.   ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13.   ~ See the License for the specific language governing permissions and
  14.   ~ limitations under the License.
  15.   -->
  17. <configuration status="error" monitorInterval="30">
  18.     <appenders>
  19.         <Console name="Console" target="SYSTEM_OUT">
  20.             <ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
  21.             <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/>
  22.         </Console>
  24.         <Send name="Send" >
  25.             <Filters>
  26.                 <ThresholdFilter level="WARN" onMatch="ACCEPT" onMismatch="DENY" />
  27.             </Filters>
  28.             <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/>
  29.         </Send>
  31.         <Send name="SendPackage" >
  32.             <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/>
  33.         </Send>
  35.         <Console name="stderr" target="SYSTEM_ERR">
  36.             <ThresholdFilter level="ERROR" onMatch="ACCEPT" onMismatch="DENY" />
  37.             <PatternLayout pattern="%d{HH:mm:ss.SSS} %-5level %class{36} %L %M - %msg%xEx%n"/>
  38.         </Console>
  39.     </appenders>
  41.     <loggers>
  42.       <root level="INFO">
  43.             <appender-ref ref="stderr"/>
  44.             <appender-ref ref="Console"/>
  45.             <appender-ref ref="Send"/>
  46.             <appender-ref ref="SendPackage"/>
  47.         </root>
  48.         <logger name="org.apache.hadoop.hive.ql.exec.StatsTask" level="info" additivity="true">
  49.             <appender-ref ref="SendPackage"/>
  50.         </logger>
  51.         <logger name="org.springframework.boot.diagnostics.LoggingFailureAnalysisReporter" level="error" additivity="true">
  52.             <appender-ref ref="stderr"/>
  53.         </logger>
  54.         <logger name="com.netflix.discovery" level="warn" additivity="true">
  55.             <appender-ref ref="Send"/>
  56.         </logger>
  57.         <logger name="org.apache.hadoop.yarn" level="warn" additivity="true">
  58.             <appender-ref ref="Send"/>
  59.         </logger>
  60.         <logger name="org.springframework" level="warn" additivity="true">
  61.             <appender-ref ref="Send"/>
  62.         </logger>
  63.         <logger name="org.apache.linkis.server.security" level="warn" additivity="true">
  64.             <appender-ref ref="Send"/>
  65.         </logger>
  66.         <logger name="org.apache.hadoop.hive.ql.exec.mr.ExecDriver" level="fatal" additivity="true">
  67.             <appender-ref ref="Send"/>
  68.         </logger>
  69.         <logger name="org.apache.hadoop.hdfs.KeyProviderCache" level="fatal" additivity="true">
  70.             <appender-ref ref="Send"/>
  71.         </logger>
  72.         <logger name="org.spark_project.jetty" level="ERROR" additivity="true">
  73.             <appender-ref ref="Send"/>
  74.         </logger>
  75.         <logger name="org.eclipse.jetty" level="ERROR" additivity="true">
  76.             <appender-ref ref="Send"/>
  77.         </logger>
  78.         <logger name="org.springframework" level="ERROR" additivity="true">
  79.             <appender-ref ref="Send"/>
  80.         </logger>
  81.         <logger name="org.reflections.Reflections" level="ERROR" additivity="true">
  82.             <appender-ref ref="Send"/>
  83.         </logger>
  85.         <logger name="org.apache.hadoop.ipc.Client" level="ERROR" additivity="true">
  86.             <appender-ref ref="Send"/>
  87.         </logger>
  89.    </loggers>
  90. </configuration>