Version Adaptation

Explain where manual modification is required for Apache, CDH, HDP and other version adaptations

Enter the root directory of the project and execute the following commands in sequence

  1. mvn -N  install
  2. mvn clean install -Dmaven.test.skip=true

linkis-dist -> package -> linkis-dml.sql(db folder)

Switch the corresponding engine version to the version you need. If the version you use is consistent with the official version, you do not need to modify this step

for example:

  1. If Spark is 3.0.0, this is SET @ SPARK_ LABEL=”spark-3.0.0”;
  2. If hive is 2.1.1-cdh6.3.2, adjust 2.1.1 first Cdh6.3.2 (during construction), this is SET @ HIVE LABEL=”hive-2.1.1_cdh6.3.2”;
  1. -- variable：
  2. SET @SPARK_LABEL="spark-2.4.3";
  3. SET @HIVE_LABEL="hive-2.3.3";
  4. SET @PYTHON_LABEL="python-python2";
  5. SET @PIPELINE_LABEL="pipeline-1";
  6. SET @JDBC_LABEL="jdbc-4";
  7. SET @PRESTO_LABEL="presto-0.234";
  8. SET @IO_FILE_LABEL="io_file-1.0";
  9. SET @OPENLOOKENG_LABEL="openlookeng-1.5.0";
engineversion
hadoop2.7.2
hive2.3.3
spark2.4.3
flink1.12.2
engineversion
hadoop3.1.1
hive3.1.2
spark3.0.1
flink1.13.2

For Linkis version < 1.3.2

  1. <hadoop.version>3.1.1</hadoop.version>
  2. <scala.version>2.12.10</scala.version>
  3. <scala.binary.version>2.12</scala.binary.version>
  5. <!-- hadoop-hdfs replace with hadoop-hdfs-client -->
  6. <dependency>
  7.     <groupId>org.apache.hadoop</groupId>
  8.     <artifactId>hadoop-hdfs-client</artifactId>
  9.     <version>${hadoop.version}</version>
  10. <dependency>

For Linkis version >= 1.3.2, we only need to set scala.version and scala.binary.version if necessary

  1. <scala.version>2.12.10</scala.version>
  2. <scala.binary.version>2.12</scala.binary.version>

Because we can directly compile with hadoop-3.3 or hadoop-2.7 profile. Profile hadoop-3.3 can be used for any hadoop3.x, default hadoop3.x version will be hadoop 3.3.1, Profile hadoop-2.7 can be used for any hadoop2.x, default hadoop2.x version will be hadoop 2.7.2, other hadoop version can be specified by -Dhadoop.version=xxx

  1. mvn -N  install
  2. mvn clean install -Phadoop-3.3 -Dmaven.test.skip=true
  3. mvn clean install -Phadoop-3.3 -Dhadoop.version=3.1.1 -Dmaven.test.skip=true

For Linkis version < 1.3.2

  1. <!-- Notice here <version>${hadoop.version}</version> , adjust according to whether you have encountered errors --> 
  2. <dependency>
  3.     <groupId>org.apache.hadoop</groupId>
  4.     <artifactId>hadoop-hdfs-client</artifactId>
  5.     <version>${hadoop.version}</version>
  6. </dependency>

For Linkis version >= 1.3.2,linkis-hadoop-common module no need to change

  1. <hive.version>3.1.2</hive.version>

For Linkis version < 1.3.2

  1. <spark.version>3.0.1</spark.version>

For Linkis version >= 1.3.2

  1. We can directly compile with spark-3.2 or spark-2.4-hadoop-3.3 profile, if we need to used with hadoop3, then profile hadoop-3.3 will be needed.
  2. default spark3.x version will be spark 3.2.1. if we compile with spark-3.2 then scala version will be 2.12.15 by default,
  3. so we do not need to set the scala version in Linkis project pom file(mentioned in 5.1.1).
  4. if spark2.x used with hadoop3, for compatibility reason, profile `spark-2.4-hadoop-3.3` need to be activated.
  1. mvn -N  install
  2. mvn clean install -Pspark-3.2 -Phadoop-3.3 -Dmaven.test.skip=true
  3. mvn clean install -Pspark-2.4-hadoop-3.3 -Phadoop-3.3 -Dmaven.test.skip=true
  1. <flink.version>1.13.2</flink.version>

Since some classes of Flink 1.12.2 to 1.13.2 are adjusted, it is necessary to compile and adjust Flink. Select Scala version 2.12 for compiling Flink

Version Adaptation - 图1temporary plan

Note that the following operations are all in flink

Due to flink1.12.2 to 1.13.2, some classes are adjusted, so flink needs to be compiled and adjusted, and the version of scala selected for compiling flink is version 2.12(The scala version is based on the actual version used)

flink compilation reference instruction: mvn clean install -DskipTests -P scala-2.12 -Dfast -T 4 -Dmaven.compile.fock=true

  1. -- Note that the following classes are copied from version 1.12.2 to version 1.13.2
  2. org.apache.flink.table.client.config.entries.DeploymentEntry
  3. org.apache.flink.table.client.config.entries.ExecutionEntry
  4. org.apache.flink.table.client.gateway.local.CollectBatchTableSink
  5. org.apache.flink.table.client.gateway.local.CollectStreamTableSink

org.apache.linkis.manager.label.conf.LabelCommonConfig file adjustment

  1.     public static final CommonVars<String> SPARK_ENGINE_VERSION =
  2.             CommonVars.apply("wds.linkis.spark.engine.version", "3.0.1");
  4.     public static final CommonVars<String> HIVE_ENGINE_VERSION =
  5.             CommonVars.apply("wds.linkis.hive.engine.version", "3.1.2");

org.apache.linkis.governance.common.conf.GovernanceCommonConf file adjustment

  1.   val SPARK_ENGINE_VERSION = CommonVars("wds.linkis.spark.engine.version", "3.0.1")
  3.   val HIVE_ENGINE_VERSION = CommonVars("wds.linkis.hive.engine.version", "3.1.2")
engineversion
hadoop3.1.1
hive3.1.0
spark2.3.2
json4s.version3.2.11

For Linkis version < 1.3.2

  1. <hadoop.version>3.1.1</hadoop.version>
  2. <json4s.version>3.2.11</json4s.version>
  4. <!-- hadoop-hdfs replace with hadoop-hdfs-client -->
  5. <dependency>
  6.     <groupId>org.apache.hadoop</groupId>
  7.     <artifactId>hadoop-hdfs-client</artifactId>
  8.     <version>${hadoop.version}</version>
  9. <dependency>

For Linkis version >= 1.3.2, we only need to set json4s.version if necessary

  1. <json4s.version>3.2.11</json4s.version>

Because we can directly compile with hadoop-3.3 or hadoop-2.7 profile. Profile hadoop-3.3 can be used for any hadoop3.x, default hadoop3.x version will be hadoop 3.3.1, Profile hadoop-2.7 can be used for any hadoop2.x, default hadoop2.x version will be hadoop 2.7.2, other hadoop version can be specified by -Dhadoop.version=xxx

  1. mvn -N  install
  2. mvn clean install -Phadoop-3.3 -Dmaven.test.skip=true
  3. mvn clean install -Phadoop-3.3 -Dhadoop.version=3.1.1 -Dmaven.test.skip=true
  1. <hive.version>3.1.0</hive.version>

For Linkis version < 1.3.2

  1. <spark.version>2.3.2</spark.version>

For Linkis version >= 1.3.2

  1. We can directly compile with spark-3.2 profile, if we need to use with hadoop3, then profile hadoop-3.3 will be needed.
  2. default spark3.x version will be spark 3.2.1. if we compile with spark-3.2 then scala version will be 2.12.15 by default,
  3. so we do not need to set the scala version in Linkis project pom file(mentioned in 5.1.1).
  4. if spark2.x used with hadoop3, for compatibility reason, profile `spark-2.4-hadoop-3.3` need to be activated.
  1. mvn -N  install
  2. mvn clean install -Pspark-3.2 -Phadoop-3.3 -Dmaven.test.skip=true
  3. mvn clean install -Pspark-2.4-hadoop-3.3 -Phadoop-3.3 -Dmaven.test.skip=true

org.apache.linkis.manager.label.conf.LabelCommonConfig file adjustment

  1.     public static final CommonVars<String> SPARK_ENGINE_VERSION =
  2.             CommonVars.apply("wds.linkis.spark.engine.version", "2.3.2");
  4.     public static final CommonVars<String> HIVE_ENGINE_VERSION =
  5.             CommonVars.apply("wds.linkis.hive.engine.version", "3.1.0");

org.apache.linkis.governance.common.conf.GovernanceCommonConf file adjustment

  1.   val SPARK_ENGINE_VERSION = CommonVars("wds.linkis.spark.engine.version", "2.3.2")
  3.   val HIVE_ENGINE_VERSION = CommonVars("wds.linkis.hive.engine.version", "3.1.0")
  1. <mirrors>
  2.   <!-- mirror
  3.    | Specifies a repository mirror site to use instead of a given repository. The repository that
  4.    | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
  5.    | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
  6.    |
  7.   <mirror>
  8.     <id>mirrorId</id>
  9.     <mirrorOf>repositoryId</mirrorOf>
  10.     <name>Human Readable Name for this Mirror.</name>
  11.     <url>http://my.repository.com/repo/path</url>
  12.   </mirror>
  13.    -->
  14.    <mirror>
  15.        <id>nexus-aliyun</id>
  16.        <mirrorOf>*,!cloudera</mirrorOf>
  17.        <name>Nexus aliyun</name>
  18.        <url>http://maven.aliyun.com/nexus/content/groups/public</url>
  19.    </mirror>
  20.    <mirror>
  21.        <id>aliyunmaven</id>
  22.        <mirrorOf>*,!cloudera</mirrorOf>
  23.        <name>Alibaba Cloud Public Warehouse</name>
  24.        <url>https://maven.aliyun.com/repository/public</url>
  25.    </mirror>
  26.    <mirror>
  27.        <id>aliyunmaven</id>
  28.        <mirrorOf>*,!cloudera</mirrorOf>
  29.        <name>spring-plugin</name>
  30.        <url>https://maven.aliyun.com/repository/spring-plugin</url>
  31.    </mirror>
  32.   <mirror>
  33.     <id>maven-default-http-blocker</id>
  34.     <mirrorOf>external:http:*</mirrorOf>
  35.     <name>Pseudo repository to mirror external repositories initially using HTTP.</name>
  36.     <url>http://0.0.0.0/</url>
  37.     <blocked>true</blocked>
  38.   </mirror>
  39. </mirrors>
  1.     <repositories>
  2.         <repository>
  3.             <id>cloudera</id>
  4.             <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
  5.             <releases>
  6.                 <enabled>true</enabled>
  7.             </releases>
  8.         </repository>
  9.         <!--To prevent cloudera from not being found, add Alibaba Source-->
  10.         <repository>
  11.             <id>aliyun</id>
  12.             <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
  13.             <releases>
  14.                 <enabled>true</enabled>
  15.             </releases>
  16.         </repository>
  17.     </repositories>
engineversion
hadoop2.6.0-cdh5.12.1
zookeeper3.4.5-cdh5.12.1
hive1.1.0-cdh5.12.1
spark2.3.4
flink1.12.4
pythonpython3
  1. <hadoop.version>2.6.0-cdh5.12.1</hadoop.version>
  2. <zookeeper.version>3.4.5-cdh5.12.1</zookeeper.version>
  3. <scala.version>2.11.8</scala.version>
  1. -- update
  2. <hive.version>1.1.0-cdh5.12.1</hive.version>
  3. -- add
  4. <package.hive.version>1.1.0_cdh5.12.1</package.hive.version>
  • update assembly under distribution.xml file
  1. <outputDirectory>/dist/v${package.hive.version}/lib</outputDirectory>
  2. <outputDirectory>dist/v${package.hive.version}/conf</outputDirectory>
  3. <outputDirectory>plugin/${package.hive.version}</outputDirectory>

  • update CustomerDelimitedJSONSerDe file

    1.    /* hive version is too low and needs to be noted
    2.    case INTERVAL_YEAR_MONTH:
    3.        {
    4.            wc = ((HiveIntervalYearMonthObjectInspector) oi).getPrimitiveWritableObject(o);
    5.            binaryData = Base64.encodeBase64(String.valueOf(wc).getBytes());
    6.            break;
    7.        }
    8.    case INTERVAL_DAY_TIME:
    9.        {
    10.            wc = ((HiveIntervalDayTimeObjectInspector) oi).getPrimitiveWritableObject(o);
    11.            binaryData = Base64.encodeBase64(String.valueOf(wc).getBytes());
    12.            break;
    13.        }
    14.    */
  1. <flink.version>1.12.4</flink.version>
  1. <spark.version>2.3.4</spark.version>
  1. <python.version>python3</python.version>

org.apache.linkis.manager.label.conf.LabelCommonConfig file adjustment

  1.    public static final CommonVars<String> SPARK_ENGINE_VERSION =
  2.             CommonVars.apply("wds.linkis.spark.engine.version", "2.3.4");
  4.     public static final CommonVars<String> HIVE_ENGINE_VERSION =
  5.             CommonVars.apply("wds.linkis.hive.engine.version", "1.1.0");
  7.             CommonVars.apply("wds.linkis.python.engine.version", "python3")

org.apache.linkis.governance.common.conf.GovernanceCommonConf file adjustment

  1.   val SPARK_ENGINE_VERSION = CommonVars("wds.linkis.spark.engine.version", "2.3.4")
  3.   val HIVE_ENGINE_VERSION = CommonVars("wds.linkis.hive.engine.version", "1.1.0")
  5.   val PYTHON_ENGINE_VERSION = CommonVars("wds.linkis.python.engine.version", "python3")
engineversion
hadoop3.0.0-cdh6.3.2
hive2.1.1-cdh6.3.2
spark3.0.0
  1. <hadoop.version>3.0.0-cdh6.3.2</hadoop.version> 
  2. <scala.version>2.12.10</scala.version>
  1.    <!-- hadoop-hdfs replace with hadoop-hdfs-client --> 
  2.    <dependency>
  3.       <groupId>org.apache.hadoop</groupId>
  4.       <artifactId>hadoop-hdfs-client</artifactId>
  5.     </dependency>
  1. -- update
  2. <hive.version>2.1.1-cdh6.3.2</hive.version>
  3. -- add
  4. <package.hive.version>2.1.1_cdh6.3.2</package.hive.version>

update assembly under distribution.xml file

  1. <outputDirectory>/dist/v${package.hive.version}/lib</outputDirectory>
  2. <outputDirectory>dist/v${package.hive.version}/conf</outputDirectory>
  3. <outputDirectory>plugin/${package.hive.version}</outputDirectory>
  1. <spark.version>3.0.0</spark.version>

org.apache.linkis.manager.label.conf.LabelCommonConfig file adjustment

  1.    public static final CommonVars<String> SPARK_ENGINE_VERSION =
  2.             CommonVars.apply("wds.linkis.spark.engine.version", "3.0.0");
  4.     public static final CommonVars<String> HIVE_ENGINE_VERSION =
  5.             CommonVars.apply("wds.linkis.hive.engine.version", "2.1.1_cdh6.3.2");

org.apache.linkis.governance.common.conf.GovernanceCommonConf file adjustment

  1.   val SPARK_ENGINE_VERSION = CommonVars("wds.linkis.spark.engine.version", "3.0.0")
  3.   val HIVE_ENGINE_VERSION = CommonVars("wds.linkis.hive.engine.version", "2.1.1_cdh6.3.2")
  • If the class is missing or the method in the class is missing, find the corresponding package dependency, and how to try to switch to the version with the corresponding package or class
  • If the engine version needs to use -, use _ to replace, add<package.(engine name).version>to specify the replaced version, and use ${package.(engine name). version} in the corresponding engine distribution file to replace the original
  • If sometimes there is a 403 problem when using Alibaba Cloud images to download the jars of guava, you can switch to Huawei, Tencent and other image warehouses