Recommended Configuration
Linkis builds a layer of computing middleware between the upper application and the underlying engine. As an open source distributed computing middleware, it can be well deployed and run on Intel architecture servers and mainstream virtualization environments, and supports mainstream Linux operating system environments
OS | Version |
---|---|
Red Hat Enterprise Linux | 7.0 and above |
CentOS | 7.0 and above |
Oracle Enterprise Linux | 7.0 and above |
Ubuntu LTS | 16.04 and above |
Note: The above Linux operating systems can run on physical servers and mainstream virtualization environments such as VMware, KVM, and XEN
Linkis supports 64-bit general-purpose hardware server platforms running on the Intel x86-64 architecture. The following recommendations are made for the server hardware configuration of the production environment:
CPU | Memory | Disk type | Network | Number of instances |
---|---|---|---|---|
16 cores + | 32GB + | SAS | Gigabit network card | 1+ |
Note:
- The above recommended configuration is the minimum configuration for deploying Linkis, and a higher configuration is strongly recommended for production environments
- The hard disk size configuration is recommended to be 50GB+, and the system disk and data disk are separated
Linkis binary packages are compiled based on the following software versions:
Component | Version | Description |
---|---|---|
Hadoop | 3.3.4 | |
Hive | 3.1.3 | |
Spark | 3.2.1 | |
Flink | 1.12.2 | |
openLooKeng | 1.5.0 | |
Sqoop | 1.4.6 | |
ElasticSearch | 7.6.2 | |
Presto | 0.234 | |
Python | Python2 |
Note: If the locally installed component version is inconsistent with the above, you need to modify the corresponding component version and compile the binary package yourself for installation.
Linkis recommends Chrome version 73 for front-end access
The development process requires a password-free interface, which can be replaced or appended to linkis.properties
parameter name | default value | description |
---|---|---|
wds.linkis.test.mode | false | Whether to enable debugging mode, if set to true, all microservices support password-free login, and all EngineConn open remote debugging ports |
wds.linkis.test.user | hadoop | When wds.linkis.test.mode=true, the default login user for password-free login |
Apache Linkis uses configuration files to manage admin users by default, and this configuration can be replaced or appended to linkis-mg-gateway.properties
. For multi-user access LDAP implementation.
parameter name | default value | description |
---|---|---|
wds.linkis.admin.user | hadoop | admin username |
wds.linkis.admin.password | 123456 | Admin user password |
Apache Linkis can access LDAP through parameters to achieve multi-user management, and this configuration can be replaced or added in linkis-mg-gateway.properties
.
parameter name | default value | description |
---|---|---|
wds.linkis.ldap.proxy.url | None | LDAP URL address |
wds.linkis.ldap.proxy.baseDN | None | LDAP baseDN address |
wds.linkis.ldap.proxy.userNameFormat | None |
Apache Linkis sometimes debugs exceptions when submitting tasks, such as: insufficient resources; you can replace or append this configuration in linkis-cg-linkismanager.properties
.
parameter name | default value | description |
---|---|---|
wds.linkis.manager.rm.request.enable | true | resource check |
Apache Linkis EC can enable debugging mode, and this configuration can be replaced or added in linkis-cg-linkismanager.properties
.
parameter name | default value | description |
---|---|---|
wds.linkis.engineconn.debug.enable | true | Whether to enable engine debugging |
The public-service service of Apache Linkis needs to read hive metadata; this configuration can be replaced or appended in linkis-ps-publicservice.properties
.
parameter name | default value | description |
---|---|---|
hive.meta.url | None | The URL of the HiveMetaStore database. |
hive.meta.user | none | user of the HiveMetaStore database |
hive.meta.password | None | password for the HiveMetaStore database |
Apache Linkis access uses Mysql as data storage by default, you can replace or append this configuration in linkis.properties
.
parameter name | default value | description |
---|---|---|
wds.linkis.server.mybatis.datasource.url | None | Database connection string, for example: jdbc:mysql://127.0.0.1:3306/dss?characterEncoding=UTF-8 |
wds.linkis.server.mybatis.datasource.username | None | Database user name, for example: root |
wds.linkis.server.mybatis.datasource.password | None | Database password, for example: root |
Apache Linkis supports using redis for session sharing; this configuration can be replaced or appended in linkis.properties
.
parameter name | default value | description |
---|---|---|
linkis.session.redis.cache.enabled | None | Whether to enable |
linkis.session.redis.host | 127.0.0.1 | hostname |
linkis.session.redis.port | 6379 | Port, eg |
linkis.session.redis.password | None | password |
When developing Apache Linkis, you can use this parameter to customize the database, Rest interface, and entity objects of the loading module; you can modify it in linkis-ps-publicservice.properties
, and use commas to separate multiple modules.
parameter name | default value | description |
---|---|---|
wds.linkis.server.restful.scan.packages | None | restful scan packages, for example: org.apache.linkis.basedatamanager.server.restful |
wds.linkis.server.mybatis.mapperLocations | None | Mybatis mapper file path, for example: classpath:org/apache/linkis/basedatamanager/server/dao/mapper/.xml |
wds.linkis.server.mybatis.typeAliasesPackage | None | Entity alias scanning package, for example: org.apache.linkis.basedatamanager.server.domain |
wds.linkis.server.mybatis.BasePackage | None | Database dao layer scan, for example: org.apache.linkis.basedatamanager.server.dao |
This parameter can be used to customize the route of loading modules during Apache Linkis development; it can be modified in linkis.properties
, and commas are used to separate multiple modules.
parameter name | default value | description |
---|---|---|
wds.linkis.gateway.conf.publicservice.list | cs,contextservice,data-source-manager,metadataQuery,metadatamanager,query,jobhistory,application,configuration,filesystem,udf,variable,microservice,errorcode,bml,datasource,basedata -manager | publicservice services support routing modules |
This parameter can be used to customize the route of loading modules during Apache Linkis development; it can be modified in linkis.properties
, and commas are used to separate multiple modules.
parameter name | default value | description |
---|---|---|
wds.linkis.filesystem.root.path | file:///tmp/linkis/ | Local user directory, a folder named after the user name needs to be created under this directory |
wds.linkis.filesystem.hdfs.root.path | hdfs:///tmp/ | HDFS user directory |
wds.linkis.bml.is.hdfs | true | Whether to enable hdfs |
wds.linkis.bml.hdfs.prefix | /apps-data | hdfs path |
wds.linkis.bml.local.prefix | /apps-data | local path |