Version upgrades above 1.0.3

This article briefly introduces the general upgrade process for versions above 1.0.3

1 Upgrade Instructions

  • If you are installing and using Linkis for the first time, or reinstalling Linkis, you don’t need to pay attention to the Linkis upgrade guide.
  • If components of other platforms (DataSphereStudio/Qualitis/Visualis, etc.) are involved, please confirm the compatibility between versions before upgrading, and it is best to use the recommended version.
  • It is recommended to control the version through the soft chain. You can switch the version by modifying the target address of the soft chain. For example: linkis -> /appcom/Install/LinkisInstall/linkis-1.1.3.20220615210213
  • The upgrade process mainly requires attention to the adjustment of the database table structure and the adjustment of some configuration parameters
  • In addition to the upgrade of the back-end services, the management console resources of linkis also need to be upgraded together
  • The main changes of each version can be found in the overview information of the version https://linkis.apache.org/zh-CN/docs/x.x.x/release and the version [release-note](https://linkis.apache .org/zh-CN/download/main): https://linkis.apache.org/zh-CN/download/main
  • The database changes/configuration parameter changes of each version are based on the previous version

2 Service upgrade installation

Press Deployment guide document (the installation of the management console in the document can be skipped) to install the new version.

When installing the service, if the historical data is retained, please retain the historical data, if you do not need to retain the data, just reinstall it directly

  1. Do you want to clear Linkis table information in the database?
  2.  1: Do not execute table-building statements
  3.  2: Dangerous! Clear all data and rebuild the tables
  4.  other: exit
  7. Please input the choice: ## choice 1

3. Database table upgrade

After the service installation is completed, the database structure needs to be modified, including table structure changes and table data changes

Find the corresponding version db/upgrade/x.x.x-schema/ sql change file

If it is executed across multiple versions, please execute them in the order of versions, If some versions do not have x.x.x_schema (after the linkis>=1.1.0 version, as long as the adjustment of the database table is involved, there will be a corresponding version of the schema file), it means that there is no change in the data table in this version

Version upgrades above 1.0.3 - 图1notice

Execute upgrade1.1.1 under DDL upgrade script in schema, such as if it is executed to linkis cg rm resource action Record related error, please check whether the statement creating the table is missing engine=innodb default charset=utf8; If it is missing, please add it before execution

  1. ├── linkis_ddl.sql # The current version of the full ddl database definition language table building statement
  2. ├── linkis_dml.sql # The current version of the full dml data manipulation language data additions and changes
  3. └── upgrade
  4.     ├── 1.1.0_schema # The corresponding version of sql
  5.     │ └── mysql
  6.     │ ├── linkis_ddl.sql
  7.     │ └── linkis_dml.sql
  8.     ├── 1.1.1_schema
  9.     │ └── mysql
  10.     │ ├── linkis_ddl.sql
  11.     │ └── linkis_dml.sql
  12.     └── 1.1.3_schema
  13.         └── mysql
  14.             └── linkis_ddl.sql
  1.     #If it is executed across multiple versions, please execute in order of versions, execute ddl first and then execute ddl
  2.     #For example, currently upgrade from linkis-1.0.3 to linkis-1.1.2
  3.     source upgrade/1.1.0_schema/mysql/linkis_ddl.sql
  4.     source upgrade/1.1.0_schema/mysql/linkis_dml.sql
  7.     source upgrade/1.1.1_schema/mysql/linkis_ddl.sql
  8.     source upgrade/1.1.1_schema/mysql/linkis_dml.sql

4 Adjustment of data (optional)

According to the actual situation, decide whether to make the following adjustments

4.1 TOKEN configuration

Authentication for interface calls yes 1.1.1 Version adjustment Migrate the original TOKEN configuration from ${LINKIS_HOME}/conf/token.properties to the database table linkis_mg_gateway_auth_token, For the TOKEN originally configured in token.properties, you need to manually migrate the table

4.2 UDF Adjustment

1.1.1 supports the functions of UDF multi-version control and UDF storage to BML, the table structure stored by UDF functions has been adjusted, and the historical data of UDF needs to be migrated separately

4.3 Session field key configuration

In version 1.1.1, the key of the session field was adjusted from bdp-user-ticket-id-> to linkis_user_session_ticket_id_v1, If it is an upgrade to Linkis. At the same time, DSS or other projects are deployed, but in their service lib package, the linkis-module-x.x.x.jar package of Linkis that they depend on is <1.1.1, you need to modify the file located in ${LINKIS_HOME}/conf/linkis.properties

  1. echo "wds.linkis.session.ticket.key=bdp-user-ticket-id" >> linkis.properties

4.4 Adjustment of default queue

1.1.1 Begin to adjust the default cluster name of yarn queue wds.linkis.rm.cluster sit is adjusted to default, if you have been using sit, please modify the table data Cluster name in linkis_cg_rm_external_resource_provider

If there is a third-party appconn plugin installed in ${LINKIS_HOME_OLD}/lib/linkis-engineconn-plugins in the previous version of Linkis, you need to copy it to the new version, It is best to link to the appconn path through a soft chain like:

  1. #Check if the old version has a third-party appconn installed
  2. cd ${LINKIS_HOME_OLD}/lib/linkis-engineconn-plugins/
  3. ll
  4.  appconn -> /appcom/Install/LinkisInstall/appconn
  9. #The new version is consistent
  10. cd ${LINKIS_HOME}/lib/linkis-engineconn-plugins/
  11. #soft chain
  12. ln -snf /appcom/Install/LinkisInstall/appconn appconn

If dss is installed in the original version, you need to copy the dss-gateway-support-x.x.x.jar in the original package to the ./lib/linkis-spring-cloud-services/linkis-mg-gateway/ directory of linkis like:

  1. cp ${LINKIS_HOME_OLD}/lib/linkis-spring-cloud-services/linkis-mg-gateway/dss-gateway-support-1.1.3.jar ${LINKIS_HOME}/lib/linkis-spring-cloud-services/linkis- mg-gateway/dss-gateway-support-1.1.3.jar

5 Updates to the console

To upgrade the backend, you also need to upgrade the corresponding management console resources. You don’t need to install the management console, just replace the static resources directly.

5.1 Download the front-end installation package and unzip it

Upload it to the server where the management console is located, and decompress it

  1. tar -xvf apache-linkis-x.x.x-incubating-web-bin.tar.gz

5.2 Upgrade

There are many ways to upgrade the management console, because it is only the update of resources, which can be done through the installation script of the management console, or it can directly overwrite the original resources. It is recommended to use the soft chain method, just modify the target address of the soft chain. The following takes the new version resource path method as an example

The nginx configuration file is by default in /etc/nginx/conf.d/* nginx log files are in /var/log/nginx/access.log and /var/log/nginx/error.log

Scenario 1: If it is integrated into the DSS project, modify the address of the linkis management console resource in the DSS project to point to The nginx configuration file for dss is by default in /etc/nginx/conf.d/dss.conf

  1. #Example
  2.         server {
  3.             ......
  4.             location dss/linkis {
  5.             alias /appcom/Install/linkis-web-newversion/dist; # static file directory
  6.             index index.html index.html;
  7.             }
  8.             .....
  9.         }

Scenario 2: If linkis is deployed independently Modify the configuration of Nginx, the static resource address points to the new version of the linkis console Linkis’ nginx configuration file is by default in /etc/nginx/conf.d/dss.conf

  1. #Example
  2.         server {
  3.             ......
  4.             location dss/linkis {
  5.             alias /appcom/Install/linkis-web-newversion/dist; # static file directory
  6.             index index.html index.html;
  7.             }
  8.             ......
  9.         }

Reload nginx configuration

  1. sudo nginx -s reload

5.3 Notes

  • After the management console is upgraded, because the browser may have a cache, if you want to verify the effect, it is best to clear the browser cache