Deployment

Set up flink environment

Currently inlong-sort is based on flink, before you run an inlong-sort application, you need to set up flink environment.

Currently, inlong-sort relys on flink-1.13.5. Chose flink-1.13.5-bin-scala_2.11.tgz when downloading package.

Once your flink environment is set up, you can visit web ui of flink, whose address is stored in /${your_flink_path}/conf/masters.

Prepare installation files

All installation files at inlong-sort directory.

Starting an inlong-sort application

Now you can submit job to flink with the jar compiled, refer to how to submit job to flink.

Example：

./bin/flink run -c org.apache.inlong.sort.flink.Entrance inlong-sort/sort-core-[version].jar \
--cluster-id inlong_app --zookeeper.quorum 127.0.0.1:2181 --zookeeper.path.root /inlong_sort \
--source.type tubemq --metrics.audit.proxy.hosts 127.0.0.1:10081 --sink.type hive

Notice：

• -c org.apache.inlong.sort.flink.Entrance is the main class name

• inlong-sort/sort-core-[version].jar is the compiled jar

Necessary configurations

• --cluster-id represent a specified inlong-sort application, same as the configuration of sort.appName in inlong-manager
• --zookeeper.quorum zk quorum, same as the configuration of cluster.zk.url in inlong-manager
• --zookeeper.path.root zk root path, same as the configuration of cluster.zk.root in inlong-manager
• --source.type source of the application, currently “tubemq” and “pulsar” are supported
• --metrics.audit.proxy.hosts audit proxy host address for reporting audit metrics
• --sink.type sink of the application, currently “clickhouse” and “hive” are supported

Example

--cluster-id inlong_app --zookeeper.quorum 192.127.0.1:2181 \
--zookeeper.path.root /inlong_sort --source.type tubemq --sink.type hive

All configurations