MySQL

Overview

The MySQL Load Node supports to write data into MySQL database. This document describes how to set up the MySQL Load Node to run SQL queries against MySQL database.

Supported Version

Load NodeDriverGroup IdArtifact IdJAR
MySQLMySQLmysqlmysql-connector-javaDownload

Dependencies

In order to set up the MySQL Load Node, the following provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with Sort Connectors JAR bundles.

Maven dependency

  1. <dependency>
  2. <groupId>org.apache.inlong</groupId>
  3. <artifactId>sort-connector-jdbc</artifactId>
  4. <version>1.3.0-SNAPSHOT</version>
  5. </dependency>

MySQL license is conflict with Inlong license. So We remove MySQL driver in pom.xml. User can modify pom.xml before maven packaging if User need use it.

How to create a MySQL Load Node

Usage for SQL API

  1. -- MySQL extract node
  2. CREATE TABLE `mysql_extract_table`(
  3. PRIMARY KEY (`id`) NOT ENFORCED,
  4. `id` BIGINT,
  5. `name` STRING,
  6. `age` INT
  7. ) WITH (
  8. 'connector' = 'mysql-cdc-inlong',
  9. 'url' = 'jdbc:mysql://localhost:3306/read',
  10. 'username' = 'inlong',
  11. 'password' = 'inlong',
  12. 'table-name' = 'user'
  13. )
  14. -- MySQL load node
  15. CREATE TABLE `mysql_load_table`(
  16. PRIMARY KEY (`id`) NOT ENFORCED,
  17. `id` BIGINT,
  18. `name` STRING,
  19. `age` INT
  20. ) WITH (
  21. 'connector' = 'jdbc-inlong',
  22. 'url' = 'jdbc:mysql://localhost:3306/write',
  23. 'username' = 'inlong',
  24. 'password' = 'inlong',
  25. 'table-name' = 'user'
  26. )
  27. -- write data into mysql
  28. INSERT INTO mysql_load_table
  29. SELECT id, name , age FROM mysql_extract_table;

Usage for InLong Dashboard

TODO: It will be supported in the future.

Usage for InLong Manager Client

TODO: It will be supported in the future.

MySQL Load Node Options

OptionRequiredDefaultTypeDescription
connectorrequired(none)StringSpecify what connector to use, here should be ‘jdbc-inlong’.
urlrequired(none)StringThe JDBC database url.
table-namerequired(none)StringThe name of JDBC table to connect.
driveroptional(none)StringThe class name of the JDBC driver to use to connect to this URL, if not set, it will automatically be derived from the URL.
usernameoptional(none)StringThe JDBC user name. ‘username’ and ‘password’ must both be specified if any of them is specified.
passwordoptional(none)StringThe JDBC password.
connection.max-retry-timeoutoptional60sDurationMaximum timeout between retries. The timeout should be in second granularity and shouldn’t be smaller than 1 second.
sink.buffer-flush.max-rowsoptional100IntegerThe max size of buffered records before flush. Can be set to zero to disable it.
sink.buffer-flush.intervaloptional1sDurationThe flush interval mills, over this time, asynchronous threads will flush data. Can be set to ‘0’ to disable it. Note, ‘sink.buffer-flush.max-rows’ can be set to ‘0’ with the flush interval set allowing for complete async processing of buffered actions.
sink.max-retriesoptional3IntegerThe max retry times if writing records to database failed.
sink.parallelismoptional(none)IntegerDefines the parallelism of the JDBC sink operator. By default, the parallelism is determined by the framework using the same parallelism of the upstream chained operator.
inlong.metricoptional(none)StringInlong metric label, format of value is groupId&streamId&nodeId.

Data Type Mapping

MySQL typeFlink SQL type
TINYINTTINYINT
SMALLINT
TINYINT UNSIGNED
SMALLINT
INT
MEDIUMINT
SMALLINT UNSIGNED
INT
BIGINT
INT UNSIGNED
BIGINT
BIGINT UNSIGNEDDECIMAL(20, 0)
FLOATFLOAT
DOUBLE
DOUBLE PRECISION
DOUBLE
NUMERIC(p, s)
DECIMAL(p, s)
DECIMAL(p, s)
BOOLEAN
TINYINT(1)
BOOLEAN
DATEDATE
TIME [(p)]TIME [(p)][WITHOUT TIMEZONE]
DATETIME [(p)]TIMESTAMP [(p)][WITHOUT TIMEZONE]
CHAR(n)
VARCHAR(n)
TEXT
STRING
BINARY
VARBINARY
BLOB
BYTES
ARRAY