Kafka CDC

Prepare Kafka Bundled Jar

  1. flink-sql-connector-kafka-*.jar

Supported Formats

Flink provides several Kafka CDC formats: Canal Json, Debezium Json, Debezium Avro, Ogg Json, Maxwell Json and Normal Json. If a message in a Kafka topic is a change event captured from another database using the Change Data Capture (CDC) tool, then you can use the Paimon Kafka CDC. Write the INSERT, UPDATE, DELETE messages parsed into the paimon table.

FormatsSupported
Canal CDCTrue
Debezium CDCTrue
Maxwell CDCTrue
OGG CDCTrue
JSONTrue

The JSON sources possibly missing some information. For example, Ogg and Maxwell format standards don’t contain field types; When you write JSON sources into Flink Kafka sink, it will only reserve data and row type and drop other information. The synchronization job will try best to handle the problem as follows:

  1. Usually, debezium-json contains ‘schema’ field, from which Paimon will retrieve data types. Make sure your debezium json has this field, or Paimon will use ‘STRING’ type.
  2. If missing field types, Paimon will use ‘STRING’ type as default.
  3. If missing database name or table name, you cannot do database synchronization, but you can still do table synchronization.
  4. If missing primary keys, the job might create non primary key table. You can set primary keys when submit job in table synchronization.

Synchronizing Tables

By using KafkaSyncTableAction in a Flink DataStream job or directly through flink run, users can synchronize one or multiple tables from Kafka’s one topic into one Paimon table.

To use this feature through flink run, run the following shell command.

  1. <FLINK_HOME>/bin/flink run \
  2.     /path/to/paimon-flink-action-0.9.0.jar \
  3.     kafka_sync_table
  4.     --warehouse <warehouse-path> \
  5.     --database <database-name> \
  6.     --table <table-name> \
  7.     [--partition_keys <partition_keys>] \
  8.     [--primary_keys <primary-keys>] \
  9.     [--type_mapping to-string] \
  10.     [--computed_column <'column-name=expr-name(args[, ...])'> [--computed_column ...]] \
  11.     [--kafka_conf <kafka-source-conf> [--kafka_conf <kafka-source-conf> ...]] \
  12.     [--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]] \
  13.     [--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
ConfigurationDescription
—warehouse
The path to Paimon warehouse.
—database
The database name in Paimon catalog.
—table
The Paimon table name.
—partition_keys
The partition keys for Paimon table. If there are multiple partition keys, connect them with comma, for example “dt,hh,mm”.
—primary_keys
The primary keys for Paimon table. If there are multiple primary keys, connect them with comma, for example “buyer_id,seller_id”.
—type_mapping
It is used to specify how to map MySQL data type to Paimon type.
Supported options:
  • “tinyint1-not-bool”: maps MySQL TINYINT(1) to TINYINT instead of BOOLEAN.
  • “to-nullable”: ignores all NOT NULL constraints (except for primary keys). This is used to solve the problem that Flink cannot accept the MySQL ‘ALTER TABLE ADD COLUMN column type NOT NULL DEFAULT x’ operation.
  • “to-string”: maps all MySQL types to STRING.
  • “char-to-string”: maps MySQL CHAR(length)/VARCHAR(length) types to STRING.
  • “longtext-to-bytes”: maps MySQL LONGTEXT types to BYTES.
  • “bigint-unsigned-to-bigint”: maps MySQL BIGINT UNSIGNED, BIGINT UNSIGNED ZEROFILL, SERIAL to BIGINT. You should ensure overflow won’t occur when using this option.
—computed_column
The definitions of computed columns. The argument field is from Kafka topic’s table field name. See here for a complete list of configurations.
—kafka_conf
The configuration for Flink Kafka sources. Each configuration should be specified in the format key=value. properties.bootstrap.servers, topic/topic-pattern, properties.group.id, and value.format are required configurations, others are optional.See its document for a complete list of configurations.
—catalog_conf
The configuration for Paimon catalog. Each configuration should be specified in the format “key=value”. See here for a complete list of catalog configurations.
—table_conf
The configuration for Paimon table sink. Each configuration should be specified in the format “key=value”. See here for a complete list of table configurations.

If the Paimon table you specify does not exist, this action will automatically create the table. Its schema will be derived from all specified Kafka topic’s tables,it gets the earliest non-DDL data parsing schema from topic. If the Paimon table already exists, its schema will be compared against the schema of all specified Kafka topic’s tables.

Example 1:

  1. <FLINK_HOME>/bin/flink run \
  2.     /path/to/paimon-flink-action-0.9.0.jar \
  3.     kafka_sync_table \
  4.     --warehouse hdfs:///path/to/warehouse \
  5.     --database test_db \
  6.     --table test_table \
  7.     --partition_keys pt \
  8.     --primary_keys pt,uid \
  9.     --computed_column '_year=year(age)' \
  10.     --kafka_conf properties.bootstrap.servers=127.0.0.1:9020 \
  11.     --kafka_conf topic=order \
  12.     --kafka_conf properties.group.id=123456 \
  13.     --kafka_conf value.format=canal-json \
  14.     --catalog_conf metastore=hive \
  15.     --catalog_conf uri=thrift://hive-metastore:9083 \
  16.     --table_conf bucket=4 \
  17.     --table_conf changelog-producer=input \
  18.     --table_conf sink.parallelism=4

If the kafka topic doesn’t contain message when you start the synchronization job, you must manually create the table before submitting the job. You can define the partition keys and primary keys only, and the left columns will be added by the synchronization job.

NOTE: In this case you shouldn’t use –partition_keys or –primary_keys, because those keys are defined when creating the table and can not be modified. Additionally, if you specified computed columns, you should also define all the argument columns used for computed columns.

Example 2: If you want to synchronize a table which has primary key ‘id INT’, and you want to compute a partition key ‘part=date_format(create_time,yyyy-MM-dd)’, you can create a such table first (the other columns can be omitted):

  1. CREATE TABLE test_db.test_table (
  2.     id INT,                 -- primary key
  3.     create_time TIMESTAMP,  -- the argument of computed column part
  4.     part STRING,            -- partition key
  5.     PRIMARY KEY (id, part) NOT ENFORCED
  6. ) PARTITIONED BY (part);

Then you can submit synchronization job:

  1. <FLINK_HOME>/bin/flink run \
  2.     /path/to/paimon-flink-action-0.9.0.jar \
  3.     kafka_sync_table \
  4.     --warehouse hdfs:///path/to/warehouse \
  5.     --database test_db \
  6.     --table test_table \
  7.     --computed_column 'part=date_format(create_time,yyyy-MM-dd)' \
  8.     ... (other conf)

Example 3: For some append data (such as log data), it can be treated as special CDC data with only INSERT operation type, so you can use ‘format=json’ to synchronize such data to the Paimon table.

  1. <FLINK_HOME>/bin/flink run \
  2.     /path/to/paimon-flink-action-0.9.0.jar \
  3.     kafka_sync_table \
  4.     --warehouse hdfs:///path/to/warehouse \
  5.     --database test_db \
  6.     --table test_table \
  7.     --partition_keys pt \
  8.     --computed_column 'pt=date_format(event_tm, yyyyMMdd)' \
  9.     --kafka_conf properties.bootstrap.servers=127.0.0.1:9020 \
  10.     --kafka_conf topic=test_log \
  11.     --kafka_conf properties.group.id=123456 \
  12.     --kafka_conf value.format=json \
  13.     --catalog_conf metastore=hive \
  14.     --catalog_conf uri=thrift://hive-metastore:9083 \
  15.     --table_conf sink.parallelism=4

Synchronizing Databases

By using KafkaSyncDatabaseAction in a Flink DataStream job or directly through flink run, users can synchronize the multi topic or one topic into one Paimon database.

To use this feature through flink run, run the following shell command.

  1. <FLINK_HOME>/bin/flink run \
  2.     /path/to/paimon-flink-action-0.9.0.jar \
  3.     kafka_sync_database
  4.     --warehouse <warehouse-path> \
  5.     --database <database-name> \
  6.     [--table_prefix <paimon-table-prefix>] \
  7.     [--table_suffix <paimon-table-suffix>] \
  8.     [--including_tables <table-name|name-regular-expr>] \
  9.     [--excluding_tables <table-name|name-regular-expr>] \
  10.     [--type_mapping to-string] \
  11.     [--partition_keys <partition_keys>] \
  12.     [--primary_keys <primary-keys>] \
  13.     [--kafka_conf <kafka-source-conf> [--kafka_conf <kafka-source-conf> ...]] \
  14.     [--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]] \
  15.     [--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
ConfigurationDescription
—warehouse
The path to Paimon warehouse.
—database
The database name in Paimon catalog.
—ignoreincompatible
It is default false, in this case, if MySQL table name exists in Paimon and their schema is incompatible,an exception will be thrown. You can specify it to true explicitly to ignore the incompatible tables and exception.
—table_prefix
The prefix of all Paimon tables to be synchronized. For example, if you want all synchronized tables to have “ods“ as prefix, you can specify “—tableprefix ods“.
—table_suffix
The suffix of all Paimon tables to be synchronized. The usage is same as “—table_prefix”.
—including_tables
It is used to specify which source tables are to be synchronized. You must use ‘|’ to separate multiple tables.Because ‘|’ is a special character, a comma is required, for example: ‘a|b|c’.Regular expression is supported, for example, specifying “—including_tables test|paimon.*” means to synchronize table ‘test’ and all tables start with ‘paimon’.
—excluding_tables
It is used to specify which source tables are not to be synchronized. The usage is same as “—including_tables”. “—excluding_tables” has higher priority than “—including_tables” if you specified both.
—type_mapping
It is used to specify how to map MySQL data type to Paimon type.
Supported options:
  • “tinyint1-not-bool”: maps MySQL TINYINT(1) to TINYINT instead of BOOLEAN.
  • “to-nullable”: ignores all NOT NULL constraints (except for primary keys). This is used to solve the problem that Flink cannot accept the MySQL ‘ALTER TABLE ADD COLUMN column type NOT NULL DEFAULT x’ operation.
  • “to-string”: maps all MySQL types to STRING.
  • “char-to-string”: maps MySQL CHAR(length)/VARCHAR(length) types to STRING.
  • “longtext-to-bytes”: maps MySQL LONGTEXT types to BYTES.
  • “bigint-unsigned-to-bigint”: maps MySQL BIGINT UNSIGNED, BIGINT UNSIGNED ZEROFILL, SERIAL to BIGINT. You should ensure overflow won’t occur when using this option.
—partition_keys
The partition keys for Paimon table. If there are multiple partition keys, connect them with comma, for example “dt,hh,mm”. If the keys are not in source table, the sink table won’t set partition keys.
—multiple_table_partition_keys
The partition keys for each different Paimon table. If there are multiple partition keys, connect them with comma, for example
  • —multiple_table_partition_keys tableName1=col1,col2.col3
  • —multiple_table_partition_keys tableName2=col4,col5.col6
  • —multiple_table_partition_keys tableName3=col7,col8.col9
    • If the keys are not in source table, the sink table won’t set partition keys.
    —primary_keys
    		The primary keys for Paimon table. If there are multiple primary keys, connect them with comma, for example “buyer_id,seller_id”. If the keys are not in source table, but the source table has primary keys, the sink table will use source table’s primary keys. Otherwise, the sink table won’t set primary keys.
    —kafka_conf
    		The configuration for Flink Kafka sources. Each configuration should be specified in the format key=value. properties.bootstrap.servers, topic/topic-pattern, properties.group.id, and value.format are required configurations, others are optional.See its document for a complete list of configurations.
    —catalog_conf
    		The configuration for Paimon catalog. Each configuration should be specified in the format “key=value”. See here for a complete list of catalog configurations.
    —table_conf
    		The configuration for Paimon table sink. Each configuration should be specified in the format “key=value”. See here for a complete list of table configurations.

    This action will build a single combined sink for all tables. For each Kafka topic’s table to be synchronized, if the corresponding Paimon table does not exist, this action will automatically create the table, and its schema will be derived from all specified Kafka topic’s tables. If the Paimon table already exists and its schema is different from that parsed from Kafka record, this action will try to preform schema evolution.

    Example

    Synchronization from one Kafka topic to Paimon database.

    1. <FLINK_HOME>/bin/flink run \
    2.     /path/to/paimon-flink-action-0.9.0.jar \
    3.     kafka_sync_database \
    4.     --warehouse hdfs:///path/to/warehouse \
    5.     --database test_db \
    6.     --kafka_conf properties.bootstrap.servers=127.0.0.1:9020 \
    7.     --kafka_conf topic=order \
    8.     --kafka_conf properties.group.id=123456 \
    9.     --kafka_conf value.format=canal-json \
    10.     --catalog_conf metastore=hive \
    11.     --catalog_conf uri=thrift://hive-metastore:9083 \
    12.     --table_conf bucket=4 \
    13.     --table_conf changelog-producer=input \
    14.     --table_conf sink.parallelism=4

    Synchronization from multiple Kafka topics to Paimon database.

    1. <FLINK_HOME>/bin/flink run \
    2.     /path/to/paimon-flink-action-0.9.0.jar \
    3.     kafka_sync_database \
    4.     --warehouse hdfs:///path/to/warehouse \
    5.     --database test_db \
    6.     --kafka_conf properties.bootstrap.servers=127.0.0.1:9020 \
    7.     --kafka_conf topic=order\;logistic_order\;user \
    8.     --kafka_conf properties.group.id=123456 \
    9.     --kafka_conf value.format=canal-json \
    10.     --catalog_conf metastore=hive \
    11.     --catalog_conf uri=thrift://hive-metastore:9083 \
    12.     --table_conf bucket=4 \
    13.     --table_conf changelog-producer=input \
    14.     --table_conf sink.parallelism=4

    Additional kafka_config

    There are some useful options to build Flink Kafka Source, but they are not provided by flink-kafka-connector document. They are:

    KeyDefaultTypeDescription
    schema.registry.url(none)StringWhen configuring “value.format=debezium-avro” which requires using the Confluence schema registry model for Apache Avro serialization, you need to provide the schema registry URL.