DataGen SQL Connector

Scan Source: Bounded Scan Source: UnBounded

The DataGen connector allows for reading by data generation rules.

The DataGen connector can work with Computed Column syntax. This allows you to generate records flexibly.

The DataGen connector is built-in.

Attention Complex types are not supported: Array, Map, Row. Please construct these types by computed column.

How to create a DataGen table

The boundedness of table: when the generation of field data in the table is completed, the reading is finished. So the boundedness of the table depends on the boundedness of fields.

For each field, there are two ways to generate data:

• Random generator is the default generator, you can specify random max and min values. For char/varchar/string, the length can be specified. It is a unbounded generator.
• Sequence generator, you can specify sequence start and end values. It is a bounded generator, when the sequence number reaches the end value, the reading ends.

CREATE TABLE datagen (
 f_sequence INT,
 f_random INT,
 f_random_str STRING,
 ts AS localtimestamp,
 WATERMARK FOR ts AS ts
) WITH (
 'connector' = 'datagen',
 -- optional options --
 'rows-per-second'='5',
 'fields.f_sequence.kind'='sequence',
 'fields.f_sequence.start'='1',
 'fields.f_sequence.end'='1000',
 'fields.f_random.min'='1',
 'fields.f_random.max'='1000',
 'fields.f_random_str.length'='10'
)

Connector Options