Seatunnel

The newest Apache SeaTunnel (formerly waterdrop ) (opens new window) already supports Doris’s connector, SeaTunnel can use Spark engine and Flink engine to synchronize data to Doris.

Flink Sink Doris(2.x)

Seatunnel Flink Sink Doris plugin code (opens new window)

Options

fenodes [string]

Doris Fe http url, eg: 127.0.0.1:8030

database [string]

Doris database

table [string]

Doris table

user [string]

Doris user

password [string]

Doris password

batch_size [int]

The maximum number of lines to write to Doris at a time, the default value is 100

interval [int]

The flush interval (in milliseconds), after which the asynchronous thread writes the data in the cache to Doris. Set to 0 to turn off periodic writes.

max_retries [int]

Number of retries after writing to Doris fails

doris.* [string]

Import parameters for Stream load. For example: ‘doris.column_separator’ = ‘, ‘ etc.

More Stream Load parameter configuration (opens new window)

Examples

Socket To Doris

env {
  execution.parallelism = 1
}
source {
    SocketStream {
      host = 127.0.0.1
      port = 9999
      result_table_name = "socket"
      field_name = "info"
    }
}
transform {
}
sink {
  DorisSink {
      fenodes = "127.0.0.1:8030"
      user = root
      password = 123456
      database = test
      table = test_tbl
      batch_size = 5
      max_retries = 1
      interval = 5000
    }
}

Start command

sh bin/start-seatunnel-flink.sh --config config/flink.streaming.conf