Seatunnel

The newest Apache SeaTunnel (formerly waterdrop ) already supports Doris’s connector, SeaTunnel can use Spark engine and Flink engine to synchronize data to Doris.

Flink Sink Doris(2.x)

Seatunnel Flink Sink Doris plugin code

Options

fenodes [string]

Doris Fe http url, eg: 127.0.0.1:8030

database [string]

Doris database

table [string]

Doris table

user [string]

Doris user

password [string]

Doris password

batch_size [int]

The maximum number of lines to write to Doris at a time, the default value is 100

interval [int]

The flush interval (in milliseconds), after which the asynchronous thread writes the data in the cache to Doris. Set to 0 to turn off periodic writes.

max_retries [int]

Number of retries after writing to Doris fails

doris.* [string]

Import parameters for Stream load. For example: ‘doris.column_separator’ = ‘, ‘ etc.

More Stream Load parameter configuration

Examples

Socket To Doris

1. env {
2. execution.parallelism = 1
3. }
4. source {
5. SocketStream {
6. host = 127.0.0.1
7. port = 9999
8. result_table_name = "socket"
9. field_name = "info"
10. }
11. }
12. transform {
13. }
14. sink {
15. DorisSink {
16. fenodes = "127.0.0.1:8030"
17. user = root
18. password = 123456
19. database = test
20. table = test_tbl
21. batch_size = 5
22. max_retries = 1
23. interval = 5000
24. }
25. }

Start command

1. sh bin/start-seatunnel-flink.sh --config config/flink.streaming.conf