Seatunnel

The newest Apache SeaTunnel (formerly waterdrop ) (opens new window) already supports Doris’s connector, SeaTunnel can use Spark engine and Flink engine to synchronize data to Doris.

Flink Sink Doris(2.x)

Seatunnel Flink Sink Doris plugin code (opens new window)

Options

fenodes [string]

Doris Fe http url, eg: 127.0.0.1:8030

database [string]

Doris database

table [string]

Doris table

user [string]

Doris user

password [string]

Doris password

batch_size [int]

The maximum number of lines to write to Doris at a time, the default value is 100

interval [int]

The flush interval (in milliseconds), after which the asynchronous thread writes the data in the cache to Doris. Set to 0 to turn off periodic writes.

max_retries [int]

Number of retries after writing to Doris fails

doris.* [string]

Import parameters for Stream load. For example: ‘doris.column_separator’ = ‘, ‘ etc.

More Stream Load parameter configuration (opens new window)

Examples

Socket To Doris

1. env {
2. execution.parallelism = 1
3. }
4. source {
5. SocketStream {
6. host = 127.0.0.1
7. port = 9999
8. result_table_name = "socket"
9. field_name = "info"
10. }
11. }
12. transform {
13. }
14. sink {
15. DorisSink {
16. fenodes = "127.0.0.1:8030"
17. user = root
18. password = 123456
19. database = test
20. table = test_tbl
21. batch_size = 5
22. max_retries = 1
23. interval = 5000
24. }
25. }

Start command

1. sh bin/start-seatunnel-flink.sh --config config/flink.streaming.conf