Doris Output Plugin

This plugin is used to output data to Doris for logstash, use the HTTP protocol to interact with the Doris FE Http interface, and import data through Doris’s stream load.

Learn more about Doris Stream Load

Learn more about Doris

Install and compile

1.Download source code

2.compile

Execute under extension/logstash/ directory

gem build logstash-output-doris.gemspec

You will get logstash-output-doris-{version}.gem file in the same directory

3.Plug-in installation

copy logstash-output-doris-{version}.gem to the logstash installation directory

Executing an order

./bin/logstash-plugin install logstash-output-doris-{version}.gem

Install logstash-output-doris plugin

Configuration

Example:

Create a new configuration file in the config directory and name it logstash-doris.conf

The specific configuration is as follows:

1. output {
2. doris {
3. http_hosts => [ "http://fehost:8030" ]
4. user => user_name
5. password => password
6. db => "db_name"
7. table => "table_name"
8. label_prefix => "label_prefix"
9. column_separator => ","
10. }
11. }

Configuration instructions:

Connection configuration:

Configuration Explanation
httphosts FE’s HTTP interactive address eg : [“<http://fe1:8030>“, “<http://fe2:8030>“]
user User name, the user needs to have import permission for the doris table
password Password
db Database name
table Table name
label_prefix Import the identification prefix, the final generated ID is {label_prefix}{db}{table}{time_stamp}

Load configuration:(Reference documents

Configuration Explanation
column_separator Column separator, the default is \t
columns Used to specify the correspondence between the columns in the import file and the columns in the table
where The filter conditions specified by the import task
max_filter_ratio The maximum tolerance rate of the import task, the default is zero tolerance
partition Partition information of the table to be imported
timeout timeout, the default is 600s
strict_mode Strict mode, the default is false
timezone Specify the time zone used for this import, the default is the East Eight District
exec_mem_limit Import memory limit, default is 2GB, unit is byte
format Specify the format of imported data, support csv, json, csv_with_names, csv_with_names_and_types, parquet, orc, default is csv.
jsonpaths The matching pattern needs to match the corresponding value via the jsonpaths parameter.
json_root Used to specify the root node of a json document, default value is “”.
fuzzy_parse Boolean type, true means that json will be parsed with the schema of the first row. Enabling this option can improve the efficiency of json import, but requires that the order of the keys of all json objects is the same as the first row, the default is false, only use in json format
num_as_string Boolean type, true means that when parsing json data, the numeric type will be converted to a string, and then imported without losing precision.

Other configuration:

Configuration Explanation
save_on_failure If the import fails to save locally, the default is true
save_dir Local save directory, default is /tmp
automatic_retries The maximum number of retries on failure, the default is 3
batch_size The maximum number of events processed per batch, the default is 100000
idle_flush_time Maximum interval, the default is 20 (seconds)

Start Up

Run the command to start the doris output plugin:

{logstash-home}/bin/logstash -f {logstash-home}/config/logstash-doris.conf --config.reload.automatic

Complete usage example

1. Compile doris-output-plugin

1> Download the ruby compressed package and go to ruby official website to download it. The version 2.7.1 used here

2> Compile and install, configure ruby environment variables

3> Go to the doris source extension/logstash/ directory and execute

gem build logstash-output-doris.gemspec

Get the file logstash-output-doris-0.1.0.gem, and the compilation is complete

2. Install and configure filebeat (here use filebeat as input)

1> es official website Download the filebeat tar compression package and decompress it

2> Enter the filebeat directory and modify the configuration file filebeat.yml as follows:

1. filebeat.inputs:
2. - type: log
3. paths:
4. - /tmp/doris.data
5. output.logstash:
6. hosts: ["localhost:5044"]

/tmp/doris.data is the doris data path

3> Start filebeat:

./filebeat -e -c filebeat.yml -d "publish"

3.Install logstash and doris-out-plugin

1> es official website Download the logstash tar compressed package and decompress it

2> Copy the logstash-output-doris-0.1.0.gem obtained in step 1 to the logstash installation directory

3> execute

./bin/logstash-plugin install logstash-output-doris-0.1.0.gem

Install the plugin

4> Create a new configuration file logstash-doris.conf in the config directory as follows:

1. input {
2. beats {
3. port => "5044"
4. }
5. }

7. output {
8. doris {
9. http_hosts => [ "http://127.0.0.1:8030" ]
10. user => doris
11. password => doris
12. db => "logstash_output_test"
13. table => "output"
14. label_prefix => "doris"
15. column_separator => ","
16. columns => "a,b,c,d,e"
17. }
18. }

The configuration here needs to be configured according to the configuration instructions

5> Start logstash:

./bin/logstash -f ./config/logstash-doris.conf —config.reload.automatic

4.Test Load

Add write data to /tmp/doris.data

echo a,b,c,d,e >> /tmp/doris.data

Observe the logstash log. If the status of the returned response is Success, the import was successful. At this time, you can view the imported data in the logstash_output_test.output table