High Performance Writing

This chapter introduces how to write data into TDengine with high throughput.

How to achieve high performance data writing

To achieve high performance writing, there are a few aspects to consider. In the following sections we will describe these important factors in achieving high performance writing.

Application Program

From the perspective of application program, you need to consider:

1. The data size of each single write, also known as batch size. Generally speaking, higher batch size generates better writing performance. However, once the batch size is over a specific value, you will not get any additional benefit anymore. When using SQL to write into TDengine, it’s better to put as much as possible data in single SQL. The maximum SQL length supported by TDengine is 1,048,576 bytes, i.e. 1 MB.
2. The number of concurrent connections. Normally more connections can get better result. However, once the number of connections exceeds the processing ability of the server side, the performance may downgrade.
3. The distribution of data to be written across tables or sub-tables. Writing to single table in one batch is more efficient than writing to multiple tables in one batch.

4. Data Writing Protocol.

5. Parameter binding mode is more efficient than SQL because it doesn’t have the cost of parsing SQL.

6. Writing to known existing tables is more efficient than writing to uncertain tables in automatic creating mode because the later needs to check whether the table exists or not before actually writing data into it.
7. Writing in SQL is more efficient than writing in schemaless mode because schemaless writing creates table automatically and may alter table schema.

Application programs need to take care of the above factors and try to take advantage of them. The application progam should write to single table in each write batch. The batch size needs to be tuned to a proper value on a specific system. The number of concurrent connections needs to be tuned to a proper value too to achieve the best writing throughput.

Data Source

Application programs need to read data from data source then write into TDengine. If you meet one or more of below situations, you need to setup message queues between the threads for reading from data source and the threads for writing into TDengine.

1. There are multiple data sources, the data generation speed of each data source is much slower than the speed of single writing thread. In this case, the purpose of message queues is to consolidate the data from multiple data sources together to increase the batch size of single write.
2. The speed of data generation from single data source is much higher than the speed of single writing thread. The purpose of message queue in this case is to provide buffer so that data is not lost and multiple writing threads can get data from the buffer.
3. The data for single table are from multiple data source. In this case the purpose of message queues is to combine the data for single table together to improve the write efficiency.

If the data source is Kafka, then the application program is a consumer of Kafka, you can benefit from some kafka features to achieve high performance writing:

1. Put the data for a table in single partition of single topic so that it’s easier to put the data for each table together and write in batch
2. Subscribe multiple topics to accumulate data together.
3. Add more consumers to gain more concurrency and throughput.
4. Incrase the size of single fetch to increase the size of write batch.

Tune TDengine

On the server side, database configuration parameter vgroups needs to be set carefully to maximize the system performance. If it’s set too low, the system capability can’t be utilized fully; if it’s set too big, unnecessary resource competition may be produced. A normal recommendation for vgroups parameter is 2 times of the number of CPU cores. However, depending on the actual system resources, it may still need to tuned.

For more configuration parameters, please refer to Database Configuration and Server Configuration。

Sample Programs

This section will introduce the sample programs to demonstrate how to write into TDengine with high performance.

Scenario

Below are the scenario for the sample programs of high performance writing.

• Application program reads data from data source, the sample program simulates a data source by generating data
• The speed of single writing thread is much slower than the speed of generating data, so the program starts multiple writing threads while each thread establish a connection to TDengine and each thread has a message queue of fixed size.
• Application program maps the received data to different writing threads based on table name to make sure all the data for each table is always processed by a specific writing thread.
• Each writing thread writes the received data into TDengine once the message queue becomes empty or the read data meets a threshold.

Sample Programs

The sample programs listed in this section are based on the scenario described previously. If your scenarios is different, please try to adjust the code based on the principles described in this chapter.

The sample programs assume the source data is for all the different sub tables in same super table (meters). The super table has been created before the sample program starts to writing data. Sub tables are created automatically according to received data. If there are multiple super tables in your case, please try to adjust the part of creating table automatically.

• Java
• Python

Program Inventory

Below is the list of complete code of the classes in above table and more detailed description.

FastWriteExample

The main Program is responsible for:

1. Create message queues
2. Start writing threads
3. Start reading threads
4. Output writing speed every 10 seconds

The main program provides 4 parameters for tuning：

1. The number of reading threads, default value is 1
2. The number of writing threads, default value is 2
3. The total number of tables in the generated data, default value is 1000. These tables are distributed evenly across all writing threads. If the number of tables is very big, it will cost much time to firstly create these tables.
4. The batch size of single write, default value is 3,000

The capacity of message queue also impacts performance and can be tuned by modifying program. Normally it’s always better to have a larger message queue. A larger message queue means lower possibility of being blocked when enqueueing and higher throughput. But a larger message queue consumes more memory space. The default value used in the sample programs is already big enough.

1. package com.taos.example.highvolume;

3. import org.slf4j.Logger;
4. import org.slf4j.LoggerFactory;

6. import java.sql.*;
7. import java.util.ArrayList;
8. import java.util.List;
9. import java.util.concurrent.ArrayBlockingQueue;
10. import java.util.concurrent.BlockingQueue;

13. public class FastWriteExample {
14. final static Logger logger = LoggerFactory.getLogger(FastWriteExample.class);

16. final static int taskQueueCapacity = 1000000;
17. final static List<BlockingQueue<String>> taskQueues = new ArrayList<>();
18. final static List<ReadTask> readTasks = new ArrayList<>();
19. final static List<WriteTask> writeTasks = new ArrayList<>();
20. final static DataBaseMonitor databaseMonitor = new DataBaseMonitor();

22. public static void stopAll() {
23. logger.info("shutting down");
24. readTasks.forEach(task -> task.stop());
25. writeTasks.forEach(task -> task.stop());
26. databaseMonitor.close();
27. }

29. public static void main(String[] args) throws InterruptedException, SQLException {
30. int readTaskCount = args.length > 0 ? Integer.parseInt(args[0]) : 1;
31. int writeTaskCount = args.length > 1 ? Integer.parseInt(args[1]) : 3;
32. int tableCount = args.length > 2 ? Integer.parseInt(args[2]) : 1000;
33. int maxBatchSize = args.length > 3 ? Integer.parseInt(args[3]) : 3000;

35. logger.info("readTaskCount={}, writeTaskCount={} tableCount={} maxBatchSize={}",
36. readTaskCount, writeTaskCount, tableCount, maxBatchSize);

38. databaseMonitor.init().prepareDatabase();

40. // Create task queues, whiting tasks and start writing threads.
41. for (int i = 0; i < writeTaskCount; ++i) {
42. BlockingQueue<String> queue = new ArrayBlockingQueue<>(taskQueueCapacity);
43. taskQueues.add(queue);
44. WriteTask task = new WriteTask(queue, maxBatchSize);
45. Thread t = new Thread(task);
46. t.setName("WriteThread-" + i);
47. t.start();
48. }

50. // create reading tasks and start reading threads
51. int tableCountPerTask = tableCount / readTaskCount;
52. for (int i = 0; i < readTaskCount; ++i) {
53. ReadTask task = new ReadTask(i, taskQueues, tableCountPerTask);
54. Thread t = new Thread(task);
55. t.setName("ReadThread-" + i);
56. t.start();
57. }

59. Runtime.getRuntime().addShutdownHook(new Thread(FastWriteExample::stopAll));

61. long lastCount = 0;
62. while (true) {
63. Thread.sleep(10000);
64. long numberOfTable = databaseMonitor.getTableCount();
65. long count = databaseMonitor.count();
66. logger.info("numberOfTable={} count={} speed={}", numberOfTable, count, (count - lastCount) / 10);
67. lastCount = count;
68. }
69. }
70. }

view source code

ReadTask

ReadTask reads data from data source. Each ReadTask is associated with a simulated data source, each data source generates data for a group of specific tables, and the data of any table is only generated from a single specific data source.

ReadTask puts data in message queue in blocking mode. That means, the putting operation is blocked if the message queue is full.

1. package com.taos.example.highvolume;

3. import org.slf4j.Logger;
4. import org.slf4j.LoggerFactory;

6. import java.util.Iterator;
7. import java.util.List;
8. import java.util.concurrent.BlockingQueue;

10. class ReadTask implements Runnable {
11. private final static Logger logger = LoggerFactory.getLogger(ReadTask.class);
12. private final int taskId;
13. private final List<BlockingQueue<String>> taskQueues;
14. private final int queueCount;
15. private final int tableCount;
16. private boolean active = true;

18. public ReadTask(int readTaskId, List<BlockingQueue<String>> queues, int tableCount) {
19. this.taskId = readTaskId;
20. this.taskQueues = queues;
21. this.queueCount = queues.size();
22. this.tableCount = tableCount;
23. }

25. /**
26. * Assign data received to different queues.
27. * Here we use the suffix number in table name.
28. * You are expected to define your own rule in practice.
29. *
30. * @param line record received
31. * @return which queue to use
32. */
33. public int getQueueId(String line) {
34. String tbName = line.substring(0, line.indexOf(',')); // For example: tb1_101
35. String suffixNumber = tbName.split("_")[1];
36. return Integer.parseInt(suffixNumber) % this.queueCount;
37. }

39. @Override
40. public void run() {
41. logger.info("started");
42. Iterator<String> it = new MockDataSource("tb" + this.taskId, tableCount);
43. try {
44. while (it.hasNext() && active) {
45. String line = it.next();
46. int queueId = getQueueId(line);
47. taskQueues.get(queueId).put(line);
48. }
49. } catch (Exception e) {
50. logger.error("Read Task Error", e);
51. }
52. }

54. public void stop() {
55. logger.info("stop");
56. this.active = false;
57. }
58. }

view source code

WriteTask

1. package com.taos.example.highvolume;

3. import org.slf4j.Logger;
4. import org.slf4j.LoggerFactory;

6. import java.util.concurrent.BlockingQueue;

8. class WriteTask implements Runnable {
9. private final static Logger logger = LoggerFactory.getLogger(WriteTask.class);
10. private final int maxBatchSize;

12. // the queue from which this writing task get raw data.
13. private final BlockingQueue<String> queue;

15. // A flag indicate whether to continue.
16. private boolean active = true;

18. public WriteTask(BlockingQueue<String> taskQueue, int maxBatchSize) {
19. this.queue = taskQueue;
20. this.maxBatchSize = maxBatchSize;
21. }

23. @Override
24. public void run() {
25. logger.info("started");
26. String line = null; // data getting from the queue just now.
27. SQLWriter writer = new SQLWriter(maxBatchSize);
28. try {
29. writer.init();
30. while (active) {
31. line = queue.poll();
32. if (line != null) {
33. // parse raw data and buffer the data.
34. writer.processLine(line);
35. } else if (writer.hasBufferedValues()) {
36. // write data immediately if no more data in the queue
37. writer.flush();
38. } else {
39. // sleep a while to avoid high CPU usage if no more data in the queue and no buffered records, .
40. Thread.sleep(100);
41. }
42. }
43. if (writer.hasBufferedValues()) {
44. writer.flush();
45. }
46. } catch (Exception e) {
47. String msg = String.format("line=%s, bufferedCount=%s", line, writer.getBufferedCount());
48. logger.error(msg, e);
49. } finally {
50. writer.close();
51. }
52. }

54. public void stop() {
55. logger.info("stop");
56. this.active = false;
57. }
58. }

view source code

MockDataSource

1. package com.taos.example.highvolume;

3. import java.util.Iterator;

5. /**
6. * Generate test data
7. */
8. class MockDataSource implements Iterator {
9. private String tbNamePrefix;
10. private int tableCount;
11. private long maxRowsPerTable = 1000000000L;

13. // 100 milliseconds between two neighbouring rows.
14. long startMs = System.currentTimeMillis() - maxRowsPerTable * 100;
15. private int currentRow = 0;
16. private int currentTbId = -1;

18. // mock values
19. String[] location = {"California.LosAngeles", "California.SanDiego", "California.SanJose", "California.Campbell", "California.SanFrancisco"};
20. float[] current = {8.8f, 10.7f, 9.9f, 8.9f, 9.4f};
21. int[] voltage = {119, 116, 111, 113, 118};
22. float[] phase = {0.32f, 0.34f, 0.33f, 0.329f, 0.141f};

24. public MockDataSource(String tbNamePrefix, int tableCount) {
25. this.tbNamePrefix = tbNamePrefix;
26. this.tableCount = tableCount;
27. }

29. @Override
30. public boolean hasNext() {
31. currentTbId += 1;
32. if (currentTbId == tableCount) {
33. currentTbId = 0;
34. currentRow += 1;
35. }
36. return currentRow < maxRowsPerTable;
37. }

39. @Override
40. public String next() {
41. long ts = startMs + 100 * currentRow;
42. int groupId = currentTbId % 5 == 0 ? currentTbId / 5 : currentTbId / 5 + 1;
43. StringBuilder sb = new StringBuilder(tbNamePrefix + "_" + currentTbId + ","); // tbName
44. sb.append(ts).append(','); // ts
45. sb.append(current[currentRow % 5]).append(','); // current
46. sb.append(voltage[currentRow % 5]).append(','); // voltage
47. sb.append(phase[currentRow % 5]).append(','); // phase
48. sb.append(location[currentRow % 5]).append(','); // location
49. sb.append(groupId); // groupID

51. return sb.toString();
52. }
53. }

view source code

SQLWriter

SQLWriter class encapsulates the logic of composing SQL and writing data. Please be noted that the tables have not been created before writing, but are created automatically when catching the exception of table doesn’t exist. For other exceptions caught, the SQL which caused the exception are logged for you to debug.

1. package com.taos.example.highvolume;

3. import org.slf4j.Logger;
4. import org.slf4j.LoggerFactory;

6. import java.sql.*;
7. import java.util.HashMap;
8. import java.util.Map;

10. /**
11. * A helper class encapsulate the logic of writing using SQL.
12. * <p>
13. * The main interfaces are two methods:
14. * <ol>
15. *     <li>{@link SQLWriter#processLine}, which receive raw lines from WriteTask and group them by table names.</li>
16. *     <li>{@link  SQLWriter#flush}, which assemble INSERT statement and execute it.</li>
17. * </ol>
18. * <p>
19. * There is a technical skill worth mentioning: we create table as needed when "table does not exist" error occur instead of creating table automatically using syntax "INSET INTO tb USING stb".
20. * This ensure that checking table existence is a one-time-only operation.
21. * </p>
22. *
23. * </p>
24. */
25. public class SQLWriter {
26. final static Logger logger = LoggerFactory.getLogger(SQLWriter.class);

28. private Connection conn;
29. private Statement stmt;

31. /**
32. * current number of buffered records
33. */
34. private int bufferedCount = 0;
35. /**
36. * Maximum number of buffered records.
37. * Flush action will be triggered if bufferedCount reached this value,
38. */
39. private int maxBatchSize;

42. /**
43. * Maximum SQL length.
44. */
45. private int maxSQLLength;

47. /**
48. * Map from table name to column values. For example:
49. * "tb001" -> "(1648432611249,2.1,114,0.09) (1648432611250,2.2,135,0.2)"
50. */
51. private Map<String, String> tbValues = new HashMap<>();

53. /**
54. * Map from table name to tag values in the same order as creating stable.
55. * Used for creating table.
56. */
57. private Map<String, String> tbTags = new HashMap<>();

59. public SQLWriter(int maxBatchSize) {
60. this.maxBatchSize = maxBatchSize;
61. }

64. /**
65. * Get Database Connection
66. *
67. * @return Connection
68. * @throws SQLException
69. */
70. private static Connection getConnection() throws SQLException {
71. String jdbcURL = System.getenv("TDENGINE_JDBC_URL");
72. return DriverManager.getConnection(jdbcURL);
73. }

75. /**
76. * Create Connection and Statement
77. *
78. * @throws SQLException
79. */
80. public void init() throws SQLException {
81. conn = getConnection();
82. stmt = conn.createStatement();
83. stmt.execute("use test");
84. ResultSet rs = stmt.executeQuery("show variables");
85. while (rs.next()) {
86. String configName = rs.getString(1);
87. if ("maxSQLLength".equals(configName)) {
88. maxSQLLength = Integer.parseInt(rs.getString(2));
89. logger.info("maxSQLLength={}", maxSQLLength);
90. }
91. }
92. }

94. /**
95. * Convert raw data to SQL fragments, group them by table name and cache them in a HashMap.
96. * Trigger writing when number of buffered records reached maxBachSize.
97. *
98. * @param line raw data get from task queue in format: tbName,ts,current,voltage,phase,location,groupId
99. */
100. public void processLine(String line) throws SQLException {
101. bufferedCount += 1;
102. int firstComma = line.indexOf(',');
103. String tbName = line.substring(0, firstComma);
104. int lastComma = line.lastIndexOf(',');
105. int secondLastComma = line.lastIndexOf(',', lastComma - 1);
106. String value = "(" + line.substring(firstComma + 1, secondLastComma) + ") ";
107. if (tbValues.containsKey(tbName)) {
108. tbValues.put(tbName, tbValues.get(tbName) + value);
109. } else {
110. tbValues.put(tbName, value);
111. }
112. if (!tbTags.containsKey(tbName)) {
113. String location = line.substring(secondLastComma + 1, lastComma);
114. String groupId = line.substring(lastComma + 1);
115. String tagValues = "('" + location + "'," + groupId + ')';
116. tbTags.put(tbName, tagValues);
117. }
118. if (bufferedCount == maxBatchSize) {
119. flush();
120. }
121. }

124. /**
125. * Assemble INSERT statement using buffered SQL fragments in Map {@link SQLWriter#tbValues} and execute it.
126. * In case of "Table does not exit" exception, create all tables in the sql and retry the sql.
127. */
128. public void flush() throws SQLException {
129. StringBuilder sb = new StringBuilder("INSERT INTO ");
130. for (Map.Entry<String, String> entry : tbValues.entrySet()) {
131. String tableName = entry.getKey();
132. String values = entry.getValue();
133. String q = tableName + " values " + values + " ";
134. if (sb.length() + q.length() > maxSQLLength) {
135. executeSQL(sb.toString());
136. logger.warn("increase maxSQLLength or decrease maxBatchSize to gain better performance");
137. sb = new StringBuilder("INSERT INTO ");
138. }
139. sb.append(q);
140. }
141. executeSQL(sb.toString());
142. tbValues.clear();
143. bufferedCount = 0;
144. }

146. private void executeSQL(String sql) throws SQLException {
147. try {
148. stmt.executeUpdate(sql);
149. } catch (SQLException e) {
150. // convert to error code defined in taoserror.h
151. int errorCode = e.getErrorCode() & 0xffff;
152. if (errorCode == 0x362 || errorCode == 0x218) {
153. // Table does not exist
154. createTables();
155. executeSQL(sql);
156. } else {
157. logger.error("Execute SQL: {}", sql);
158. throw e;
159. }
160. } catch (Throwable throwable) {
161. logger.error("Execute SQL: {}", sql);
162. throw throwable;
163. }
164. }

166. /**
167. * Create tables in batch using syntax:
168. * <p>
169. * CREATE TABLE [IF NOT EXISTS] tb_name1 USING stb_name TAGS (tag_value1, ...) [IF NOT EXISTS] tb_name2 USING stb_name TAGS (tag_value2, ...) ...;
170. * </p>
171. */
172. private void createTables() throws SQLException {
173. StringBuilder sb = new StringBuilder("CREATE TABLE ");
174. for (String tbName : tbValues.keySet()) {
175. String tagValues = tbTags.get(tbName);
176. sb.append("IF NOT EXISTS ").append(tbName).append(" USING meters TAGS ").append(tagValues).append(" ");
177. }
178. String sql = sb.toString();
179. try {
180. stmt.executeUpdate(sql);
181. } catch (Throwable throwable) {
182. logger.error("Execute SQL: {}", sql);
183. throw throwable;
184. }
185. }

187. public boolean hasBufferedValues() {
188. return bufferedCount > 0;
189. }

191. public int getBufferedCount() {
192. return bufferedCount;
193. }

195. public void close() {
196. try {
197. stmt.close();
198. } catch (SQLException e) {
199. }
200. try {
201. conn.close();
202. } catch (SQLException e) {
203. }
204. }
205. }

view source code

DataBaseMonitor

1. package com.taos.example.highvolume;

3. import java.sql.*;

5. /**
6. * Prepare target database.
7. * Count total records in database periodically so that we can estimate the writing speed.
8. */
9. public class DataBaseMonitor {
10. private Connection conn;
11. private Statement stmt;

13. public DataBaseMonitor init() throws SQLException {
14. if (conn == null) {
15. String jdbcURL = System.getenv("TDENGINE_JDBC_URL");
16. conn = DriverManager.getConnection(jdbcURL);
17. stmt = conn.createStatement();
18. }
19. return this;
20. }

22. public void close() {
23. try {
24. stmt.close();
25. } catch (SQLException e) {
26. }
27. try {
28. conn.close();
29. } catch (SQLException e) {
30. }
31. }

33. public void prepareDatabase() throws SQLException {
34. stmt.execute("DROP DATABASE IF EXISTS test");
35. stmt.execute("CREATE DATABASE test");
36. stmt.execute("CREATE STABLE test.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)");
37. }

39. public Long count() throws SQLException {
40. if (!stmt.isClosed()) {
41. ResultSet result = stmt.executeQuery("SELECT count(*) from test.meters");
42. result.next();
43. return result.getLong(1);
44. }
45. return null;
46. }

48. /**
49. * show test.stables;
50. *
51. *               name              |      created_time       | columns |  tags  |   tables    |
52. * ============================================================================================
53. *  meters                         | 2022-07-20 08:39:30.902 |       4 |      2 |      620000 |
54. */
55. public  Long getTableCount() throws SQLException {
56. if (!stmt.isClosed()) {
57. ResultSet result = stmt.executeQuery("show test.stables");
58. result.next();
59. return result.getLong(5);
60. }
61. return null;
62. }
63. }

view source code

Steps to Launch

Launch Java Sample Program

You need to set environment variable TDENGINE_JDBC_URL before launching the program. If TDengine Server is setup on localhost, then the default value for user name, password and port can be used, like below:

1. TDENGINE_JDBC_URL="jdbc:TAOS://localhost:6030?user=root&password=taosdata"

Launch in IDE

1. Clone TDengine repository

1. git clone git@github.com:taosdata/TDengine.git --depth 1 2. Use IDE to open docs/examples/java directory 3. Configure environment variable TDENGINE_JDBC_URL, you can also configure it before launching the IDE, if so you can skip this step. 4. Run class com.taos.example.highvolume.FastWriteExample

Launch on server

If you want to launch the sample program on a remote server, please follow below steps:

1. Package the sample programs. Execute below command under directory TDengine/docs/examples/java ：

1. mvn package 2. Create examples/java directory on the server

1. mkdir -p examples/java 3. Copy dependencies (below commands assume you are working on a local Windows host and try to launch on a remote Linux host)

• Copy dependent packages

  1. scp -r .\target\lib <user>@<host>:~/examples/java - Copy the jar of sample programs

  1. scp -r .\target\javaexample-1.0.jar <user>@<host>:~/examples/java 4. Configure environment variable Edit ~/.bash_profile or ~/.bashrc and add below:

1. export TDENGINE_JDBC_URL="jdbc:TAOS://localhost:6030?user=root&password=taosdata"

If your TDengine server is not deployed on localhost or doesn’t use default port, you need to change the above URL to correct value in your environment. 5. Launch the sample program

1. java -classpath lib/*:javaexample-1.0.jar com.taos.example.highvolume.FastWriteExample <read_thread_count> <white_thread_count> <total_table_count> <max_batch_size> 6. The sample program doesn’t exit unless you press CTRL + C to terminate it. Below is the output of running on a server of 16 cores, 64GB memory and SSD hard disk.

1. root@vm85$ java -classpath lib/*:javaexample-1.0.jar com.taos.example.highvolume.FastWriteExample 2 12 2. 18:56:35.896 [main] INFO c.t.e.highvolume.FastWriteExample - readTaskCount=2, writeTaskCount=12 tableCount=1000 maxBatchSize=3000 3. 18:56:36.011 [WriteThread-0] INFO c.taos.example.highvolume.WriteTask - started 4. 18:56:36.015 [WriteThread-0] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 5. 18:56:36.021 [WriteThread-1] INFO c.taos.example.highvolume.WriteTask - started 6. 18:56:36.022 [WriteThread-1] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 7. 18:56:36.031 [WriteThread-2] INFO c.taos.example.highvolume.WriteTask - started 8. 18:56:36.032 [WriteThread-2] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 9. 18:56:36.041 [WriteThread-3] INFO c.taos.example.highvolume.WriteTask - started 10. 18:56:36.042 [WriteThread-3] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 11. 18:56:36.093 [WriteThread-4] INFO c.taos.example.highvolume.WriteTask - started 12. 18:56:36.094 [WriteThread-4] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 13. 18:56:36.099 [WriteThread-5] INFO c.taos.example.highvolume.WriteTask - started 14. 18:56:36.100 [WriteThread-5] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 15. 18:56:36.100 [WriteThread-6] INFO c.taos.example.highvolume.WriteTask - started 16. 18:56:36.101 [WriteThread-6] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 17. 18:56:36.103 [WriteThread-7] INFO c.taos.example.highvolume.WriteTask - started 18. 18:56:36.104 [WriteThread-7] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 19. 18:56:36.105 [WriteThread-8] INFO c.taos.example.highvolume.WriteTask - started 20. 18:56:36.107 [WriteThread-8] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 21. 18:56:36.108 [WriteThread-9] INFO c.taos.example.highvolume.WriteTask - started 22. 18:56:36.109 [WriteThread-9] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 23. 18:56:36.156 [WriteThread-10] INFO c.taos.example.highvolume.WriteTask - started 24. 18:56:36.157 [WriteThread-11] INFO c.taos.example.highvolume.WriteTask - started 25. 18:56:36.158 [WriteThread-10] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 26. 18:56:36.158 [ReadThread-0] INFO com.taos.example.highvolume.ReadTask - started 27. 18:56:36.158 [ReadThread-1] INFO com.taos.example.highvolume.ReadTask - started 28. 18:56:36.158 [WriteThread-11] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576 29. 18:56:46.369 [main] INFO c.t.e.highvolume.FastWriteExample - count=18554448 speed=1855444 30. 18:56:56.946 [main] INFO c.t.e.highvolume.FastWriteExample - count=39059660 speed=2050521 31. 18:57:07.322 [main] INFO c.t.e.highvolume.FastWriteExample - count=59403604 speed=2034394 32. 18:57:18.032 [main] INFO c.t.e.highvolume.FastWriteExample - count=80262938 speed=2085933 33. 18:57:28.432 [main] INFO c.t.e.highvolume.FastWriteExample - count=101139906 speed=2087696 34. 18:57:38.921 [main] INFO c.t.e.highvolume.FastWriteExample - count=121807202 speed=2066729 35. 18:57:49.375 [main] INFO c.t.e.highvolume.FastWriteExample - count=142952417 speed=2114521 36. 18:58:00.689 [main] INFO c.t.e.highvolume.FastWriteExample - count=163650306 speed=2069788 37. 18:58:11.646 [main] INFO c.t.e.highvolume.FastWriteExample - count=185019808 speed=2136950

Program Inventory

Sample programs in Python uses multi-process and cross-process message queues.

main function

main function is responsible for creating message queues and fork child processes, there are 3 kinds of child processes:

1. Monitoring process, initializes database and calculating writing speed
2. Reading process (n), reads data from data source
3. Writing process (m), writes data into TDengine

main function provides 5 parameters:

1. The number of reading tasks, default value is 1
2. The number of writing tasks, default value is 1
3. The number of tables, default value is 1,000
4. The capacity of message queue, default value is 1,000,000 bytes
5. The batch size in single write, default value is 3000

1. def main():
2. set_global_config()
3. logging.info(f"READ_TASK_COUNT={READ_TASK_COUNT}, WRITE_TASK_COUNT={WRITE_TASK_COUNT}, "
4. f"TABLE_COUNT={TABLE_COUNT}, QUEUE_SIZE={QUEUE_SIZE}, MAX_BATCH_SIZE={MAX_BATCH_SIZE}")

6. monitor_process = Process(target=run_monitor_process)
7. monitor_process.start()
8. time.sleep(3)  # waiting for database ready.

10. task_queues: List[Queue] = []
11. # create task queues
12. for i in range(WRITE_TASK_COUNT):
13. queue = Queue(max_size_bytes=QUEUE_SIZE)
14. task_queues.append(queue)

16. # create write processes
17. for i in range(WRITE_TASK_COUNT):
18. p = Process(target=run_write_task, args=(i, task_queues[i]))
19. p.start()
20. logging.debug(f"WriteTask-{i} started with pid {p.pid}")
21. write_processes.append(p)

23. # create read processes
24. for i in range(READ_TASK_COUNT):
25. queues = assign_queues(i, task_queues)
26. p = Process(target=run_read_task, args=(i, queues))
27. p.start()
28. logging.debug(f"ReadTask-{i} started with pid {p.pid}")
29. read_processes.append(p)

31. try:
32. monitor_process.join()
33. except KeyboardInterrupt:
34. monitor_process.terminate()
35. [p.terminate() for p in read_processes]
36. [p.terminate() for p in write_processes]
37. [q.close() for q in task_queues]

40. def assign_queues(read_task_id, task_queues):
41. """
42. Compute target queues for a specific read task.
43. """
44. ratio = WRITE_TASK_COUNT / READ_TASK_COUNT
45. from_index = math.floor(read_task_id * ratio)
46. end_index = math.ceil((read_task_id + 1) * ratio)
47. return task_queues[from_index:end_index]

50. if __name__ == '__main__':
51. main()

view source code

run_monitor_process

Monitoring process initializes database and monitoring writing speed.

1. def run_monitor_process():
2. log = logging.getLogger("DataBaseMonitor")
3. conn = get_connection()
4. conn.execute("DROP DATABASE IF EXISTS test")
5. conn.execute("CREATE DATABASE test")
6. conn.execute("CREATE STABLE test.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) "
7. "TAGS (location BINARY(64), groupId INT)")

9. def get_count():
10. res = conn.query("SELECT count(*) FROM test.meters")
11. rows = res.fetch_all()
12. return rows[0][0] if rows else 0

14. last_count = 0
15. while True:
16. time.sleep(10)
17. count = get_count()
18. log.info(f"count={count} speed={(count - last_count) / 10}")
19. last_count = count

view source code

run_read_task function

Reading process reads data from other data system and distributes to the message queue allocated for it.

2. def run_read_task(task_id: int, task_queues: List[Queue]):
3. table_count_per_task = TABLE_COUNT // READ_TASK_COUNT
4. data_source = MockDataSource(f"tb{task_id}", table_count_per_task)
5. try:
6. for batch in data_source:
7. for table_id, rows in batch:
8. # hash data to different queue
9. i = table_id % len(task_queues)
10. # block putting forever when the queue is full
11. task_queues[i].put_many(rows, block=True, timeout=-1)
12. except KeyboardInterrupt:
13. pass

view source code

MockDataSource

Below is the simulated data source, we assume table name exists in each generated data.

1. import time

4. class MockDataSource:
5. samples = [
6. "8.8,119,0.32,California.LosAngeles,0",
7. "10.7,116,0.34,California.SanDiego,1",
8. "9.9,111,0.33,California.SanJose,2",
9. "8.9,113,0.329,California.Campbell,3",
10. "9.4,118,0.141,California.SanFrancisco,4"
11. ]

13. def __init__(self, tb_name_prefix, table_count):
14. self.table_name_prefix = tb_name_prefix + "_"
15. self.table_count = table_count
16. self.max_rows = 10000000
17. self.current_ts = round(time.time() * 1000) - self.max_rows * 100
18. # [(tableId, tableName, values),]
19. self.data = self._init_data()

21. def _init_data(self):
22. lines = self.samples * (self.table_count // 5 + 1)
23. data = []
24. for i in range(self.table_count):
25. table_name = self.table_name_prefix + str(i)
26. data.append((i, table_name, lines[i]))  # tableId, row
27. return data

29. def __iter__(self):
30. self.row = 0
31. return self

33. def __next__(self):
34. """
35. next 1000 rows for each table.
36. return: {tableId:[row,...]}
37. """
38. # generate 1000 timestamps
39. ts = []
40. for _ in range(1000):
41. self.current_ts += 100
42. ts.append(str(self.current_ts))
43. # add timestamp to each row
44. # [(tableId, ["tableName,ts,current,voltage,phase,location,groupId"])]
45. result = []
46. for table_id, table_name, values in self.data:
47. rows = [table_name + ',' + t + ',' + values for t in ts]
48. result.append((table_id, rows))
49. return result

view source code

run_write_task function

Writing process tries to read as much as possible data from message queue and writes in batch.

1. def run_write_task(task_id: int, queue: Queue):
2. from sql_writer import SQLWriter
3. log = logging.getLogger(f"WriteTask-{task_id}")
4. writer = SQLWriter(get_connection)
5. lines = None
6. try:
7. while True:
8. try:
9. # get as many as possible
10. lines = queue.get_many(block=False, max_messages_to_get=MAX_BATCH_SIZE)
11. writer.process_lines(lines)
12. except Empty:
13. time.sleep(0.01)
14. except KeyboardInterrupt:
15. pass
16. except BaseException as e:
17. log.debug(f"lines={lines}")
18. raise e

view source code

SQLWriter

SQLWriter class encapsulates the logic of composing SQL and writing data. Please be noted that the tables have not been created before writing, but are created automatically when catching the exception of table doesn’t exist. For other exceptions caught, the SQL which caused the exception are logged for you to debug. This class also checks the SQL length, and passes the maximum SQL length by parameter maxSQLLength according to actual TDengine limit.

1. import logging
2. import taos

5. class SQLWriter:
6. log = logging.getLogger("SQLWriter")

8. def __init__(self, get_connection_func):
9. self._tb_values = {}
10. self._tb_tags = {}
11. self._conn = get_connection_func()
12. self._max_sql_length = self.get_max_sql_length()
13. self._conn.execute("USE test")

15. def get_max_sql_length(self):
16. rows = self._conn.query("SHOW variables").fetch_all()
17. for r in rows:
18. name = r[0]
19. if name == "maxSQLLength":
20. return int(r[1])
21. return 1024 * 1024

23. def process_lines(self, lines: str):
24. """
25. :param lines: [[tbName,ts,current,voltage,phase,location,groupId]]
26. """
27. for line in lines:
28. ps = line.split(",")
29. table_name = ps[0]
30. value = '(' + ",".join(ps[1:-2]) + ') '
31. if table_name in self._tb_values:
32. self._tb_values[table_name] += value
33. else:
34. self._tb_values[table_name] = value

36. if table_name not in self._tb_tags:
37. location = ps[-2]
38. group_id = ps[-1]
39. tag_value = f"('{location}',{group_id})"
40. self._tb_tags[table_name] = tag_value
41. self.flush()

43. def flush(self):
44. """
45. Assemble INSERT statement and execute it.
46. When the sql length grows close to MAX_SQL_LENGTH, the sql will be executed immediately, and a new INSERT statement will be created.
47. In case of "Table does not exit" exception, tables in the sql will be created and the sql will be re-executed.
48. """
49. sql = "INSERT INTO "
50. sql_len = len(sql)
51. buf = []
52. for tb_name, values in self._tb_values.items():
53. q = tb_name + " VALUES " + values
54. if sql_len + len(q) >= self._max_sql_length:
55. sql += " ".join(buf)
56. self.execute_sql(sql)
57. sql = "INSERT INTO "
58. sql_len = len(sql)
59. buf = []
60. buf.append(q)
61. sql_len += len(q)
62. sql += " ".join(buf)
63. self.execute_sql(sql)
64. self._tb_values.clear()

66. def execute_sql(self, sql):
67. try:
68. self._conn.execute(sql)
69. except taos.Error as e:
70. error_code = e.errno & 0xffff
71. # Table does not exit
72. if error_code == 9731:
73. self.create_tables()
74. else:
75. self.log.error("Execute SQL: %s", sql)
76. raise e
77. except BaseException as baseException:
78. self.log.error("Execute SQL: %s", sql)
79. raise baseException

81. def create_tables(self):
82. sql = "CREATE TABLE "
83. for tb in self._tb_values.keys():
84. tag_values = self._tb_tags[tb]
85. sql += "IF NOT EXISTS " + tb + " USING meters TAGS " + tag_values + " "
86. try:
87. self._conn.execute(sql)
88. except BaseException as e:
89. self.log.error("Execute SQL: %s", sql)
90. raise e

view source code

Steps to Launch

Launch Sample Program in Python

1. Prerequisites

2. TDengine client driver has been installed

3. Python3 has been installed, the the version >= 3.8
4. TDengine Python connector taospy has been installed
5. Install faster-fifo to replace python builtin multiprocessing.Queue

1. pip3 install faster-fifo 3. Click the “Copy” in the above sample programs to copy fast_write_example.py 、 sql_writer.py and mockdatasource.py. 4. Execute the program

1. python3 fast_write_example.py <READ_TASK_COUNT> <WRITE_TASK_COUNT> <TABLE_COUNT> <QUEUE_SIZE> <MAX_BATCH_SIZE>

Below is the output of running on a server of 16 cores, 64GB memory and SSD hard disk.

1. root@vm85$ python3 fast_write_example.py 8 8 2. 2022-07-14 19:13:45,869 [root] - READ_TASK_COUNT=8, WRITE_TASK_COUNT=8, TABLE_COUNT=1000, QUEUE_SIZE=1000000, MAX_BATCH_SIZE=3000 3. 2022-07-14 19:13:48,882 [root] - WriteTask-0 started with pid 718347 4. 2022-07-14 19:13:48,883 [root] - WriteTask-1 started with pid 718348 5. 2022-07-14 19:13:48,884 [root] - WriteTask-2 started with pid 718349 6. 2022-07-14 19:13:48,884 [root] - WriteTask-3 started with pid 718350 7. 2022-07-14 19:13:48,885 [root] - WriteTask-4 started with pid 718351 8. 2022-07-14 19:13:48,885 [root] - WriteTask-5 started with pid 718352 9. 2022-07-14 19:13:48,886 [root] - WriteTask-6 started with pid 718353 10. 2022-07-14 19:13:48,886 [root] - WriteTask-7 started with pid 718354 11. 2022-07-14 19:13:48,887 [root] - ReadTask-0 started with pid 718355 12. 2022-07-14 19:13:48,888 [root] - ReadTask-1 started with pid 718356 13. 2022-07-14 19:13:48,889 [root] - ReadTask-2 started with pid 718357 14. 2022-07-14 19:13:48,889 [root] - ReadTask-3 started with pid 718358 15. 2022-07-14 19:13:48,890 [root] - ReadTask-4 started with pid 718359 16. 2022-07-14 19:13:48,891 [root] - ReadTask-5 started with pid 718361 17. 2022-07-14 19:13:48,892 [root] - ReadTask-6 started with pid 718364 18. 2022-07-14 19:13:48,893 [root] - ReadTask-7 started with pid 718365 19. 2022-07-14 19:13:56,042 [DataBaseMonitor] - count=6676310 speed=667631.0 20. 2022-07-14 19:14:06,196 [DataBaseMonitor] - count=20004310 speed=1332800.0 21. 2022-07-14 19:14:16,366 [DataBaseMonitor] - count=32290310 speed=1228600.0 22. 2022-07-14 19:14:26,527 [DataBaseMonitor] - count=44438310 speed=1214800.0 23. 2022-07-14 19:14:36,673 [DataBaseMonitor] - count=56608310 speed=1217000.0 24. 2022-07-14 19:14:46,834 [DataBaseMonitor] - count=68757310 speed=1214900.0 25. 2022-07-14 19:14:57,280 [DataBaseMonitor] - count=80992310 speed=1223500.0 26. 2022-07-14 19:15:07,689 [DataBaseMonitor] - count=93805310 speed=1281300.0 27. 2022-07-14 19:15:18,020 [DataBaseMonitor] - count=106111310 speed=1230600.0 28. 2022-07-14 19:15:28,356 [DataBaseMonitor] - count=118394310 speed=1228300.0 29. 2022-07-14 19:15:38,690 [DataBaseMonitor] - count=130742310 speed=1234800.0 30. 2022-07-14 19:15:49,000 [DataBaseMonitor] - count=143051310 speed=1230900.0 31. 2022-07-14 19:15:59,323 [DataBaseMonitor] - count=155276310 speed=1222500.0 32. 2022-07-14 19:16:09,649 [DataBaseMonitor] - count=167603310 speed=1232700.0 33. 2022-07-14 19:16:19,995 [DataBaseMonitor] - count=179976310 speed=1237300.0

High Performance Writing - 图2note

Don’t establish connection to TDengine in the parent process if using Python connector in multi-process way, otherwise all the connections in child processes are blocked always. This is a known issue.