Star Schema Benchmark

Star Schema Benchmark(SSB) is a lightweight performance test set in the data warehouse scenario. SSB provides a simplified star schema data based on TPC-H, which is mainly used to test the performance of multi-table JOIN query under star schema. In addition, the industry usually flattens SSB into a wide table model (Referred as: SSB flat) to test the performance of the query engine, refer to Clickhouse.

This document mainly introduces the performance of Doris on the SSB 100G test set.

Note 1: The standard test set including SSB usually has a large gap with the actual business scenario, and some tests will perform parameter tuning for the test set. Therefore, the test results of the standard test set can only reflect the performance of the database in a specific scenario. It is recommended that users use actual business data for further testing.

Note 2: The operations involved in this document are all performed in the Ubuntu Server 20.04 environment, and CentOS 7 as well.

With 13 queries on the SSB standard test data set, we conducted a comparison test based on Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 and Apache Doris 0.15.0 RC04 versions.

On the SSB flat wide table, the overall performance of Apache Doris 1.2.0-rc01 has been improved by nearly 4 times compared with Apache Doris 1.1.3, and nearly 10 times compared with Apache Doris 0.15.0 RC04.

On the SQL test with standard SSB, the overall performance of Apache Doris 1.2.0-rc01 has been improved by nearly 2 times compared with Apache Doris 1.1.3, and nearly 31 times compared with Apache Doris 0.15.0 RC04.

1. Hardware Environment

2. Software Environment

• Doris deployed 3BEs and 1FE;
• Kernel version: Linux version 5.4.0-96-generic (buildd@lgw01-amd64-051)
• OS version: Ubuntu Server 20.04 LTS 64-bit
• Doris software versions: Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 and Apache Doris 0.15.0 RC04
• JDK: openjdk version “11.0.14” 2022-01-18

3. Test Data Volume

4. Test Results

We use Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 and Apache Doris 0.15.0 RC04 for comparative testing. The test results are as follows:

Interpretation of Results

• The data set corresponding to the test results is scale 100, about 600 million.
• The test environment is configured as the user’s common configuration, with 4 cloud servers, 16-core 64G SSD, and 1 FE, 3 BEs deployment.
• We select the user’s common configuration test to reduce the cost of user selection and evaluation, but the entire test process will not consume so many hardware resources.

5. Standard SSB Test Results

Here we use Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 and Apache Doris 0.15.0 RC04 for comparative testing. In the test, we use Query Time（ms） as the main performance indicator. The test results are as follows:

Interpretation of Results

• The data set corresponding to the test results is scale 100, about 600 million.
• The test environment is configured as the user’s common configuration, with 4 cloud servers, 16-core 64G SSD, and 1 FE 3 BEs deployment.
• We select the user’s common configuration test to reduce the cost of user selection and evaluation, but the entire test process will not consume so many hardware resources.

6. Environment Preparation

Please first refer to the official documentation to install and deploy Apache Doris first to obtain a Doris cluster which is working well(including at least 1 FE 1 BE, 1 FE 3 BEs is recommended).

The scripts mentioned in the following documents are stored in the Apache Doris codebase: ssb-tools

7. Data Preparation

7.1 Download and Install the SSB Data Generation Tool.

Execute the following script to download and compile the ssb-dbgen tool.

1. sh build-ssb-dbgen.sh

After successful installation, the dbgen binary will be generated under the ssb-dbgen/ directory.

7.2 Generate SSB Test Set

Execute the following script to generate the SSB dataset:

1. sh gen-ssb-data.sh -s 100 -c 100

Note 1: Check the script help via sh gen-ssb-data.sh -h.

Note 2: The data will be generated under the ssb-data/ directory with the suffix .tbl. The total file size is about 60GB and may need a few minutes to an hour to generate.

Note 3: -s 100 indicates that the test set size factor is 100, -c 100 indicates that 100 concurrent threads generate the data of the lineorder table. The -c parameter also determines the number of files in the final lineorder table. The larger the parameter, the larger the number of files and the smaller each file.

With the -s 100 parameter, the resulting dataset size is:

7.3 Create Table

7.3.1 Prepare the doris-cluster.conf File.

Before import the script, you need to write the FE’s ip port and other information in the doris-cluster.conf file.

The file is located under ${DORIS_HOME}/tools/ssb-tools/conf/.

The content of the file includes FE’s ip, HTTP port, user name, password and the DB name of the data to be imported:

1. export FE_HOST="xxx"
2. export FE_HTTP_PORT="8030"
3. export FE_QUERY_PORT="9030"
4. export USER="root"
5. export PASSWORD='xxx'
6. export DB="ssb"

7.3.2 Execute the Following Script to Generate and Create the SSB Table:

1. sh create-ssb-tables.sh

Or copy the table creation statements in create-ssb-tables.sql and create-ssb-flat-table.sql and then execute them in the MySQL client.

The following is the lineorder_flat table build statement. Create the lineorder_flat table in the above create-ssb-flat-table.sh script, and perform the default number of buckets (48 buckets). You can delete this table and adjust the number of buckets according to your cluster scale node configuration, so as to obtain a better test result.

1. CREATE TABLE `lineorder_flat` (
2. `LO_ORDERDATE` date NOT NULL COMMENT "",
3. `LO_ORDERKEY` int(11) NOT NULL COMMENT "",
4. `LO_LINENUMBER` tinyint(4) NOT NULL COMMENT "",
5. `LO_CUSTKEY` int(11) NOT NULL COMMENT "",
6. `LO_PARTKEY` int(11) NOT NULL COMMENT "",
7. `LO_SUPPKEY` int(11) NOT NULL COMMENT "",
8. `LO_ORDERPRIORITY` varchar(100) NOT NULL COMMENT "",
9. `LO_SHIPPRIORITY` tinyint(4) NOT NULL COMMENT "",
10. `LO_QUANTITY` tinyint(4) NOT NULL COMMENT "",
11. `LO_EXTENDEDPRICE` int(11) NOT NULL COMMENT "",
12. `LO_ORDTOTALPRICE` int(11) NOT NULL COMMENT "",
13. `LO_DISCOUNT` tinyint(4) NOT NULL COMMENT "",
14. `LO_REVENUE` int(11) NOT NULL COMMENT "",
15. `LO_SUPPLYCOST` int(11) NOT NULL COMMENT "",
16. `LO_TAX` tinyint(4) NOT NULL COMMENT "",
17. `LO_COMMITDATE` date NOT NULL COMMENT "",
18. `LO_SHIPMODE` varchar(100) NOT NULL COMMENT "",
19. `C_NAME` varchar(100) NOT NULL COMMENT "",
20. `C_ADDRESS` varchar(100) NOT NULL COMMENT "",
21. `C_CITY` varchar(100) NOT NULL COMMENT "",
22. `C_NATION` varchar(100) NOT NULL COMMENT "",
23. `C_REGION` varchar(100) NOT NULL COMMENT "",
24. `C_PHONE` varchar(100) NOT NULL COMMENT "",
25. `C_MKTSEGMENT` varchar(100) NOT NULL COMMENT "",
26. `S_NAME` varchar(100) NOT NULL COMMENT "",
27. `S_ADDRESS` varchar(100) NOT NULL COMMENT "",
28. `S_CITY` varchar(100) NOT NULL COMMENT "",
29. `S_NATION` varchar(100) NOT NULL COMMENT "",
30. `S_REGION` varchar(100) NOT NULL COMMENT "",
31. `S_PHONE` varchar(100) NOT NULL COMMENT "",
32. `P_NAME` varchar(100) NOT NULL COMMENT "",
33. `P_MFGR` varchar(100) NOT NULL COMMENT "",
34. `P_CATEGORY` varchar(100) NOT NULL COMMENT "",
35. `P_BRAND` varchar(100) NOT NULL COMMENT "",
36. `P_COLOR` varchar(100) NOT NULL COMMENT "",
37. `P_TYPE` varchar(100) NOT NULL COMMENT "",
38. `P_SIZE` tinyint(4) NOT NULL COMMENT "",
39. `P_CONTAINER` varchar(100) NOT NULL COMMENT ""
40. ) ENGINE=OLAP
41. DUPLICATE KEY(`LO_ORDERDATE`, `LO_ORDERKEY`)
42. COMMENT "OLAP"
43. PARTITION BY RANGE(`LO_ORDERDATE`)
44. (PARTITION p1 VALUES [('0000-01-01'), ('1993-01-01')),
45. PARTITION p2 VALUES [('1993-01-01'), ('1994-01-01')),
46. PARTITION p3 VALUES [('1994-01-01'), ('1995-01-01')),
47. PARTITION p4 VALUES [('1995-01-01'), ('1996-01-01')),
48. PARTITION p5 VALUES [('1996-01-01'), ('1997-01-01')),
49. PARTITION p6 VALUES [('1997-01-01'), ('1998-01-01')),
50. PARTITION p7 VALUES [('1998-01-01'), ('1999-01-01')))
51. DISTRIBUTED BY HASH(`LO_ORDERKEY`) BUCKETS 48
52. PROPERTIES (
53. "replication_num" = "1",
54. "colocate_with" = "groupxx1",
55. "in_memory" = "false",
56. "storage_format" = "DEFAULT"
57. );

7.4 Import data

We use the following command to complete all data import of SSB test set and SSB FLAT wide table data synthesis and then import into the table.

1. sh bin/load-ssb-data.sh -c 10

-c 5 means start 10 concurrent threads to import (5 by default). In the case of a single BE node, the lineorder data generated by sh gen-ssb-data.sh -s 100 -c 100 will also generate the data of the ssb-flat table in the end. If more threads are enabled, the import speed can be accelerated. But it will cost extra memory.

Notes.

1. To get faster import speed, you can add flush_thread_num_per_store=5 in be.conf and then restart BE. This configuration indicates the number of disk writing threads for each data directory, 2 by default. Larger data can improve write data throughput, but may increase IO Util. (Reference value: 1 mechanical disk, with 2 by default, the IO Util during the import process is about 12%. When it is set to 5, the IO Util is about 26%. If it is an SSD disk, it is almost 0%) .
2. The flat table data is imported by ‘INSERT INTO … SELECT … ‘.

7.5 Checking Imported data

1. select count(*) from part;
2. select count(*) from customer;
3. select count(*) from supplier;
4. select count(*) from date;
5. select count(*) from lineorder;
6. select count(*) from lineorder_flat;

The amount of data should be consistent with the number of rows of generated data.

7.6 Query Test

• SSB-Flat Query Statement: ssb-flat-queries
• Standard SSB Queries: ssb-queries

7.6.1 SSB FLAT Test for SQL

1. --Q1.1
2. SELECT SUM(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
3. FROM lineorder_flat
4. WHERE  LO_ORDERDATE >= 19930101  AND LO_ORDERDATE <= 19931231 AND LO_DISCOUNT BETWEEN 1 AND 3  AND LO_QUANTITY < 25;
5. --Q1.2
6. SELECT SUM(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
7. FROM lineorder_flat
8. WHERE LO_ORDERDATE >= 19940101 AND LO_ORDERDATE <= 19940131  AND LO_DISCOUNT BETWEEN 4 AND 6 AND LO_QUANTITY BETWEEN 26 AND 35;

10. --Q1.3
11. SELECT SUM(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
12. FROM lineorder_flat
13. WHERE  weekofyear(LO_ORDERDATE) = 6 AND LO_ORDERDATE >= 19940101  AND LO_ORDERDATE <= 19941231 AND LO_DISCOUNT BETWEEN 5 AND 7  AND LO_QUANTITY BETWEEN 26 AND 35;

15. --Q2.1
16. SELECT SUM(LO_REVENUE), (LO_ORDERDATE DIV 10000) AS YEAR, P_BRAND
17. FROM lineorder_flat WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA'
18. GROUP BY YEAR, P_BRAND
19. ORDER BY YEAR, P_BRAND;

21. --Q2.2
22. SELECT  SUM(LO_REVENUE), (LO_ORDERDATE DIV 10000) AS YEAR, P_BRAND
23. FROM lineorder_flat
24. WHERE P_BRAND >= 'MFGR#2221' AND P_BRAND <= 'MFGR#2228'  AND S_REGION = 'ASIA'
25. GROUP BY YEAR, P_BRAND
26. ORDER BY YEAR, P_BRAND;

28. --Q2.3
29. SELECT SUM(LO_REVENUE), (LO_ORDERDATE DIV 10000) AS YEAR, P_BRAND
30. FROM lineorder_flat
31. WHERE P_BRAND = 'MFGR#2239' AND S_REGION = 'EUROPE'
32. GROUP BY YEAR, P_BRAND
33. ORDER BY YEAR, P_BRAND;

35. --Q3.1
36. SELECT C_NATION, S_NATION, (LO_ORDERDATE DIV 10000) AS YEAR, SUM(LO_REVENUE) AS revenue
37. FROM lineorder_flat
38. WHERE C_REGION = 'ASIA' AND S_REGION = 'ASIA' AND LO_ORDERDATE >= 19920101  AND LO_ORDERDATE <= 19971231
39. GROUP BY C_NATION, S_NATION, YEAR
40. ORDER BY YEAR ASC, revenue DESC;

42. --Q3.2
43. SELECT C_CITY, S_CITY, (LO_ORDERDATE DIV 10000) AS YEAR, SUM(LO_REVENUE) AS revenue
44. FROM lineorder_flat
45. WHERE C_NATION = 'UNITED STATES' AND S_NATION = 'UNITED STATES' AND LO_ORDERDATE >= 19920101 AND LO_ORDERDATE <= 19971231
46. GROUP BY C_CITY, S_CITY, YEAR
47. ORDER BY YEAR ASC, revenue DESC;

49. --Q3.3
50. SELECT C_CITY, S_CITY, (LO_ORDERDATE DIV 10000) AS YEAR, SUM(LO_REVENUE) AS revenue
51. FROM lineorder_flat
52. WHERE C_CITY IN ('UNITED KI1', 'UNITED KI5') AND S_CITY IN ('UNITED KI1', 'UNITED KI5') AND LO_ORDERDATE >= 19920101 AND LO_ORDERDATE <= 19971231
53. GROUP BY C_CITY, S_CITY, YEAR
54. ORDER BY YEAR ASC, revenue DESC;

56. --Q3.4
57. SELECT C_CITY, S_CITY, (LO_ORDERDATE DIV 10000) AS YEAR, SUM(LO_REVENUE) AS revenue
58. FROM lineorder_flat
59. WHERE C_CITY IN ('UNITED KI1', 'UNITED KI5') AND S_CITY IN ('UNITED KI1', 'UNITED KI5') AND LO_ORDERDATE >= 19971201  AND LO_ORDERDATE <= 19971231
60. GROUP BY C_CITY, S_CITY, YEAR
61. ORDER BY YEAR ASC, revenue DESC;

63. --Q4.1
64. SELECT (LO_ORDERDATE DIV 10000) AS YEAR, C_NATION, SUM(LO_REVENUE - LO_SUPPLYCOST) AS profit
65. FROM lineorder_flat
66. WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND P_MFGR IN ('MFGR#1', 'MFGR#2')
67. GROUP BY YEAR, C_NATION
68. ORDER BY YEAR ASC, C_NATION ASC;

70. --Q4.2
71. SELECT (LO_ORDERDATE DIV 10000) AS YEAR,S_NATION, P_CATEGORY, SUM(LO_REVENUE - LO_SUPPLYCOST) AS profit
72. FROM lineorder_flat
73. WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND LO_ORDERDATE >= 19970101 AND LO_ORDERDATE <= 19981231 AND P_MFGR IN ('MFGR#1', 'MFGR#2')
74. GROUP BY YEAR, S_NATION, P_CATEGORY
75. ORDER BY YEAR ASC, S_NATION ASC, P_CATEGORY ASC;

77. --Q4.3
78. SELECT (LO_ORDERDATE DIV 10000) AS YEAR, S_CITY, P_BRAND, SUM(LO_REVENUE - LO_SUPPLYCOST) AS profit
79. FROM lineorder_flat
80. WHERE S_NATION = 'UNITED STATES' AND LO_ORDERDATE >= 19970101 AND LO_ORDERDATE <= 19981231 AND P_CATEGORY = 'MFGR#14'
81. GROUP BY YEAR, S_CITY, P_BRAND
82. ORDER BY YEAR ASC, S_CITY ASC, P_BRAND ASC;

7.6.2 SSB Standard Test for SQL

1. --Q1.1
2. SELECT SUM(lo_extendedprice * lo_discount) AS REVENUE
3. FROM lineorder, dates
4. WHERE
5. lo_orderdate = d_datekey
6. AND d_year = 1993
7. AND lo_discount BETWEEN 1 AND 3
8. AND lo_quantity < 25;
9. --Q1.2
10. SELECT SUM(lo_extendedprice * lo_discount) AS REVENUE
11. FROM lineorder, dates
12. WHERE
13. lo_orderdate = d_datekey
14. AND d_yearmonth = 'Jan1994'
15. AND lo_discount BETWEEN 4 AND 6
16. AND lo_quantity BETWEEN 26 AND 35;

18. --Q1.3
19. SELECT
20. SUM(lo_extendedprice * lo_discount) AS REVENUE
21. FROM lineorder, dates
22. WHERE
23. lo_orderdate = d_datekey
24. AND d_weeknuminyear = 6
25. AND d_year = 1994
26. AND lo_discount BETWEEN 5 AND 7
27. AND lo_quantity BETWEEN 26 AND 35;

29. --Q2.1
30. SELECT SUM(lo_revenue), d_year, p_brand
31. FROM lineorder, dates, part, supplier
32. WHERE
33. lo_orderdate = d_datekey
34. AND lo_partkey = p_partkey
35. AND lo_suppkey = s_suppkey
36. AND p_category = 'MFGR#12'
37. AND s_region = 'AMERICA'
38. GROUP BY d_year, p_brand
39. ORDER BY p_brand;

41. --Q2.2
42. SELECT SUM(lo_revenue), d_year, p_brand
43. FROM lineorder, dates, part, supplier
44. WHERE
45. lo_orderdate = d_datekey
46. AND lo_partkey = p_partkey
47. AND lo_suppkey = s_suppkey
48. AND p_brand BETWEEN 'MFGR#2221' AND 'MFGR#2228'
49. AND s_region = 'ASIA'
50. GROUP BY d_year, p_brand
51. ORDER BY d_year, p_brand;

53. --Q2.3
54. SELECT SUM(lo_revenue), d_year, p_brand
55. FROM lineorder, dates, part, supplier
56. WHERE
57. lo_orderdate = d_datekey
58. AND lo_partkey = p_partkey
59. AND lo_suppkey = s_suppkey
60. AND p_brand = 'MFGR#2239'
61. AND s_region = 'EUROPE'
62. GROUP BY d_year, p_brand
63. ORDER BY d_year, p_brand;

65. --Q3.1
66. SELECT
67. c_nation,
68. s_nation,
69. d_year,
70. SUM(lo_revenue) AS REVENUE
71. FROM customer, lineorder, supplier, dates
72. WHERE
73. lo_custkey = c_custkey
74. AND lo_suppkey = s_suppkey
75. AND lo_orderdate = d_datekey
76. AND c_region = 'ASIA'
77. AND s_region = 'ASIA'
78. AND d_year >= 1992
79. AND d_year <= 1997
80. GROUP BY c_nation, s_nation, d_year
81. ORDER BY d_year ASC, REVENUE DESC;

83. --Q3.2
84. SELECT
85. c_city,
86. s_city,
87. d_year,
88. SUM(lo_revenue) AS REVENUE
89. FROM customer, lineorder, supplier, dates
90. WHERE
91. lo_custkey = c_custkey
92. AND lo_suppkey = s_suppkey
93. AND lo_orderdate = d_datekey
94. AND c_nation = 'UNITED STATES'
95. AND s_nation = 'UNITED STATES'
96. AND d_year >= 1992
97. AND d_year <= 1997
98. GROUP BY c_city, s_city, d_year
99. ORDER BY d_year ASC, REVENUE DESC;

101. --Q3.3
102. SELECT
103. c_city,
104. s_city,
105. d_year,
106. SUM(lo_revenue) AS REVENUE
107. FROM customer, lineorder, supplier, dates
108. WHERE
109. lo_custkey = c_custkey
110. AND lo_suppkey = s_suppkey
111. AND lo_orderdate = d_datekey
112. AND (
113. c_city = 'UNITED KI1'
114. OR c_city = 'UNITED KI5'
115. )
116. AND (
117. s_city = 'UNITED KI1'
118. OR s_city = 'UNITED KI5'
119. )
120. AND d_year >= 1992
121. AND d_year <= 1997
122. GROUP BY c_city, s_city, d_year
123. ORDER BY d_year ASC, REVENUE DESC;

125. --Q3.4
126. SELECT
127. c_city,
128. s_city,
129. d_year,
130. SUM(lo_revenue) AS REVENUE
131. FROM customer, lineorder, supplier, dates
132. WHERE
133. lo_custkey = c_custkey
134. AND lo_suppkey = s_suppkey
135. AND lo_orderdate = d_datekey
136. AND (
137. c_city = 'UNITED KI1'
138. OR c_city = 'UNITED KI5'
139. )
140. AND (
141. s_city = 'UNITED KI1'
142. OR s_city = 'UNITED KI5'
143. )
144. AND d_yearmonth = 'Dec1997'
145. GROUP BY c_city, s_city, d_year
146. ORDER BY d_year ASC, REVENUE DESC;

148. --Q4.1
149. SELECT /*+SET_VAR(parallel_fragment_exec_instance_num=4, enable_vectorized_engine=true, batch_size=4096, enable_cost_based_join_reorder=true, enable_projection=true) */
150. d_year,
151. c_nation,
152. SUM(lo_revenue - lo_supplycost) AS PROFIT
153. FROM dates, customer, supplier, part, lineorder
154. WHERE
155. lo_custkey = c_custkey
156. AND lo_suppkey = s_suppkey
157. AND lo_partkey = p_partkey
158. AND lo_orderdate = d_datekey
159. AND c_region = 'AMERICA'
160. AND s_region = 'AMERICA'
161. AND (
162. p_mfgr = 'MFGR#1'
163. OR p_mfgr = 'MFGR#2'
164. )
165. GROUP BY d_year, c_nation
166. ORDER BY d_year, c_nation;

168. --Q4.2
169. SELECT /*+SET_VAR(parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, enable_cost_based_join_reorder=true, enable_projection=true) */
170. d_year,
171. s_nation,
172. p_category,
173. SUM(lo_revenue - lo_supplycost) AS PROFIT
174. FROM dates, customer, supplier, part, lineorder
175. WHERE
176. lo_custkey = c_custkey
177. AND lo_suppkey = s_suppkey
178. AND lo_partkey = p_partkey
179. AND lo_orderdate = d_datekey
180. AND c_region = 'AMERICA'
181. AND s_region = 'AMERICA'
182. AND (
183. d_year = 1997
184. OR d_year = 1998
185. )
186. AND (
187. p_mfgr = 'MFGR#1'
188. OR p_mfgr = 'MFGR#2'
189. )
190. GROUP BY d_year, s_nation, p_category
191. ORDER BY d_year, s_nation, p_category;

193. --Q4.3
194. SELECT /*+SET_VAR(parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, enable_cost_based_join_reorder=true, enable_projection=true) */
195. d_year,
196. s_city,
197. p_brand,
198. SUM(lo_revenue - lo_supplycost) AS PROFIT
199. FROM dates, customer, supplier, part, lineorder
200. WHERE
201. lo_custkey = c_custkey
202. AND lo_suppkey = s_suppkey
203. AND lo_partkey = p_partkey
204. AND lo_orderdate = d_datekey
205. AND s_nation = 'UNITED STATES'
206. AND (
207. d_year = 1997
208. OR d_year = 1998
209. )
210. AND p_category = 'MFGR#14'
211. GROUP BY d_year, s_city, p_brand
212. ORDER BY d_year, s_city, p_brand;