Benchmarking

All of Skytable’s components are developed from scratch in-house, which makes it hard (if not impossible) to utilize other benchmarking suites. This is why all of Skytable’s release bundles come with thesky-bench benchmarking tool. It’s important to understand how the tool works to understand what you’re benchmarking and what the results mean.

The overall goal with the benchmark tool is to simulate how the database would perform in the real-world and reduce synthetic factors.

info

The ability to simulate different workloads is currently being worked on. This means that across releases, the default workload that the benchmark tool runs may vary. See the release notes to see if the benchmark workload has changed.

Using the benchmark tool​

You will need to select a workload from thebelow section on workloads. As workloads needroot access to the database for creating and removing spaces and tables, you will need to also provide theroot account password.

You may run workloads like this using an argument for the password:

sky-bench --workload <workload_name> --password <root password>

You can also use theSKYDB_PASSWORD environment variable if you do not want to use--password. You can tune the number of threads, connections, rows created and such to your liking to simulate the environment that you think matches your production setting:

• --connections <count>: Set the number of client connections (defaults to8 * number of logical CPUs)
• --threads <count>: Total number of threads to use. (defaults to the logical CPU count)
• --rowcount <count>: Set the number of unique rows to run workload sequences on
• --keysize <size>: Set the number of bytes to use for the primary key

See the help menu usingsky-bench --help to use additional configuration options.

Benchmark workloads​

Workloads are used to emulate various usage scenarios, for example by varying the read/write proportions, changing the distributions of keys and such. While we intend to add more workloads down the line, the default workload is currentlyuniform_std_v1.

uniform_std_v1​

This workload executes an uniform proportion of operations (hence called “uniform”) for unique rows. It does the following:

• Creates a spacedb
• Creates a modeldb.db with the following definition:create model db.db(k: binary, v: uint64)
• Now:
  • 1,000,000 unique rows are inserted usingINSERT (in parallel)
  • 1,000,000 of the unique rows that were created in the previous step are modified usingUPDATE
  • 1,000,000 of the unique rows that were created and modified earlier are fetched using aSELECT
  • 1,000,000 of the unique rows that were created are individually removed usingDELETE
  • Hence, a total of 4,000,000 queries are run

How to run:

sky-bench --workload 'uniform_std_v1'

tip

Now go ahead and run your own benchmarks to see the performance of Skytable for yourself. We know you’ll love it 🚀