Development - Testing - 《Apache Cassandra v4.1 Documentation》

Java tests
Python tests
Performance Testing

Java tests
JUnit tests
Stress and FQLTool tests
JVM distributed tests
Upgrade tests
Running multiple tests
Running coverage analysis
Micro-benchmarks
Python tests
Docker
Setup Docker
Pull the Docker image
Start the container
Setup Python environment
CQLSH tests
Running selected tests
Python distributed tests
Run the tests - quick examples
Setting up PyTest
Running tests with specific configuration
Listing the tests
Filtering tests
Based on configuration
Based on resource usage
Based on the test type
Filtering examples
Upgrade tests
Upgrade manifest
Running upgrades with local distribution
Logging
Performance Testing Creating tests is one of the most important and also most difficult parts of developing Cassandra. There are different ways to test your code depending on what you’re working on. Cassandra tests can be divided into three main categories, based on the way how they are executed:
** - tests implemented in Java and being a part of the Cassandra project. You can distinguish the following subcategories there:
** - consists of unit tests, single-node integration tests and some tool tests; those tests may run a server with limited functionality in the same JVM as the test code
** - integrated tests against one or multiple nodes, each running in their own classloader; also contains upgrade tests
** - micro-benchmarks implemented with JMH framework
** - CQLSH tests are Python tests written with the pytest test framework. They verify the CQLSH client that can be found in the bin directory. They aim at verifying CQLSH specific behavior like output formatting, autocompletion, parsing, etc).
** - Python distributed tests are implemented on top of the PyTest framework and located outside the main Cassandra project in the separate repository apache/cassandra-dtest. They test Cassandra via CCM verifying operation results, logs, and cluster state. Python Distributed tests are Cassandra version agnostic. They include upgrade tests. apache/cassandra-dtest how to do it. here. Continuous Integration systems. If you are not a committer, and don’t have access to a premium CircleCI plan, ask one of the committers to test your patch on the project’s ci-cassandra.apache.org.
Java tests
test/unit directory. Ideally, you’ll create a unit test for your implementation that exclusively covers the class you created (the unit under test). Unfortunately, this is not always possible, because Cassandra doesn’t have a very mock friendly code base. Often you’ll find yourself in a situation where you have to use the embedded Cassandra instance to interact with your test. If you want to use CQL in your test, you can extend CQLTester and use some convenient helper methods, as shown here:
```
@Testpublic void testBatchAndList() throws Throwable{   createTable("CREATE TABLE %s (k int PRIMARY KEY, l list<int>)");   execute("BEGIN BATCH " +           "UPDATE %1$s SET l = l +[ 1 ] WHERE k = 0; " +           "UPDATE %1$s SET l = l + [ 2 ] WHERE k = 0; " +           "UPDATE %1$s SET l = l + [ 3 ] WHERE k = 0; " +           "APPLY BATCH");   assertRows(execute("SELECT l FROM %s WHERE k = 0"),              row(list(1, 2, 3)));}
```
JUnit tests
To run the unit tests:
```
ant test
```
test/unit). It would take about an hour or more to finish. To run the specific test class or even a method, use the following command:
```
ant testsome -Dtest.name=<TestClassName> -Dtest.methods=<testMethodName>
```
test.name property is for either a simple or fully qualified class name
test.methods property is optional; if not specified, all test cases from the specified class are executed. Though, you can also specify multiple methods separating them by comma ant jar to build the distribution artifacts. When the test runs some tool as an external process, the tool expects Cassandra artifacts to be in the build directory. test/unit directory. There are, however, some other test categories that have tests in individual directories:
test/burn - to run them, call ant test-burn or ant burn-testsome; ant burn-test-jar builds a self-contained jar for e.g. remote execution; not currently used for running burn tests in our scripts. ant burn-test-jar exists only on 4.0+ branches
test/long - to run them, call ant long-test or ant long-testsome
test/memory - to run them, call ant test-memory
test/microbench discussed in Micro-benchmarks
test/distributed discussed in JVM distributed tests
Stress and FQLTool tests
Stress and FQLTool are separate modules located under the tools directory in the Cassandra project. They have their own source code and unit tests. To run the tests for those tools, first, build jar artifacts for them by calling:
```
ant fqltool-build fqltool-build-testant stress-build stress-build-test
```
Then you can execute the tests with either one of the commands:
```
ant fqltool-testant stress-testand stress-test-some
```
or using your IDE.
JVM distributed tests
apache/cassandra-in-jvm-dtest-api) for that purpose. Those tests are intended to test features that require more started nodes or verify specific behaviors when the nodes get restarted, including upgrading them from one version to another. The tests are located at the test/distributed directory of the Cassandra project; however, only org.apache.cassandra.distributed.test and org.apache.cassandra.upgrade packages contain the actual tests. The rest of the files are various utilities related to the distributed test framework. ant test-jvm-dtest command runs all the distributed JVM tests. It is not very useful; thus, there is also ant test-jvm-dtest-some, which allows specifying test class and test name in the similar way as you could do that for the ant testsome command, for example:
```
ant test-jvm-dtest-some -Dtest.name=org.apache.cassandra.distributed.test.SchemaTestant test-jvm-dtest-some -Dtest.name=org.apache.cassandra.distributed.test.SchemaTest -Dtest.methods=readRepair
```
Distributed tests can also be run using IDE (in fact, you can even debug them).
Upgrade tests
JVM upgrade tests can be run precisely in the same way as any other JVM distributed tests. However, running them requires some preparation - for example, if a test verifies the upgrade from Cassandra 3.0 and Cassandra 3.11 to the current version (say Cassandra 4.0), you need to have prepared dtest uber JARs for all involved versions. To do this:

Check out Cassandra 3.0 based branch you want to test the upgrade from into some other directory
ant dtest-jar command
build/dtest-3.0.x.jar to the build directory of your target Cassandra project
Repeat the procedure for Cassandra 3.11
Once you have dtest jars of all the involved versions for the upgrade test, you can finally execute the test using your favorite method, say:
```
ant test-jvm-dtest-some -Dtest.name=org.apache.cassandra.distributed.upgrade.MixedModeReadTest
```
Running multiple tests
testlist.txt, and put it into your project directory. Here is an example of that file:
```
org/apache/cassandra/db/ReadCommandTest.javaorg/apache/cassandra/db/ReadCommandVerbHandlerTest.java
```
ant testclasslist, which uses the text file to run the listed tests. Note that, by default, it applies to the tests under the test/unit directory and takes the testlist.txt file, but this behavior can be modified by providing additional parameters:
```
ant testclasslist -Dtest.classlistprefix=<category> -Dtest.classlistfile=<class list file>
```
distributed-tests-set.txt file (paths to test classes relative to test/distributed directory), you can do that by calling:
```
ant testclasslist -Dtest.classlistprefix=distributed -Dtest.classlistfile=distributed-tests-set.txt
```
Running coverage analysis
Coverage reports from the executed JVM tests can be obtained in two ways - through IDE - for example, IntelliJ supports running tests with coverage analysis (another run button next to the one for running in debug mode). codecoverage. Basically, it works for all the ways mentioned above of running JVM tests - the only difference is that instead of specifying the target directly, you pass it as a property called taskname. For example - given the original test command is:
```
ant testsome -Dtest.name=org.apache.cassandra.utils.concurrent.AccumulatorTest
```
to run it with coverage analysis, do:
```
ant codecoverage -Dtaskname=testsome -Dtest.name=org.apache.cassandra.utils.concurrent.AccumulatorTest
```
test, testsome, test-long, etc., even testclasslist. You can find the coverage report in build/jacoco (index.html is the entry point for the HTML version, but there are also XML and CSV reports). ant jacoco-cleanup.
Micro-benchmarks
ant command:
```
ant build-jmh
```
test/microbench directory) or the tests matching the name specified by the benchmark.name property when executing the ant microbench command. Whether you run all benchmarks or just a selected one, only classes under the microbench package are selected. The class selection pattern is actually .*microbench.*${benchmark.name}. For example, in order to run org.apache.cassandra.test.microbench.ChecksumBench, execute:
```
ant microbench -Dbenchmark.name=ChecksumBench
```
ant microbench command runs the benchmarks with default parameters as defined in the build.xml file (see the microbench target definition). If you want to run JMH with custom parameters, consider using the test/bin/jmh script. In addition to allowing you to customize JMH options, it also sets up the environment and JVM options by running Cassandra init script (conf/cassandra-env.sh). Therefore, it lets the environment for running the tests to be more similar to the production environment. For example:
```
test/bin/jmh -gc true org.apache.cassandra.test.microbench.CompactionBench.compactTest
```
test/bin/jmh -l or test/bin/jmh -lp (also showing the default parameters). The list of all options can be shown by running test/bin/jmh -h
Python tests
Docker
Cassandra CI.
Setup Docker
If you are on Linux, you need to install Docker using the system package manager. Docker Desktop or some other approach.
Pull the Docker image
this repository. You can use either docker/testing/ubuntu2004_j11.docker or docker/testing/ubuntu2004_j11_w_dependencies.docker The second choice has prefetched dependencies for building each main Cassandra branch. Those images can be either built locally (as per instructions in the GitHub repo) or pulled from the Docker Hub - see here. First, pull the image from Docker Hub (it will either fetch or update the image you previously fetched):
```
docker pull apache/cassandra-testing-ubuntu2004-java11-w-dependencies
```
Start the container
```
docker run -di -m 8G --cpus 4 \--mount type=bind,source=/path/to/cassandra/project,target=/home/cassandra/cassandra \--mount type=bind,source=/path/to/cassandra-dtest,target=/home/cassandra/cassandra-dtest \--name test \apache/cassandra-testing-ubuntu2004-java11-w-dependencies \dumb-init bash
```
docker exec command:
```
docker exec -it `docker container ls -f name=test -q` bash
```
Setup Python environment
here for details) with all the required dependencies is good to be set up. If you are familiar with the Python ecosystem, you know what it is all about. Otherwise, follow the instructions; it should be enough to run the tests. For Python distributed tests do:
```
cd /home/cassandra/cassandra-dtestvirtualenv --python=python3 --clear --always-copy ../dtest-venvsource ../dtest-venv/bin/activateCASS_DRIVER_NO_CYTHON=1 pip install -r requirements.txt
```
For CQLSH tests, replace some paths:
```
cd /home/cassandra/cassandra/pylibvirtualenv --python=python3 --clear --always-copy ../../cqlsh-venvsource ../../cqlsh-venv/bin/activateCASS_DRIVER_NO_CYTHON=1 pip install -r requirements.txt
```
virtualenv based environment and point to bin/python under the created dtest-venv directory (or cqlsh-venv, or whichever name you have chosen). Whether you want to play with Python distributed tests or CQLSH tests, you need to select the right virtual environment. Remember to switch to the one you want:
```
deactivatesource /home/cassandra/dtest-venv/bin/activate
```
or
```
deactivatesource /home/cassandra/cqlsh-venv/bin/activate
```
CQLSH tests
pylib/cqlshlib/test directory. There is a helper script that runs the tests for you. In particular, it builds the Cassandra project, creates a virtual environment, runs the CCM cluster, executes the tests, and eventually removes the cluster. You find the script in the pylib directory. The only argument it requires is the Cassandra project directory:
```
cassandra@b69a382da7cd:~/cassandra/pylib$ ./cassandra-cqlsh-tests.sh /home/cassandra/cassandra
```
README for further information.
Running selected tests
You may run all test tests from the selected file by passing that file as an argument:
```
~/cassandra/pylib/cqlshlib$ pytest test/test_constants.py
```
To run a specific test case, you need to specify the module, class name, and the test name, for example:
```
~/cassandra/pylib/cqlshlib$ pytest cqlshlib.test.test_cqlsh_output:TestCqlshOutput.test_boolean_output
```
Python distributed tests
dtest (Cassandra distributed test). These dtests automatically setup Cassandra clusters with certain configurations and simulate use cases you want to test. http://www.datastax.com/dev/blog/how-to-write-a-dtest\[How to Write a Dtest]“. Looking at existing, recently updated tests in the project is another good activity. New tests must follow certain style conventions that are checked before contributions are accepted. In contrast to Cassandra, dtest issues and pull requests are managed on github, therefore you should make sure to link any created dtests in your Cassandra ticket and also refer to the ticket number in your dtest PR. Creating a good dtest can be tough, but it should not prevent you from submitting patches! Please ask in the corresponding JIRA ticket how to write a good dtest for the patch. In most cases a reviewer or committer will able to support you, and in some cases they may offer to write a dtest for you.
Run the tests - quick examples
Setup Python environment section for details). Tests are implemented with the PyTest framework, so you use the pytest command to run them. Let’s run some tests:
```
pytest --cassandra-dir=/home/cassandra/cassandra schema_metadata_test.py::TestSchemaMetadata::test_clustering_order
```
test_clustering_order test case from TestSchemaMetadata class, located in the schema_metadata_test.py file. You may also provide the file and class to run all test cases from that class:
```
pytest --cassandra-dir=/home/cassandra/cassandra schema_metadata_test.py::TestSchemaMetadata
```
or just the file name to run all test cases from all classes defined in that file.
```
pytest --cassandra-dir=/home/cassandra/cassandra schema_metadata_test.py
```
You may also specify more individual targets:
```
pytest --cassandra-dir=/home/cassandra/cassandra schema_metadata_test.py::TestSchemaMetadata::test_basic_table_datatype  schema_metadata_test.py::TestSchemaMetadata::test_udf
```
here You probably noticed that --cassandra-dir=/home/cassandra/cassandra is constantly added to the command line. It is one of the cassandra-dtest custom arguments - the mandatory one - unless it is defined, you cannot run any Cassandra dtest.
Setting up PyTest
--help. You see tons of possible parameters - some of them are native PyTest options, and some come from Cassandra DTest. When you look carefully at the help note, you notice that some commonly used options, usually fixed for all the invocations, can be put into the pytest.ini file. In particular, it is quite practical to define the following:
```
cassandra_dir = /home/cassandra/cassandralog_cli = Truelog_cli_level = DEBUG
```
--cassandra-dir param each time you run a test. The other two options set up console logging - remove them if you want logs stored only in log files.
Running tests with specific configuration
There are a couple of options to enforce exact test configuration (their names are quite self-explanatory):

--use-vnodes
--num-token=xxx - enables the support of virtual nodes with a certain number of tokens
--use-off-heap-memtables - use off-heap memtables instead of the default heap-based

—data-dir-count-per-instance=xxx - the number of data directories configured per each instanceJVM_EXTRA_OPTS` environment variable before running the test.

Listing the tests

--collect-only to the pytest command. That additional -q option will print the results in the same format as you would pass the test name to the pytest command:

pytest --collect-only -q

lists all the tests pytest would run if no particular test is specified. Similarly, to list test cases in some class, do:

$ pytest --collect-only -q schema_metadata_test.py::TestSchemaMetadataschema_metadata_test.py::TestSchemaMetadata::test_creating_and_dropping_keyspaceschema_metadata_test.py::TestSchemaMetadata::test_creating_and_dropping_tableschema_metadata_test.py::TestSchemaMetadata::test_creating_and_dropping_table_with_2ary_indexesschema_metadata_test.py::TestSchemaMetadata::test_creating_and_dropping_user_typesschema_metadata_test.py::TestSchemaMetadata::test_creating_and_dropping_udfschema_metadata_test.py::TestSchemaMetadata::test_creating_and_dropping_udaschema_metadata_test.py::TestSchemaMetadata::test_basic_table_datatypeschema_metadata_test.py::TestSchemaMetadata::test_collection_table_datatypeschema_metadata_test.py::TestSchemaMetadata::test_clustering_orderschema_metadata_test.py::TestSchemaMetadata::test_compact_storageschema_metadata_test.py::TestSchemaMetadata::test_compact_storage_compositeschema_metadata_test.py::TestSchemaMetadata::test_nondefault_table_settingsschema_metadata_test.py::TestSchemaMetadata::test_indexesschema_metadata_test.py::TestSchemaMetadata::test_durable_writesschema_metadata_test.py::TestSchemaMetadata::test_static_columnschema_metadata_test.py::TestSchemaMetadata::test_udt_tableschema_metadata_test.py::TestSchemaMetadata::test_udfschema_metadata_test.py::TestSchemaMetadata::test_uda

You can copy/paste the selected test case to the pytest command to run it.

Filtering tests

Based on configuration

Most tests run with any configuration, but a subset of tests (test cases) only run if a specific configuration is used. In particular, there are tests annotated with:

@pytest.mark.vnodes - the test is only invoked when the support of virtual nodes is enabled
@pytest.mark.no_vnodes - the test is only invoked when the support of virtual nodes is disabled
@pytest.mark.no_offheap_memtables - the test is only invoked if off-heap memtables are not used vnodes is obviously mutually exclusive. If a test is marked to run only with vnodes, it does not run when vnodes is disabled; similarly, when a test is marked to run only without vnodes, it does not run when vnodes is enabled - therefore, there are always some tests which would not run with a single configuration.
Based on resource usage
There are also tests marked with: @pytest.mark.resource_intensive which means that the test requires more resources than a regular test because it usually starts a cluster of several nodes. The meaning of resource-intensive is hardcoded to 32GB of available memory, and unless your machine or docker container has at least that amount of RAM, such test is skipped. There are a couple of arguments that allow for some control of that automatic exclusion:
--force-resource-intensive-tests - forces the execution of tests marked as resource_intensive, regardless of whether there is enough memory available or not
--only-resource-intensive-tests - only run tests marked as resource_intensive - it makes all the tests without resource_intensive annotation to be filtered out; technically, it is equivalent to passing native PyTest argument: -m resource_intensive

--skip-resource-intensive-tests - skip all tests marked as resource_intensive - it is the opposite argument to the previous one, and it is equivalent to the PyTest native argument: -m 'not resource_intensive'

Based on the test type

Upgrade tests are marked with: @pytest.mark.upgrade_test -m 'not upgrade_test'), and you have to add some extra options to run them: --execute-upgrade-tests - enables execution of upgrade tests along with other tests - when this option is added, the upgrade tests are not filtered out --execute-upgrade-tests-only - execute only upgrade tests and filter out all other tests which do not have @pytest.mark.upgrade_test annotation (just like running PyTest with -m 'upgrade_test')

Filtering examples

--collect-only option, you can learn which tests would be invoked. To list all the applicable tests for the current configuration, use the following command:

pytest --collect-only -q --execute-upgrade-tests --force-resource-intensive-tests

List tests specific to vnodes (which would only run if vnodes are enabled):

pytest --collect-only -q --execute-upgrade-tests --force-resource-intensive-tests --use-vnodes -m vnodes

List tests that are not resource-intensive

pytest --collect-only -q --execute-upgrade-tests --skip-resource-intensive-tests

Upgrade tests

Upgrade tests always involve more than one product version. There are two kinds of upgrade tests regarding the product versions they span - let’s call them fixed and generated. In case of fixed tests, the origin and target versions are hardcoded. They look pretty usual, for example:

pytest --collect-only -q --execute-upgrade-tests --execute-upgrade-tests-only upgrade_tests/upgrade_supercolumns_test.py

prints:

upgrade_tests/upgrade_supercolumns_test.py::TestSCUpgrade::test_upgrade_super_columns_through_all_versionsupgrade_tests/upgrade_supercolumns_test.py::TestSCUpgrade::test_upgrade_super_columns_through_limited_versions

When you look into the code, you will see the fixed upgrade path:

def test_upgrade_super_columns_through_all_versions(self):    self._upgrade_super_columns_through_versions_test(upgrade_path=[indev_2_2_x, indev_3_0_x, indev_3_11_x, indev_trunk])

The generated upgrade tests are listed several times - the first occurrence of the test case is a generic test definition, and then it is repeated many times in generated test classes. For example:

pytest --cassandra-dir=/home/cassandra/cassandra --collect-only -q --execute-upgrade-tests --execute-upgrade-tests-only upgrade_tests/cql_tests.py -k test_set

prints:

upgrade_tests/cql_tests.py::cls::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_2_2_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_current_3_11_x_To_indev_3_11_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_current_4_0_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_indev_2_2_x_To_indev_3_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_indev_2_2_x_To_indev_3_11_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_indev_3_0_x_To_indev_3_11_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_indev_3_0_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_indev_3_11_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_indev_4_0_x_To_indev_trunk::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_current_2_2_x_To_indev_2_2_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_current_3_11_x_To_indev_3_11_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_current_4_0_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_indev_2_2_x_To_indev_3_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_indev_2_2_x_To_indev_3_11_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_indev_3_0_x_To_indev_3_11_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_indev_3_0_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_indev_3_11_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_indev_4_0_x_To_indev_trunk::test_set

test_set, and the class name is TestCQL - the suffix of the class name is automatically generated from the provided specification. The first component is the cluster specification - there are two variants: Nodes2RF1 and Nodes3RF3 - they denote that the upgrade is tested on 2 nodes cluster with a keyspace using replication factor = 1. Analogously the second variant uses 3 nodes cluster with RF = 3. Upgrade_indev_3_11_x_To_indev_4_0_x - which means that this test upgrades from the development version of Cassandra 3.11 to the development version of Cassandra 4.0 - the meaning of indev/current and where they are defined is explained later. UpgradeTester class, and they have the specifications defined at the end of the file. In this particular case, it is something like:

topology_specs = [    {'NODES': 3,     'RF': 3,     'CL': ConsistencyLevel.ALL},    {'NODES': 2,     'RF': 1},]specs = [dict(s, UPGRADE_PATH=p, __test__=True)for s, p in itertools.product(topology_specs, build_upgrade_pairs())]

build_upgrade_pairs() function. That list of specifications is used to dynamically generate upgrade tests. build_upgrade_pairs() returns the list of upgrade paths (actually just the origin and target version). That list is generated according to the upgrade manifest.

Upgrade manifest

upgrade_tests/upgrade_manifest.py. As you noticed, Cassandra origin and target version descriptions mentioned in the upgrade test consist of indev or current prefix followed by version string. The definitions of each such version description can be found in the manifest, for example:

indev_3_11_x = VersionMeta(name='indev_3_11_x', family=CASSANDRA_3_11, variant='indev', version='github:apache/cassandra-3.11', min_proto_v=3, max_proto_v=4, java_versions=(8,))current_3_11_x = VersionMeta(name='current_3_11_x', family=CASSANDRA_3_11, variant='current', version='3.11.10', min_proto_v=3, max_proto_v=4, java_versions=(8,))

There are a couple of different properties which describe those two versions:

name - is a name as you can see in the names of the generated test classes
family - families is an enumeration defined in the beginning of the upgrade manifest - say family CASSANDRA_3_11 is just a string "3.11". Some major features were introduced or removed with new version families, and therefore some checks can be done or some features can be enabled/disabled according to that, for example:
```
if self.cluster.version() < CASSANDRA_4_0:    node1.nodetool("enablethrift")
```
But it is also used to determine whether our checked-out version matches the target version in the upgrade pair (more on that later)
variant and version - there are indev or current variants:
indev variant means that the development version of Cassandra will be used. That is, that version is checked out from the Git repository and built before running the upgrade (CCM does it). In this case, the version string is specified as github:apache/cassandra-3.11, which means that it will checkout the cassandra-3.11 branch from the GitHub repository whose alias is apache. Aliases are defined in CCM configuration file, usually located at ~/.ccm/config - in this particular case, it could be something like:
```
[aliases]apache:git@github.com:apache/cassandra.git
```
current variant means that a released version of Cassandra will be used. It means that Cassandra distribution denoted by the specified version (3.11.10 in this case) is downloaded from the Apache repository/mirror - again, the repository can be defined in CCM configuration file, under repositories section, something like:
```
[repositories]cassandra=https://archive.apache.org/dist/cassandra
```
min_proto_v, max_proto_v - the range of usable Cassandra driver protocol versions
java_versions - supported Java versions MANIFEST map which may look similar to:
```
MANIFEST = {current_2_1_x: [indev_2_2_x, indev_3_0_x, indev_3_11_x],current_2_2_x: [indev_2_2_x, indev_3_0_x, indev_3_11_x],current_3_0_x: [indev_3_0_x, indev_3_11_x, indev_4_0_x],current_3_11_x: [indev_3_11_x, indev_4_0_x],current_4_0_x: [indev_4_0_x, indev_trunk],   indev_2_2_x: [indev_3_0_x, indev_3_11_x],   indev_3_0_x: [indev_3_11_x, indev_4_0_x],   indev_3_11_x: [indev_4_0_x],   indev_4_0_x: [indev_trunk]}
```
It is a simple map where for the origin version (as a key), there is a list of possible target versions (as a value). Say:
```
current_4_0_x: [indev_4_0_x, indev_trunk]
```
current_4_0_x to indev_4_0_x and from current_4_0_x to indev_trunk will be considered. You may make changes to that upgrade scenario in your development branch according to your needs. There is a command-line option that allows filtering across upgrade scenarios: --upgrade-version-selection=xxx. The possible values for that options are as follows:
indev - which is the default, only selects those upgrade scenarios where the target version is in indev variant
both - selects upgrade paths where either both origin and target versions are in the same variant or have the same version family
releases - selects upgrade paths between versions in current variant or from the current to indev variant if both have the same version family
all - no filtering at all - all variants are tested
Running upgrades with local distribution
cassandra_dir property, as the target version if the following preconditions are satisfied:
indev variant,
the version family set in the version description matches the version family of your local distribution cassandra-4.0 branch, likely matching indev_4_0_x. It means that the upgrade path with target version indev_4_0_x uses your local distribution. There is a handy command line option which will filter out all the upgrade tests which do not match the local distribution: --upgrade-target-version-only. Given you are on cassandra-4.0 branch, when applied to the previous example, it will be something similar to:
```
pytest --cassandra-dir=/home/cassandra/cassandra --collect-only -q --execute-upgrade-tests --execute-upgrade-tests-only upgrade_tests/cql_tests.py -k test_set --upgrade-target-version-only
```
prints:
```
upgrade_tests/cql_tests.py::cls::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_current_4_0_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_indev_3_0_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes3RF3_Upgrade_indev_3_11_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_current_4_0_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_indev_3_0_x_To_indev_4_0_x::test_setupgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_indev_3_11_x_To_indev_4_0_x::test_set
```
indev and family matches 4.0.
Logging
--log-xxx are pretty well described in the help message (pytest --help) and in PyTest documentation, so it will not be discussed further. However, most of the tests start with the cluster of Cassandra nodes, and each node generates its own logging information and has its own data directories. By default the logs from the nodes are copied to the unique directory created under logs subdirectory under root of dtest project. For example:
```
(venv) cassandra@b69a382da7cd:~/cassandra-dtest$ ls logs/ -11627455923457_test_set1627456019264_test_set1627456474949_test_set1627456527540_test_listlast
```
last item is a symbolic link to the directory containing the logs from the last executed test. Each such directory includes logs from each started node - system, debug, GC as well as standard streams registered upon each time the node was started:
```
(venv) cassandra@b69a382da7cd:~/cassandra-dtest$ ls logs/last -1node1.lognode1_debug.lognode1_gc.lognode1_startup-1627456480.3398306-stderr.lognode1_startup-1627456480.3398306-stdout.lognode1_startup-1627456507.2186499-stderr.lognode1_startup-1627456507.2186499-stdout.lognode2.lognode2_debug.lognode2_gc.lognode2_startup-1627456481.10463-stderr.lognode2_startup-1627456481.10463-stdout.log
```
--delete-logs command-line option is added to PyTest. The nodes also produce data files which may be sometimes useful to examine to resolve some failures. Those files are usually deleted when the test is completed, but there are some options to control that behavior:
--keep-test-dir - keep the whole CCM directory with data files and logs when the test completes
--keep-failed-test-dir – only keep that directory when the test has failed -s option to the command line and then look for "dtest_setup INFO" messages. For example:
```
05:56:06,383 dtest_setup INFO cluster ccm directory: /tmp/dtest-0onwvgkr
```
/tmp/dtest-0onwvgkr, and all node directories can be found under the test subdirectory:
```
(venv) cassandra@b69a382da7cd:~/cassandra-dtest$ ls /tmp/dtest-0onwvgkr/test -1cluster.confnode1node2
```
Performance Testing
Performance tests for Cassandra are a special breed of tests that are not part of the usual patch contribution process. In fact, many people contribute a lot of patches to Cassandra without ever running performance tests. However, they are important when working on performance improvements; such improvements must be measurable. Several tools exist for running performance tests. Here are a few to investigate:
Micro-benchmarks
cassandra-stress: built-in Cassandra stress tool
tlp-stress
NoSQLBench

Testing

Java tests

JUnit tests

Stress and FQLTool tests

JVM distributed tests

Upgrade tests

Running multiple tests

Running coverage analysis

Micro-benchmarks

Python tests

Docker

Setup Docker

Pull the Docker image

Start the container

Setup Python environment

CQLSH tests

Running selected tests

Python distributed tests

Run the tests - quick examples

Setting up PyTest

Running tests with specific configuration

Listing the tests

Filtering tests

Based on configuration

Based on resource usage

Based on the test type

Filtering examples

Upgrade tests

Upgrade manifest

Running upgrades with local distribution

Logging

Performance Testing