Production recommendations

cassandra.yaml and jvm.options files have a number of notes and recommendations for production usage. This page expands on some of the information in the files.

Tokens

vnodes facilitate flexible expansion with more streaming peers when a new node bootstraps into a cluster. Limiting the negative impact of streaming (I/O and CPU overhead) enables incremental cluster expansion. However, more tokens leads to sharing data with more peers, and results in decreased availability. These two factors must be balanced based on a cluster’s characteristic reads and writes. To learn more, Cassandra Availability in Virtual Nodes, Joseph Lynch and Josh Snyder is recommended reading. cassandra.yaml file: num_tokens: 16 Here are the most common token counts with a brief explanation of when and why you would use each one. allocate_tokens_for_local_replication_factor in cassandra.yaml is set to an appropriate number of replicates, to ensure even token allocation.

Read ahead

Read ahead is an operating system feature that attempts to keep as much data as possible loaded in the page cache. Spinning disks can have long seek times causing high latency, so additional throughput on reads using page cache can improve performance. By leveraging read ahead, the OS can pull additional data into memory without the cost of additional seeks. This method works well when the available RAM is greater than the size of the hot dataset, but can be problematic when the reverse is true (dataset > RAM). The larger the hot dataset, the less read ahead is useful. Read ahead is definitely not useful in the following cases:

• Small partitions, such as tables with a single partition key
• Solid state drives (SSDs) Read ahead can actually increase disk usage, and in some cases result in as much as a 5x latency and throughput performance penalty. Read-heavy, key/value tables with small (under 1KB) rows are especially prone to this problem. The recommended read ahead settings are: blockdev tool. /dev/sda1 to 4KB:

  $ blockdev --setra 8 /dev/sda1

  Diving Deep, using external tools section.

  Compression

  chunk_length_in_kb element in the compression map of a table’s schema settings for WITH COMPRESSION. The default setting is 16KB starting with Cassandra 4.0. Since the entire compressed buffer must be read off-disk, using a compression chunk length that is too large can lead to significant overhead when reading small records. Combined with the default read ahead setting, the result can be massive read amplification for certain workloads. Therefore, picking an appropriate value for this setting is important. The Last Pickle blogpost on compression performance.

  Compaction

  compaction strategies available for different workloads. We recommend reading about the different strategies to understand which is the best for your environment. Different tables may, and frequently do use different compaction strategies in the same cluster.

  Encryption

  It is significantly better to set up peer-to-peer encryption and client server encryption when setting up your production cluster. Setting it up after the cluster is serving production traffic is challenging to do correctly. If you ever plan to use network encryption of any type, we recommend setting it up when initially configuring your cluster. Changing these configurations later is not impossible, but mistakes can result in downtime or data loss.

  Ensure keyspaces are created with NetworkTopologyStrategy

  SimpleStrategy. Production keyspaces should use the NetworkTopologyStrategy (NTS). For example:

  CREATE KEYSPACE mykeyspace WITH replication =     {   'class': 'NetworkTopologyStrategy',   'datacenter1': 3};

  NetworkTopologyStrategy can take advantage of the ability to configure multiple racks and data centers.

  Configure racks and snitch

  Correctly configuring or changing racks after a cluster has been provisioned is an unsupported process. Migrating from a single rack to multiple racks is also unsupported and can result in data loss. Using GossipingPropertyFileSnitch is the most flexible solution for on-premise or mixed cloud environments. Ec2Snitch is reliable for AWS EC2 only environments.