Production recommendations

The cassandra.yaml and jvm.options files have a number of notes and recommendations for production usage. This page expands on some of the information in the files.

Tokens

Using more than one token-range per node is referred to as virtual nodes, or vnodes. vnodes facilitate flexible expansion with more streaming peers when a new node bootstraps into a cluster. Limiting the negative impact of streaming (I/O and CPU overhead) enables incremental cluster expansion. However, more tokens leads to sharing data with more peers, and results in decreased availability. These two factors must be balanced based on a cluster’s characteristic reads and writes. To learn more, Cassandra Availability in Virtual Nodes, Joseph Lynch and Josh Snyder is recommended reading.

Change the number of tokens using the setting in the cassandra.yaml file:

num_tokens: 16

Here are the most common token counts with a brief explanation of when and why you would use each one.

In addition to setting the token count, it’s extremely important that allocate_tokens_for_local_replication_factor in cassandra.yaml is set to an appropriate number of replicates, to ensure even token allocation.

Read ahead

Read ahead is an operating system feature that attempts to keep as much data as possible loaded in the page cache. Spinning disks can have long seek times causing high latency, so additional throughput on reads using page cache can improve performance. By leveraging read ahead, the OS can pull additional data into memory without the cost of additional seeks. This method works well when the available RAM is greater than the size of the hot dataset, but can be problematic when the reverse is true (dataset > RAM). The larger the hot dataset, the less read ahead is useful.

Read ahead is definitely not useful in the following cases:

• Small partitions, such as tables with a single partition key

• Solid state drives (SSDs)

Read ahead can actually increase disk usage, and in some cases result in as much as a 5x latency and throughput performance penalty. Read-heavy, key/value tables with small (under 1KB) rows are especially prone to this problem.

The recommended read ahead settings are:

Read ahead can be adjusted on Linux systems using the blockdev tool.

For example, set the read ahead of the disk /dev/sda1 to 4KB:

$ blockdev --setra 8 /dev/sda1

All systems are different, so use these recommendations as a starting point and tune, based on your SLA and throughput requirements. To understand how read ahead impacts disk resource usage, we recommend carefully reading through the Diving Deep, using external tools section.

Compression

Compressed data is stored by compressing fixed-size byte buffers and writing the data to disk. The buffer size is determined by the chunk_length_in_kb element in the compression map of a table’s schema settings for WITH COMPRESSION. The default setting is 16KB starting with Cassandra 4.0.

Since the entire compressed buffer must be read off-disk, using a compression chunk length that is too large can lead to significant overhead when reading small records. Combined with the default read ahead setting, the result can be massive read amplification for certain workloads. Therefore, picking an appropriate value for this setting is important.

LZ4Compressor is the default and recommended compression algorithm. If you need additional information on compression, read The Last Pickle blogpost on compression performance.

Compaction

There are different compaction strategies available for different workloads. We recommend reading about the different strategies to understand which is the best for your environment. Different tables may, and frequently do use different compaction strategies in the same cluster.

Encryption

It is significantly better to set up peer-to-peer encryption and client server encryption when setting up your production cluster. Setting it up after the cluster is serving production traffic is challenging to do correctly. If you ever plan to use network encryption of any type, we recommend setting it up when initially configuring your cluster. Changing these configurations later is not impossible, but mistakes can result in downtime or data loss.

Ensure keyspaces are created with NetworkTopologyStrategy

Production clusters should never use SimpleStrategy. Production keyspaces should use the NetworkTopologyStrategy (NTS). For example:

CREATE KEYSPACE mykeyspace WITH replication =     {
   'class': 'NetworkTopologyStrategy',
   'datacenter1': 3
};

Cassandra clusters initialized with NetworkTopologyStrategy can take advantage of the ability to configure multiple racks and data centers.

Configure racks and snitch

Correctly configuring or changing racks after a cluster has been provisioned is an unsupported process. Migrating from a single rack to multiple racks is also unsupported and can result in data loss. Using GossipingPropertyFileSnitch is the most flexible solution for on-premise or mixed cloud environments. Ec2Snitch is reliable for AWS EC2 only environments.