Use Nodetool

nodetool allows you to narrow problems from the cluster down to a particular node and gives a lot of insight into the state of the Cassandra process itself. There are dozens of useful commands (see nodetool help for all the commands), but briefly some of the most useful for troubleshooting:

Cluster Status

nodetool status to assess status of the cluster:

$ nodetool status <optional keyspace>Datacenter: dc1=======================Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address    Load       Tokens       Owns (effective)  Host ID                               RackUN  127.0.1.1  4.69 GiB   1            100.0%            35ea8c9f-b7a2-40a7-b9c5-0ee8b91fdd0e  r1UN  127.0.1.2  4.71 GiB   1            100.0%            752e278f-b7c5-4f58-974b-9328455af73f  r2UN  127.0.1.3  4.69 GiB   1            100.0%            9dc1a293-2cc0-40fa-a6fd-9e6054da04a7  r3

nodetool status on multiple nodes in a cluster to see the full view. nodetool status plus a little grep to see which nodes are down:

$ nodetool status | grep -v '^UN'Datacenter: dc1===============Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address    Load       Tokens       Owns (effective)  Host ID                               RackDatacenter: dc2===============Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address    Load       Tokens       Owns (effective)  Host ID                               RackDN  127.0.0.5  105.73 KiB  1            33.3%             df303ac7-61de-46e9-ac79-6e630115fd75  r1

dc2 and rack r1. This may indicate an issue on 127.0.0.5 warranting investigation.

Coordinator Query Latency

nodetool proxyhistograms:

$ nodetool proxyhistogramsPercentile       Read Latency      Write Latency      Range Latency   CAS Read Latency  CAS Write Latency View Write Latency                     (micros)           (micros)           (micros)           (micros)           (micros)           (micros)50%                    454.83             219.34               0.00               0.00               0.00               0.0075%                    545.79             263.21               0.00               0.00               0.00               0.0095%                    654.95             315.85               0.00               0.00               0.00               0.0098%                    785.94             379.02               0.00               0.00               0.00               0.0099%                   3379.39            2346.80               0.00               0.00               0.00               0.00Min                     42.51             105.78               0.00               0.00               0.00               0.00Max                  25109.16           43388.63               0.00               0.00               0.00               0.00

select * from keyspace.table), CAS read (compare phase of CAS) and CAS write (set phase of compare and set). These can be useful for narrowing down high level latency problems, for example in this case if a client had a 20 millisecond timeout on their reads they might experience the occasional timeout from this node but less than 1% (since the 99% read latency is 3.3 milliseconds < 20 milliseconds).

Local Query Latency

nodetool tablehistograms to get a better idea of what is happening locally on a node:

$ nodetool tablehistograms keyspace tablePercentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count                              (micros)          (micros)           (bytes)50%             0.00             73.46            182.79             17084               10375%             1.00             88.15            315.85             17084               10395%             2.00            126.93            545.79             17084               10398%             2.00            152.32            654.95             17084               10399%             2.00            182.79            785.94             17084               103Min             0.00             42.51             24.60             14238                87Max             2.00          12108.97          17436.92             17084               103

This shows you percentile breakdowns particularly critical metrics. SizeTieredCompactionStrategy typically has many more reads per read than LeveledCompactionStrategy does for update heavy workloads. local write latency. In this case we see that while the p50 is quite good at 73 microseconds, the maximum latency is quite slow at 12 milliseconds. High write max latencies often indicate a slow commitlog volume (slow to fsync) or large writes that quickly saturate commitlog segments. local read latency. We can see that local Cassandra reads are (as expected) slower than local writes, and the read speed correlates highly with the number of sstables read per read. The fourth and fifth columns show distributions of partition size and column count per partition. These are useful for determining if the table has on average skinny or wide partitions and can help you isolate bad data patterns. For example if you have a single cell that is 2 megabytes, that is probably going to cause some heap pressure when it’s read.

Threadpool State

nodetool tpstats to view the current outstanding requests on a particular node. This is useful for trying to find out which resource (read threads, write threads, compaction, request response threads) the Cassandra process lacks. For example:

$ nodetool tpstatsPool Name                         Active   Pending      Completed   Blocked  All time blockedReadStage                              2         0             12         0                 0MiscStage                              0         0              0         0                 0CompactionExecutor                     0         0           1940         0                 0MutationStage                          0         0              0         0                 0GossipStage                            0         0          10293         0                 0Repair-Task                            0         0              0         0                 0RequestResponseStage                   0         0             16         0                 0ReadRepairStage                        0         0              0         0                 0CounterMutationStage                   0         0              0         0                 0MemtablePostFlush                      0         0             83         0                 0ValidationExecutor                     0         0              0         0                 0MemtableFlushWriter                    0         0             30         0                 0ViewMutationStage                      0         0              0         0                 0CacheCleanupExecutor                   0         0              0         0                 0MemtableReclaimMemory                  0         0             30         0                 0PendingRangeCalculator                 0         0             11         0                 0SecondaryIndexManagement               0         0              0         0                 0HintsDispatcher                        0         0              0         0                 0Native-Transport-Requests              0         0            192         0                 0MigrationStage                         0         0             14         0                 0PerDiskMemtableFlushWriter_0           0         0             30         0                 0Sampler                                0         0              0         0                 0ViewBuildExecutor                      0         0              0         0                 0InternalResponseStage                  0         0              0         0                 0AntiEntropyStage                       0         0              0         0                 0Message type           Dropped                  Latency waiting in queue (micros)                                             50%               95%               99%               MaxREAD                         0               N/A               N/A               N/A               N/ARANGE_SLICE                  0              0.00              0.00              0.00              0.00_TRACE                       0               N/A               N/A               N/A               N/AHINT                         0               N/A               N/A               N/A               N/AMUTATION                     0               N/A               N/A               N/A               N/ACOUNTER_MUTATION             0               N/A               N/A               N/A               N/ABATCH_STORE                  0               N/A               N/A               N/A               N/ABATCH_REMOVE                 0               N/A               N/A               N/A               N/AREQUEST_RESPONSE             0              0.00              0.00              0.00              0.00PAGED_RANGE                  0               N/A               N/A               N/A               N/AREAD_REPAIR                  0               N/A               N/A               N/A               N/A

RequestResponseState queue is backing up, that means that the coordinators are waiting on a lot of downstream replica requests and may indicate a lack of token awareness, or very high consistency levels being used on read requests (for example reading at ALL ties up RF RequestResponseState threads whereas LOCAL_ONE only uses a single thread in the ReadStage threadpool). On the other hand if you see a lot of pending compactions that may indicate that your compaction threads cannot keep up with the volume of writes and you may need to tune either the compaction strategy or the concurrent_compactors or compaction_throughput options. The second section shows drops (errors) and latency distributions for all the major request types. Drops are cumulative since process start, but if you have any that indicate a serious problem as the default timeouts to qualify as a drop are quite high (~5-10 seconds). Dropped messages often warrants further investigation.

Compaction State

page cache, and can put a lot of load on your disk drives. There are great os tools <os-iostat> to determine if this is the case, but often it’s a good idea to check if compactions are even running using nodetool compactionstats:

$ nodetool compactionstatspending tasks: 2- keyspace.table: 2id                                   compaction type keyspace table completed total    unit  progress2062b290-7f3a-11e8-9358-cd941b956e60 Compaction      keyspace table 21848273  97867583 bytes 22.32%Active compaction remaining time :   0h00m04s

keyspace.table table, has completed 21.8 megabytes of 97 and Cassandra estimates (based on the configured compaction throughput) that this will take 4 seconds. You can also pass -H to get the units in a human readable format. concurrent_compactors or compaction_throughput options.

Paths used for data files

data_file_directories. Resembling the structure of keyspaces and tables, Cassandra is creating subdirectories within data_file_directories. However, directories aren’t removed even if the tables and keyspaces are dropped. While these directories are kept with the reason of holding snapshots, they are subject to removal. This is where operators need to know which directories are still in use. Running the nodetool datapaths command is an easy way to list in which directories Cassandra is actually storing sstable data on disk.

% nodetool datapaths -- system_authKeyspace: system_auth    Table: role_permissions    Paths:        /var/lib/cassandra/data/system_auth/role_permissions-3afbe79f219431a7add7f5ab90d8ec9c    Table: network_permissions    Paths:        /var/lib/cassandra/data/system_auth/network_permissions-d46780c22f1c3db9b4c1b8d9fbc0cc23    Table: resource_role_permissons_index    Paths:        /var/lib/cassandra/data/system_auth/resource_role_permissons_index-5f2fbdad91f13946bd25d5da3a5c35ec    Table: roles    Paths:        /var/lib/cassandra/data/system_auth/roles-5bc52802de2535edaeab188eecebb090    Table: role_members    Paths:        /var/lib/cassandra/data/system_auth/role_members-0ecdaa87f8fb3e6088d174fb36fe5c0d

keyspace and keyspace.table arguments can be given to query specific data paths. Using the --format option the output can be formatted as YAML or JSON.

Using nodetool