Monitoring

Dropwizard Metrics library. Metrics can be queried via JMX, Virtual Tables, or pushed to external monitoring systems using a variety of built-in reporters or third-party reporter plug-ins. Metrics are collected for a single node. It’s up to the operator to use an external monitoring system to aggregate them.

Metric Types

All metrics reported by cassandra fit into one of the following types. Gauge An instantaneous measurement of a value. Counter AtomicLong instance. Typically this is consumed by monitoring the change since the last call to see if there is a large increase compared to the norm. Histogram Measures the statistical distribution of values in a stream of data. + In addition to minimum, maximum, mean, etc., it also measures median, 75th, 90th, 95th, 98th, 99th, and 99.9th percentiles. Timer Measures both the rate that a particular piece of code is called and the histogram of its duration. Latency Timer plus a Counter that tracks the total latency accrued since starting. The former is useful if you track the change in total latency since the last check. Each metric name of this type will have ‘Latency’ and ‘TotalLatency’ appended to it. Meter A meter metric which measures mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving average throughputs.

Table Metrics

Each table in Cassandra has metrics responsible for tracking its state and performance. Keyspace and Table name. Reported name format: Metric Name org.apache.cassandra.metrics.Table.<MetricName>.<Keyspace>.<Table> JMX MBean org.apache.cassandra.metrics:type=Table keyspace=<Keyspace> scope=<Table> name=<MetricName>

Keyspace Metrics

Each keyspace in Cassandra has metrics responsible for tracking its state and performance. Table Metrics above, only they are aggregated at the Keyspace level. The keyspace specific metrics are specified in the table below. Reported name format: Metric Name org.apache.cassandra.metrics.keyspace.<MetricName>.<Keyspace> JMX MBean org.apache.cassandra.metrics:type=Keyspace scope=<Keyspace> name=<MetricName>

ThreadPool Metrics

Cassandra splits work of a particular type into its own thread pool. This provides back-pressure and asynchrony for requests on a node. It’s important to monitor the state of these thread pools since they can tell you how saturated a node is. ThreadPool name. The thread pools are also categorized under a specific type. Reported name format: Metric Name org.apache.cassandra.metrics.ThreadPools.<MetricName>.<Path>.<ThreadPoolName> JMX MBean org.apache.cassandra.metrics:type=ThreadPools path=<Path> scope=<ThreadPoolName> name=<MetricName> The following thread pools can be monitored.

Client Request Metrics

Client requests have their own set of metrics that encapsulate the work happening at coordinator level. RequestType. Reported name format: Metric Name org.apache.cassandra.metrics.ClientRequest.<MetricName>.<RequestType> JMX MBean org.apache.cassandra.metrics:type=ClientRequest scope=<RequestType> name=<MetricName> RequestType CASRead Description Metrics related to transactional read requests. Metrics RequestType CASWrite Description Metrics related to transactional write requests. Metrics RequestType Read Description Metrics related to standard read requests. Metrics RequestType RangeSlice Description Metrics related to token range read requests. Metrics RequestType Write Description Metrics related to regular write requests. Metrics RequestType ViewWrite Description Metrics related to materialized view write wrtes. Metrics

Cache Metrics

Table Metrics might be more useful. Reported name format: Metric Name org.apache.cassandra.metrics.Cache.<MetricName>.<CacheName> JMX MBean org.apache.cassandra.metrics:type=Cache scope=<CacheName> name=<MetricName> The following caches are covered:

CQL Metrics

Metrics specific to CQL prepared statement caching. Reported name format: Metric Name org.apache.cassandra.metrics.CQL.<MetricName> JMX MBean org.apache.cassandra.metrics:type=CQL name=<MetricName>

DroppedMessage Metrics

Hinted Handoff Reported name format: Metric Name org.apache.cassandra.metrics.DroppedMessage.<MetricName>.<Type> JMX MBean org.apache.cassandra.metrics:type=DroppedMessage scope=<Type> name=<MetricName> The different types of messages tracked are:

Streaming Metrics

Streaming operations, such as repair, bootstrap, rebuild. These metrics are specific to a peer endpoint, with the source node being the node you are pulling the metrics from. Reported name format: Metric Name org.apache.cassandra.metrics.Streaming.<MetricName>.<PeerIP> JMX MBean org.apache.cassandra.metrics:type=Streaming scope=<PeerIP> name=<MetricName>

Compaction Metrics

Compaction work. Reported name format: Metric Name org.apache.cassandra.metrics.Compaction.<MetricName> JMX MBean org.apache.cassandra.metrics:type=Compaction name=<MetricName>

CommitLog Metrics

CommitLog Reported name format: Metric Name org.apache.cassandra.metrics.CommitLog.<MetricName> JMX MBean org.apache.cassandra.metrics:type=CommitLog name=<MetricName>

Storage Metrics

Metrics specific to the storage engine. Reported name format: Metric Name org.apache.cassandra.metrics.Storage.<MetricName> JMX MBean org.apache.cassandra.metrics:type=Storage name=<MetricName>

HintedHandoff Metrics

Storage Metrics in the metric name Reported name format: Metric Name org.apache.cassandra.metrics.HintedHandOffManager.<MetricName> JMX MBean org.apache.cassandra.metrics:type=HintedHandOffManager name=<MetricName>

HintsService Metrics

Storage Metrics in the metric name Reported name format: Metric Name org.apache.cassandra.metrics.HintsService.<MetricName> JMX MBean org.apache.cassandra.metrics:type=HintsService name=<MetricName>

SSTable Index Metrics

Metrics specific to the SSTable index metadata. Reported name format: Metric Name org.apache.cassandra.metrics.Index.<MetricName>.RowIndexEntry JMX MBean org.apache.cassandra.metrics:type=Index scope=RowIndexEntry name=<MetricName>

BufferPool Metrics

Metrics specific to the internal recycled buffer pool Cassandra manages. This pool is meant to keep allocations and GC lower by recycling on and off heap buffers. Reported name format: Metric Name org.apache.cassandra.metrics.BufferPool.<MetricName> JMX MBean org.apache.cassandra.metrics:type=BufferPool name=<MetricName>

Client Metrics

Metrics specifc to client managment. Reported name format: Metric Name org.apache.cassandra.metrics.Client.<MetricName> JMX MBean org.apache.cassandra.metrics:type=Client name=<MetricName>

Batch Metrics

Metrics specifc to batch statements. Reported name format: Metric Name org.apache.cassandra.metrics.Batch.<MetricName> JMX MBean org.apache.cassandra.metrics:type=Batch name=<MetricName>

JVM Metrics

Metric Reporters.

BufferPool

Metric Name jvm.buffers.<direct|mapped>.<MetricName> JMX MBean java.nio:type=BufferPool name=<direct|mapped>

FileDescriptorRatio

Metric Name jvm.fd.<MetricName> JMX MBean java.lang:type=OperatingSystem name=<OpenFileDescriptorCount|MaxFileDescriptorCount>

GarbageCollector

Metric Name jvm.gc.<gc_type>.<MetricName> JMX MBean java.lang:type=GarbageCollector name=<gc_type>

Memory

Metric Name jvm.memory.<heap/non-heap/total>.<MetricName> JMX MBean java.lang:type=Memory

MemoryPool

Metric Name jvm.memory.pools.<memory_pool>.<MetricName> JMX MBean java.lang:type=MemoryPool name=<memory_pool>

JMX

Any JMX based client can access metrics from cassandra. Mx4jTool and place mx4j-tools.jar into the classpath. On startup you will see in the log:

  1. HttpAdaptor version 3.0.2 started on port 8081

conf/cassandra-env.sh and uncomment:

  1. #MX4J_ADDRESS="-Dmx4jaddress=0.0.0.0"#MX4J_PORT="-Dmx4jport=8081"

Metric Reporters

built-in reporters or third-party reporter plugins.