• Monitoring
  • Metric Types
  • Table Metrics
  • DroppedMessage Metrics
  • Streaming Metrics
  • Compaction Metrics
  • CommitLog Metrics
  • Storage Metrics
  • HintedHandoff Metrics
  • HintsService Metrics
  • SSTable Index Metrics
  • BufferPool Metrics
  • Client Metrics
  • Batch Metrics
  • JVM Metrics
  • BufferPool
  • FileDescriptorRatio
  • GarbageCollector
  • Memory
  • MemoryPool
  • JMX
  • Metric Reporters

    Monitoring

    Dropwizard Metrics library. These metrics can be queried via JMX or pushed to external monitoring systems using a number of built in and third party reporter plugins. Metrics are collected for a single node. It’s up to the operator to use an external monitoring system to aggregate them.

    Metric Types

    All metrics reported by cassandra fit into one of the following types. Gauge An instantaneous measurement of a value. Counter AtomicLong instance. Typically this is consumed by monitoring the change since the last call to see if there is a large increase compared to the norm. Histogram Measures the statistical distribution of values in a stream of data. + In addition to minimum, maximum, mean, etc., it also measures median, 75th, 90th, 95th, 98th, 99th, and 99.9th percentiles. Timer Measures both the rate that a particular piece of code is called and the histogram of its duration. Latency Timer plus a Counter that tracks the total latency accrued since starting. The former is useful if you track the change in total latency since the last check. Each metric name of this type will have ‘Latency’ and ‘TotalLatency’ appended to it. Meter A meter metric which measures mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving average throughputs.

    Table Metrics

    Each table in Cassandra has metrics responsible for tracking its state and performance. Keyspace and Table name. Reported name format: Metric Name org.apache.cassandra.metrics.Table.<MetricName>.<Keyspace>.<Table> JMX MBean org.apache.cassandra.metrics:type=Table keyspace=<Keyspace> scope=<Table> name=<MetricName> Misses and MissLatency are only defined for the ChunkCache ====== CQL Metrics Metrics specific to CQL prepared statement caching. Reported name format: Metric Name org.apache.cassandra.metrics.CQL.<MetricName> JMX MBean org.apache.cassandra.metrics:type=CQL name=<MetricName>

    DroppedMessage Metrics

    Hinted Handoff Reported name format: Metric Name org.apache.cassandra.metrics.DroppedMessage.<MetricName>.<Type> JMX MBean org.apache.cassandra.metrics:type=DroppedMessage scope=<Type> name=<MetricName> The different types of messages tracked are:

    Streaming Metrics

    Streaming operations, such as repair, bootstrap, rebuild. These metrics are specific to a peer endpoint, with the source node being the node you are pulling the metrics from. Reported name format: Metric Name org.apache.cassandra.metrics.Streaming.<MetricName>.<PeerIP> JMX MBean org.apache.cassandra.metrics:type=Streaming scope=<PeerIP> name=<MetricName>

    Compaction Metrics

    Compaction work. Reported name format: Metric Name org.apache.cassandra.metrics.Compaction.<MetricName> JMX MBean org.apache.cassandra.metrics:type=Compaction name=<MetricName>

    CommitLog Metrics

    CommitLog Reported name format: Metric Name org.apache.cassandra.metrics.CommitLog.<MetricName> JMX MBean org.apache.cassandra.metrics:type=CommitLog name=<MetricName>

    Storage Metrics

    Metrics specific to the storage engine. Reported name format: Metric Name org.apache.cassandra.metrics.Storage.<MetricName> JMX MBean org.apache.cassandra.metrics:type=Storage name=<MetricName>

    HintedHandoff Metrics

    Storage Metrics in the metric name Reported name format: Metric Name org.apache.cassandra.metrics.HintedHandOffManager.<MetricName> JMX MBean org.apache.cassandra.metrics:type=HintedHandOffManager name=<MetricName>

    HintsService Metrics

    Storage Metrics in the metric name Reported name format: Metric Name org.apache.cassandra.metrics.HintsService.<MetricName> JMX MBean org.apache.cassandra.metrics:type=HintsService name=<MetricName>

    SSTable Index Metrics

    Metrics specific to the SSTable index metadata. Reported name format: Metric Name org.apache.cassandra.metrics.Index.<MetricName>.RowIndexEntry JMX MBean org.apache.cassandra.metrics:type=Index scope=RowIndexEntry name=<MetricName>

    BufferPool Metrics

    Metrics specific to the internal recycled buffer pool Cassandra manages. This pool is meant to keep allocations and GC lower by recycling on and off heap buffers. Reported name format: Metric Name org.apache.cassandra.metrics.BufferPool.<MetricName> JMX MBean org.apache.cassandra.metrics:type=BufferPool name=<MetricName>

    Client Metrics

    Metrics specifc to client managment. Reported name format: Metric Name org.apache.cassandra.metrics.Client.<MetricName> JMX MBean org.apache.cassandra.metrics:type=Client name=<MetricName>

    Batch Metrics

    Metrics specifc to batch statements. Reported name format: Metric Name org.apache.cassandra.metrics.Batch.<MetricName> JMX MBean org.apache.cassandra.metrics:type=Batch name=<MetricName>

    JVM Metrics

    Metric Reporters.

    BufferPool

    Metric Name jvm.buffers.<direct|mapped>.<MetricName> JMX MBean java.nio:type=BufferPool name=<direct|mapped>

    FileDescriptorRatio

    Metric Name jvm.fd.<MetricName> JMX MBean java.lang:type=OperatingSystem name=<OpenFileDescriptorCount|MaxFileDescriptorCount>

    GarbageCollector

    Metric Name jvm.gc.<gc_type>.<MetricName> JMX MBean java.lang:type=GarbageCollector name=<gc_type>

    Memory

    Metric Name jvm.memory.<heap/non-heap/total>.<MetricName> JMX MBean java.lang:type=Memory

    MemoryPool

    Metric Name jvm.memory.pools.<memory_pool>.<MetricName> JMX MBean java.lang:type=MemoryPool name=<memory_pool>

    JMX

    Any JMX based client can access metrics from cassandra. Mx4jTool and place mx4j-tools.jar into the classpath. On startup you will see in the log:
    1. HttpAdaptor version 3.0.2 started on port 8081
    conf/cassandra-env.sh and uncomment:
    1. #MX4J_ADDRESS="-Dmx4jaddress=0.0.0.0"#MX4J_PORT="-Dmx4jport=8081"

    Metric Reporters

    built in and third party reporter plugins. metrics reporter config project. There is a sample configuration file located at conf/metrics-reporter-config-sample.yaml. -Dcassandra.metricsReporterConfigFile=metrics-reporter-config.yaml. The specified .yaml file plus any 3rd party reporter jars must all be in Cassandra’s classpath.