Cassandraintermediate

Cassandra Monitoring

The nodetool commands and JMX metrics that reveal compaction backlog, tombstone pressure and coordinator latency.

2 min readIntermediateUpdated 31 Jul 2026Edit this page

Cluster state

nodetool status                # up/down, ownership, load per node
nodetool describecluster       # schema agreement
nodetool info                  # heap, uptime, cache hit rates for one node
nodetool netstats              # streaming and read repair activity
nodetool compactionstats       # active and pending compactions
nodetool tpstats               # thread pool activity and dropped messages

nodetool tpstats deserves particular attention: dropped messages mean the node could not process requests within their timeout and silently discarded them. Any non-zero dropped MUTATION count means writes were lost from that node's perspective and will need repair to reconcile.

Metrics that matter

Pending compactions

Source: nodetool compactionstats

A growing backlog means compaction cannot keep up with writes. Reads slow down as SSTable counts climb.

SSTables per read

Source: nodetool tablehistograms

The 99th percentile shows how many files a read merges. Rising values point at compaction falling behind or the wrong strategy.

Tombstones per read

Source: nodetool tablehistograms

Approaching tombstone_warn_threshold means queries are scanning deletion markers. See Tombstones.

Coordinator read and write latency

Source: JMX ClientRequest metrics

End-to-end latency as clients experience it, per consistency level. More meaningful than per-table local latency.

Dropped messages

Source: nodetool tpstats

Requests discarded after timing out internally. Any sustained non-zero value is data the cluster did not apply.

Hints in flight

Source: nodetool statushandoff

Growing hint counts mean a replica is unreachable. Hints expiring means repair is now required.

JMX and Prometheus

Cassandra exposes metrics through JMX. The usual approach is the JMX exporter agent:

java -javaagent:/opt/jmx_prometheus_javaagent.jar=7070:/opt/cassandra.yml \
     -jar cassandra.jar

Metric families worth collecting:

org.apache.cassandra.metrics:type=ClientRequest — latency and timeouts per request type.
org.apache.cassandra.metrics:type=Table — SSTables per read, tombstones, partition sizes.
org.apache.cassandra.metrics:type=Compaction — pending tasks and completed bytes.
org.apache.cassandra.metrics:type=ThreadPools — dropped messages and blocked tasks.
JVM metrics — heap usage and garbage collection pause times.

Garbage collection

A GC pause looks exactly like a dead node

Cassandra runs on the JVM, and a stop-the-world pause longer than the failure detector's threshold causes peers to mark the node down, triggering hints and possibly a spurious repair burden. Monitor GC pause duration as a first-class metric, not as a JVM curiosity.

grep -i "GCInspector" /var/log/cassandra/system.log | tail -20

Logs worth watching

grep -iE "tombstone|dropped|timeout|GCInspector|Not marking" /var/log/cassandra/system.log

Tombstone warnings — a data model problem.
"Dropped N messages" — the node is overloaded.
Large partition warnings during compaction — a partition key problem.

Alerts worth having

Any node not UN in nodetool status.
Pending compactions above an established baseline.
Dropped messages of any type, sustained.
Coordinator p99 latency above the workload's budget.
Disk usage above the level at which compaction can still complete.
Repair not completed for a keyspace within gc_grace_seconds.
Snapshots older than the copy-and-clear job's interval.