Skip to content
Navigation

Type at least two characters. Search covers page titles, headings, tags and database names.

↑ ↓ to navigateEnter to openEsc to close0 pages
ScyllaDBintermediate

ScyllaDB Monitoring

Prometheus metrics, per-shard visibility, and the alerts that matter on a shard-per-core cluster.

2 min readIntermediateUpdated Edit this page
ScyllaDB

ScyllaDB exposes metrics on a Prometheus endpoint rather than through JMX. Anything built against Cassandra's JMX metric names needs rewriting.

curl -s localhost:9180/metrics | head

The vendor ships a monitoring stack (Prometheus, Alertmanager and Grafana dashboards) that is worth starting from rather than assembling dashboards by hand.

Metrics that matter

scylla_reactor_utilization

Source: /metrics, per shard

CPU utilisation per shard. Imbalance between shards on one node is the signature of a hot partition; node averages hide it completely.

scylla_transport_requests_served

Source: /metrics

Request throughput by type. Read with latency histograms to separate a traffic increase from a slowdown.

scylla_storage_proxy_coordinator_*_latency

Source: /metrics

Coordinator-side read and write latency — what clients actually experience.

scylla_compaction_manager_compactions

Source: /metrics

Active and pending compactions. A growing backlog means writes outpace compaction.

scylla_io_queue_delay

Source: /metrics

Queueing delay in the I/O scheduler. High delay with moderate CPU means the cluster is storage bound.

scylla_database_total_writes_failed

Source: /metrics

Failed operations, including timeouts. Any sustained non-zero rate is work the cluster did not complete.

Per-shard visibility is the point

nodetool and the REST API

nodetool status
nodetool tablestats shop.events
nodetool compactionstats
 
# The same information over the REST API, which is what nodetool uses.
curl -s http://localhost:10000/storage_service/nodes/up
curl -s http://localhost:10000/compaction_manager/compactions

ScyllaDB Manager

sctool reports on scheduled repairs and backups, and its task history is what you alert on:

sctool task list --cluster prod
sctool progress --cluster prod repair/<task-id>

Alerts worth having

  • Any node down, or reporting a state other than normal.
  • Per-shard reactor utilisation sustained near saturation while other shards are idle.
  • Coordinator p99 latency above the workload's budget.
  • Failed reads or writes at any sustained rate.
  • Pending compactions above the established baseline.
  • I/O queue delay rising.
  • Disk usage above the level where compaction can still complete.
  • Repair not completed for a keyspace within gc_grace_seconds — the single most important scheduled-task alert.