Skip to content
Navigation

Type at least two characters. Search covers page titles, headings, tags and database names.

↑ ↓ to navigateEnter to openEsc to close0 pages
Distributed SQLadvanced

YugabyteDB Operations

Node lifecycle, rolling upgrades, backup and restore, and the metrics that show cluster health.

2 min readAdvancedUpdated Edit this page
YugabyteDB

Node lifecycle

# Move data off a tablet server before removing it.
yb-admin --master_addresses <masters> change_blacklist ADD <tserver-host>:9100
 
# Watch until the load has drained.
yb-admin --master_addresses <masters> get_load_move_completion
 
# Then stop the process and remove it from the cluster.
yb-admin --master_addresses <masters> change_blacklist REMOVE <tserver-host>:9100

Master nodes are changed with change_master_config; the master quorum must retain a majority throughout.

Rolling upgrades

Upgrade masters first, then tablet servers, one node at a time:

  1. Verify the cluster is healthy and no tablets are under-replicated.
  2. Upgrade and restart each master in turn, waiting for the quorum to re-form.
  3. Upgrade tablet servers one at a time, waiting for each to report healthy.
  4. Run yb-admin upgrade_ysql where the version requires it, to update system catalogs.
  5. Promote auto-flags / finalise the upgrade only after the new version has been observed working.

Backup and restore

# Distributed snapshot of a database, exported to object storage.
yb-admin --master_addresses <masters> create_database_snapshot ysql.shop
yb-admin --master_addresses <masters> list_snapshots
yb-admin --master_addresses <masters> export_snapshot <snapshot-id> /backups/shop.snapshot
 
# Restore.
yb-admin --master_addresses <masters> import_snapshot /backups/shop.snapshot
yb-admin --master_addresses <masters> restore_snapshot <snapshot-id>

Point-in-time recovery is available through snapshot schedules:

yb-admin --master_addresses <masters> \
  create_snapshot_schedule 1440 10080 ysql.shop     # interval and retention, minutes
 
yb-admin --master_addresses <masters> \
  restore_snapshot_schedule <schedule-id> "2026-07-30 14:22:00"

Standard PostgreSQL tools also work through the YSQL interface (ysql_dump), which is convenient for a single table and slow for a whole database — the same trade-off as elsewhere.

Monitoring

under-replicated tablets

Source: yb-admin / master UI

Tablets with fewer than the configured replicas. Any sustained value means reduced fault tolerance.

tablet leader distribution

Source: master UI, /tablet-servers

Leaders should be spread evenly. Concentration on one server means a hotspot or an imbalance after a restart.

handler_latency_yb_tserver_TabletServerService_Write

Source: Prometheus /prometheus-metrics

Write latency at the tablet server, the floor under application write latency.

rpc_inbound_calls_queued

Source: Prometheus /prometheus-metrics

Queued RPCs. Growth means the server cannot keep up with incoming requests.

hybrid_clock_skew

Source: Prometheus /prometheus-metrics

Clock skew between nodes, against max_clock_skew_usec.

transaction conflict / abort counters

Source: Prometheus /prometheus-metrics

Rising aborts mean contention that the application must be retrying.

yb-admin --master_addresses <masters> list_all_tablet_servers
yb-admin --master_addresses <masters> get_universe_config

The master web UI lists tablets, their leaders and any under-replication, which is the quickest health check during an incident.