Skip to content
Navigation

Type at least two characters. Search covers page titles, headings, tags and database names.

↑ ↓ to navigateEnter to openEsc to close0 pages
Time Seriesintermediate

VictoriaMetrics Operations

Single-node versus cluster deployment, retention and downsampling, backup with vmbackup, and the metrics to watch.

2 min readIntermediateUpdated Edit this page

Single node or cluster

The single-node binary handles a substantial ingestion rate and is dramatically simpler to operate. The cluster version splits into three components:

  • vminsert — accepts writes and shards them across storage nodes.
  • vmstorage — stores data; stateful, and the component that holds your data.
  • vmselect — executes queries by fanning out to storage nodes.

Retention and downsampling

victoria-metrics \
  -storageDataPath=/var/lib/victoria-metrics \
  -retentionPeriod=12 \
  -memory.allowedPercent=60 \
  -maxLabelsPerTimeseries=30

-retentionPeriod is in months by default. Retention removes whole partitions, so it is cheap.

Downsampling — storing older data at lower resolution — is an Enterprise feature. On the open-source version, achieve the same effect with recording rules in vmalert that write aggregated series into a separate retention scope, or by aggregating at ingest with stream aggregation.

Backup

# Snapshot, then copy it.
curl http://victoriametrics:8428/snapshot/create
 
vmbackup -storageDataPath=/var/lib/victoria-metrics \
         -snapshotName=<snapshot> \
         -dst=s3://acme-backups/vm/$(date -u +%Y%m%d)
 
# Incremental against a previous backup.
vmbackup -storageDataPath=/var/lib/victoria-metrics \
         -snapshotName=<snapshot> \
         -dst=s3://acme-backups/vm/$(date -u +%Y%m%d) \
         -origin=s3://acme-backups/vm/20260730
 
vmrestore -src=s3://acme-backups/vm/20260731 \
          -storageDataPath=/var/lib/victoria-metrics

Delete snapshots after copying (/snapshot/delete) — like hard-link snapshots elsewhere, they keep old data alive on disk.

Monitoring

VictoriaMetrics exposes its own metrics at /metrics, and the project ships Grafana dashboards.

vm_rows_inserted_total

Source: /metrics

Ingestion rate. Compare against expectation to detect a stalled scrape or an exporter outage.

vm_cache_entries{type="storage/tsid"}

Source: /metrics

Active series held in the index cache; the practical measure of cardinality pressure.

vm_slow_row_inserts_total

Source: /metrics

Rows that took the slow insert path, typically due to out-of-order timestamps or index misses.

vm_free_disk_space_bytes

Source: /metrics

Free space. Background merges need headroom; VictoriaMetrics switches to read-only when it runs low.

vm_rows_ignored_total

Source: /metrics

Samples rejected — for example past a cardinality limit. Any sustained rate is data you are not storing.

vmagent_remotewrite_pending_data_bytes

Source: vmagent /metrics

Buffered data waiting to be sent. Growth means the backend is unavailable or too slow.

Cardinality limits

-storage.maxHourlySeries=1000000
-storage.maxDailySeries=5000000

These reject new series past a threshold rather than letting the index grow without bound. Rejection is visible in vm_rows_ignored_total, which makes a cardinality explosion an alert instead of a gradual collapse. See Cardinality.