VictoriaMetrics: Single-Node vs Cluster Deployment

VictoriaMetrics solves the same problem as Prometheus long-term storage, but in two very different shapes: a single binary that scales vertically, and a cluster of three component types that scales horizontally. Picking wrong is expensive — the cluster version is significantly harder to operate, while the single-node version quietly handles far more than most teams assume. This guide covers the decision, the architecture, a minimal cluster you can actually start, and the HA patterns the documentation recommends.

The Decision: When Single-Node Is Enough

The documented guidance is refreshingly blunt: use the single-node version for ingestion rates below roughly one million data points per second. It scales with CPU, RAM and disk, supports HA by running two identical instances, and is much easier to configure and operate. It can also discover and scrape Prometheus targets directly, so a small deployment needs nothing but Grafana in front of it.

Reach for the cluster version when you need horizontal scale beyond one machine, multi-tenancy with independent namespaces, or per-component independent scaling.

Cluster Architecture in Three Components

The critical property is the shared-nothing design: vmstorage nodes do not talk to each other and do not share data. That is what makes scaling and maintenance simple — but it also means the cluster's availability depends on having more than one node per service, and that a load balancer such as vmauth or nginx must sit in front of the read and write paths: requests starting with /insert route to port 8480 on vminsert, and /select to port 8481 on vmselect.

A Minimal Cluster

# vmstorage
vmstorage -storageDataPath=/var/lib/vmstorage -retentionPeriod=12            -httpListenAddr=:8482

# vminsert (points at the storage node)
vminsert -storageNode=vmstorage-host:8400 -httpListenAddr=:8480

# vmselect (also points at the storage node)
vmselect -storageNode=vmstorage-host:8400 -httpListenAddr=:8481

Every component needs a distinct -httpListenAddr if you are testing several of them on one host, and vmstorage needs distinct data-path and inter-node ports. The documentation's recommendation is to run at least two nodes per service so the cluster survives a single node being unavailable during an upgrade or hardware failure.

Getting Data In: vmagent and URL Formats

vmagent is the collector: it scrapes Prometheus targets, accepts push protocols (Prometheus remote write, Influx, OpenTSDB, Graphite, OpenTelemetry) and writes to VictoriaMetrics. The remote-write URL differs by deployment type, which is the detail people get wrong when migrating:

# single-node
-remoteWrite.url=http://vm-host:8428/api/v1/write

# cluster (tenant 0, Prometheus protocol)
-remoteWrite.url=http://vminsert-host:8480/insert/0/prometheus/api/v1/write

# cluster, multitenant writes via labels
-remoteWrite.url=http://vminsert-host:8480/insert/multitenant/prometheus/api/v1/write

Scraping can be sharded across multiple vmagent instances with -promscrape.cluster.membersCount plus a distinct -promscrape.cluster.memberNum per instance, and -promscrape.cluster.replicationFactor when you want the same target scraped by several agents for resilience.

High Availability Patterns

• Single-node HA: run two identically configured instances and point Grafana at one; if it dies, switch. Simple and well understood.
• Cluster HA: at least two nodes per component, with vmauth or nginx in front. Cluster mode already supports replication, so you do not need several -remoteWrite.url flags for the same cluster.
• Multi-AZ: the documentation does not recommend stretching one cluster across availability zones — cross-AZ latency and error rates hurt. Run an independent cluster per AZ and put vmagent in front to replicate into each, with additional vmselect nodes for cross-AZ reads.

Operating Notes

1. Keep all components of one cluster in the same subnet with high bandwidth and low latency.
2. Monitor ingestion health with the standard dashboards, and treat vm_rows_invalid_total on the server side as your parsing-error alarm — streaming ingest APIs may not return parse errors to the client.
3. Watch out for inflated time-series stats in cluster mode: vmselect sums per-node stats, so replicated or rerouted samples can be double-counted. Use /api/v1/status/tsdb with that caveat in mind.
4. Size retention deliberately (-retentionPeriod) and use vmbackup/vmrestore rather than copying data directories.

Sizing: Series Cardinality Beats Sample Rate

The documented guidance — single-node for ingestion below roughly one million data points per second — is a ceiling of the software, not a description of your workload. What actually determines whether one node stays comfortable is the number of active time series and the retention window, because VictoriaMetrics stores one index entry per series and one compressed row per (series, timestamp) pair.

Two rules follow. Before choosing a cluster to solve a cardinality problem, measure: /api/v1/status/tsdb on a single node reports the number of series, the labels with the highest cardinality and the series with the most label churn. A surprising share of "we need horizontal scale" tickets are really "someone put a request ID in a label". Second, memory scales with active series rather than disk, so budget roughly 10 KB of RAM per active series as a starting point and then watch process_resident_memory_bytes under real load instead of trusting any table, including this one.

Retention, Downsampling and Disk Layout

Retention is a single flag per instance — -retentionPeriod, in months when given as a bare number, or with h/d/w/y suffixes — and in cluster mode the same value applies to every tenant stored on a given vmstorage. That is a deliberate simplification: per-tenant retention is not something the cluster offers out of the box, so plan it by running separate clusters, or by splitting tenants across differently configured storage nodes.

# single node: two years of raw samples, refuse to fill the disk
victoria-metrics -storageDataPath=/var/lib/vm \
  -retentionPeriod=24 \
  -storage.minFreeDiskSpaceBytes=10GB

# cluster: set retention on every vmstorage, never on vminsert or vmselect
vmstorage -storageDataPath=/var/lib/vmstorage \
  -retentionPeriod=24 \
  -storage.minFreeDiskSpaceBytes=10GB

# rewrite older samples to coarser resolution to halve long-term disk usage
vmstorage -downsampling.period=30d:5m,180d:1h

The -storage.minFreeDiskSpaceBytes flag matters more than it looks: when free space drops below the threshold, the server refuses new samples rather than filling the disk and taking the whole instance down. For detection, alert on vm_free_disk_space_bytes well above that threshold. Downsampling is the other big lever, and it is not free: dashboards that depend on short spikes will look smoother after the fact, so enable it consciously and tell the people who own the latency graphs.

Migrating From Prometheus Without Losing History

There are three migrations people actually run, and they cost very different amounts of time and risk.

1. Cut the collector over. Keep Prometheus running, point a vmagent or a Prometheus remote_write block at VictoriaMetrics, and let new data land in the new system while the old one keeps serving queries. This is the low-risk path and the one to take when evaluating throughput.
2. Backfill existing data. Use vmctl to read a Prometheus TSDB and write it into VictoriaMetrics. Expect hours for hundreds of gigabytes, and a maintenance window if the source is under load.
3. Replace Prometheus entirely. Let vmagent scrape directly, delete the Prometheus deployment, and keep Alertmanager only if you want its routing and silences.

# backfill from a Prometheus snapshot directory
vmctl prometheus \
  --prom-snapshot=/prom-snapshots/20260914 \
  --vm-addr=http://vm-host:8428 \
  --vm-concurrency=8 \
  --vm-batch-size=200000

# verify series arrived
curl -s 'http://vm-host:8428/api/v1/series?match[]=up' | head -c 300

Watch the retention interaction carefully: ingesting two years of backfilled data into a system configured for 30 days of retention simply deletes most of it as it arrives. Set -retentionPeriod at the destination before the backfill starts, not after it finishes.

Backup, Restore and Disaster Recovery

Never copy the data directory of a running VictoriaMetrics process — the on-disk storage is not consistent while it is being written. Use the dedicated tools, which take a consistent view and produce a portable diff.

# full backup straight to object storage
vmbackup -storageDataPath=/var/lib/vmstorage \
  -snapshot.createURL=http://localhost:8482/snapshot/create \
  -dst=s3://my-bucket/vm-backups/latest

# incremental, using the previous backup as the origin
vmbackup -storageDataPath=/var/lib/vmstorage \
  -snapshot.createURL=http://localhost:8482/snapshot/create \
  -dst=s3://my-bucket/vm-backups/latest \
  -origin=s3://my-bucket/vm-backups/previous

# restore into an empty directory, then start the process
vmrestore -src=s3://my-bucket/vm-backups/latest \
  -storageDataPath=/var/lib/vmstorage

Note that vmselect and vminsert hold no durable state at all: a cluster's data lives entirely in the vmstorage directories, so the backup scope is the storage nodes only. The flip side is that losing a single vmstorage node loses the shard it owned. That is exactly why every component runs at least two nodes, and why -replicationFactor=2 on vminsert and vmselect is worth enabling before an incident rather than during one.

Tenancy and Isolation

Tenancy is the requirement that most often forces the cluster decision, so it is worth being precise. Single-node VictoriaMetrics is effectively single-tenant: one namespace, no per-tenant accounting. The cluster introduces account IDs into the request path — /insert/<accountID>/prometheus/api/v1/write and /select/<accountID>/prometheus/api/v1/query — and the standard dashboards, including storage-usage panels, become per-tenant.

# per-tenant writes using the URL path
curl -X POST 'http://vminsert:8480/insert/5/prometheus/api/v1/write' --data-binary @-

# the same tenant selected back out
curl 'http://vmselect:8481/select/5/prometheus/api/v1/query?query=up'

# or carry the tenant in a header instead of the path
curl -H 'AccountID: 5' -X POST 'http://vminsert:8480/insert/prometheus/api/v1/write' --data-binary @-

If the requirement is "customers must not see each other's metrics and I must be able to bill by series", the cluster is not optional. Put vmauth in front of it so the mapping from Basic Auth user to account ID happens in one place, rather than being re-derived by every client and every exporter.

相关阅读：Prometheus SNMP Exporter 监控网络设备、Prometheus Alertmanager 路由与静默配置、Grafana Loki 日志聚合与 LogQL 以及 Calico、Cilium 与 Flannel CNI 对比。

原文链接：VictoriaMetrics documentation - Cluster version