NetApp ONTAP Performance Monitoring Commands Guide

2026年8月25日 LOG admin

When an ONTAP cluster slows down, the first question is always: is it CPU, network, or disk? NetApp's knowledge base answers that with a specific set of performance commands, each aimed at a different layer of the stack. This article explains the six essential ONTAP 9 performance monitoring commands - statistics show-periodic, sysstat -x 1, sysstat -M 1, statit, wafltop and qos statistics - and what each column actually tells you about your storage system's health.

This expanded guide goes further than the list: it shows how to capture a comparable sample before and after a change, which privilege level each command needs, how to read the columns that matter, and how to chain the commands into a CPU-versus-network-versus-disk triage workflow that gets you to a suspect in minutes instead of hours.

statistics show-periodic: Cluster-Wide Live View

cluster1::*> statistics show-periodic -node node1

Useful columns: cpu busy shows CPU usage across CPU domains; data columns show Ethernet throughput on all NICs of the node; cluster columns show how much data is flowing through the Cluster LIFs; disk columns show disk read/write activity. This is the best first command for a broad overview.

The command is best thought of as a dashboard rather than a diagnostic. It refreshes until you break out, which makes it ideal for watching what happens while you kick off a backup or a SnapMirror transfer. When you need a fixed window, or a narrower view, add the interval, iteration and object options:

cluster1::*> statistics show-periodic -node node1 -interval 2 -iterations 60
cluster1::*> statistics show-periodic -node node1 -object cpu -instance 0
cluster1::*> statistics show-periodic -node node1 -object disk
cluster1::*> statistics show-periodic -node node1 -object volume -instance vol_data

Read it in three passes. First look at cpu busy: if it sits below 50 percent while users complain, CPU is not the problem and you can skip sysstat -M entirely. Second, compare the data and cluster columns to the physical link speed of the ports involved — a 10GbE LIF carrying 9 Gbps is saturated, and no amount of tuning on the storage side will fix a full pipe. Third, look at the disk columns: sustained high read or write throughput combined with high CPU is the signature of a workload the cache cannot absorb. The command needs only normal admin privileges, so it is the one to run first on a production cluster.

statistics start, stop and show: Capturing a Sample

Live output disappears the moment it scrolls. When you need a comparable before-and-after record — or evidence to attach to a support case — use the sampling form of the same statistics subsystem:

cluster1::*> statistics start -node node1 -sample-id before_change
(wait 5 to 10 minutes)
cluster1::*> statistics stop -sample-id before_change
cluster1::*> statistics show -sample-id before_change -preset perf
cluster1::*> statistics show -sample-id before_change -filter "object=volume&counter=latency|iops"

Presets select a curated set of counters for a purpose: perf and perf-ontap for general system health, throughput for bandwidth-oriented objects, latency for response-time views, and storage for aggregate and disk objects. The discipline that matters is consistency: sample for the same duration, on the same node, with the same filter before and after a change, then compare. A five-minute sample during a quiet period compared against a five-minute sample during the nightly backup proves nothing.

sysstat -x 1: CPU and Consistency Points

cluster1::> node run local sysstat -x 1

Columns to watch: CPU tells overall CPU utilization; CP time is the time spent flushing a Consistency Point (CP) from RAM to disk - useful when troubleshooting back-to-back CPs; CP_Ty identifies CP types; Cache hit and Disk util show cache effectiveness and disk busy percentage. This command, statit and wafltop are node shell commands (the last three need advanced or diagnostic privilege, priv set diag).

The single most useful column is CP time. WAFL periodically flushes modified buffers to disk, and that flush is described as a CP. When the write rate exceeds what the aggregate can absorb, CPs arrive back to back and the system spends its time doing CP work instead of serving requests; the symptom on the client side is rising write latency with no obvious CPU headroom problem. A CP time figure consistently above a few hundred milliseconds per second of wall clock, combined with a low Cache hit, points away from CPU and towards the write path: too few spindles for the workload, an aggregate near capacity, a snapshot or SnapMirror job consuming the same disks, or a storage efficiency job rehydrating data.

Cache hit is worth reading as a trend rather than an absolute. A read cache hit rate that used to be 98 percent and is now 85 percent means the working set grew past the controller cache, or another workload started competing for it — a change in behaviour, not necessarily a fault.

sysstat -M 1: Per-Domain CPU Breakdown

cluster1::*> set diag
cluster1::*> node run -node netapp-01 sysstat -M 1

This view shows utilization of network CPU processing (Nwk_Exmpt) and Data Processing / WAFL (WAFL_Ex), plus per-CPU core utilization. If WAFL_Ex is saturated while Nwk_Exmpt is low, the bottleneck is in the data path (WAFL), not the network.

The value of the domain view is that it converts a single "CPU busy" number into an answer. ONTAP splits processor time across domains so that a saturated network path and a saturated storage path look different. As a rule of thumb:

High WAFL_Ex, low Nwk_Exmpt — the data path is the limit. Look at aggregate layout, snapshots, storage efficiency, and the mix of writes to disk.
High Nwk_Exmpt, normal WAFL_Ex — network processing is the limit. Check link utilisation, TCP retransmits, NIC offload settings and whether a single LIF is carrying all the traffic.
Both high — the node is genuinely busy, and the correct response is moving workloads between nodes or adding controllers, not tuning.
Both low but latency is high — the bottleneck is downstream: disk service time, a saturated backend, or a client-side problem.

statit: Per-Disk Latency and IOPS

Cluster::*> node run -node filer
filer> priv set advanced
filer*> statit -b
(wait 30 or more seconds)
filer*> statit -e

Key columns: ut% is the percentage of time the disk had I/O commands; xfers shows disk IOPS per drive (xfers = ureads + writes + cpreads + greads + gwrites); chain is the average I/O size in 4KB units; usecs is the average disk round-trip latency per 4K block. High usecs with high ut% identifies hot drives in an aggregate.

statit is the tool that answers "which disk is slow", and it collects data between the -b start and -e end, so the sample window is the time you leave between them. Two patterns are worth memorising. First, an aggregate where one drive shows 90 percent ut% and much higher usecs than its peers is not a busy aggregate but a misbalanced one — usually caused by an odd RAID group size, disks of mixed speed or capacity in the same group, or heavy read traffic landing on a few spindles. Second, uniformly high ut% across every disk in the aggregate means the aggregate itself is out of IOPS headroom; adding capacity, moving a volume, or reducing the write amplification (snapshots, dedupe, hash) is the fix. Rising chain values on reads indicate random rather than sequential access, which is normal for databases and pathological for backup streams.

For a cluster-level alternative that does not require shell access, use the statistics subsystem with the disk object:

cluster1::*> statistics show-periodic -object disk -instance 0a.00.0 -interval 5

wafltop and qos statistics

wafltop gives a real-time view of WAFL activity including file system operations and buffer usage. qos statistics commands (e.g. qos statistics volume latency show) expose per-volume latency and throughput, which is how you confirm which workload is suffering before you change anything. Combine these with Active IQ Unified Manager for threshold-based alerting and root-cause analysis.

Where sysstat -M tells you that the data path is saturated, wafltop helps you see what the data path is doing: which WAFL operations dominate, how buffer space is being consumed, and how many file system operations per second the node is handling. It is a diagnostic-privilege tool and it is most useful during a live incident — for example when CPU is high and you need to distinguish a legitimate workload from a snapshot storm, a rehydration job, or an unexpectedly chatty NFS client. Because it is interactive, capture output rather than reading it from a screenshot when you plan to attach evidence to a case.

qos statistics in Practice: Attributing Latency to a Workload

Once you know something is slow, the question becomes which volume, and which layer of that volume's path, is responsible. The qos statistics object answers both, because ONTAP already accounts for latency per workload.

cluster1::*> qos statistics volume latency show
cluster1::*> qos statistics volume performance show
cluster1::*> qos statistics workload latency show
cluster1::*> qos statistics volume latency show -iterations 30 -interval 5

The latency view decomposes response time into components — total latency alongside the network, cluster, data and disk contributions — which is the fastest way to place the blame. If total latency is high but the disk component is negligible, the delay is in the stack above the disks (network, cluster interconnect, or client-side). If the disk component dominates, statit is your next command. The performance view gives throughput and IOPS per volume, which tells you which workload owns the resources. Use these commands before and after any change, and remember that a volume showing high latency is not necessarily the cause: it may simply be the front of the queue for a shared aggregate.

Working the Problem: CPU, Network or Disk?

The six commands are most effective in a fixed order, because each one narrows the search space for the next:

statistics show-periodic -node <node> for the broad view — CPU busy, per-NIC throughput, cluster LIF throughput, disk activity.
If CPU busy is high, sysstat -M 1 to find which CPU domain is saturated. Data path or network path?
If a specific workload is slow rather than the whole node, qos statistics volume latency show to identify the volume and the latency component.
If the disk component dominates, statit for per-disk ut%, usecs and chain.
If the network component dominates, move down a layer: check LIF and port counters with network interface show -instance and ifstat, and confirm with an external bandwidth test between two clients so a switch problem does not masquerade as a storage problem.
In parallel, check the event log for anything the performance counters cannot show: event log show -severity warning. Disk firmware faults, link flaps, and cluster interconnect errors all produce symptoms that look like performance problems.

Two complementary articles on this site cover the network layer of that workflow: troubleshooting Ethernet ports with ifstat and monitoring the reachability of ONTAP network ports.

Common Bottleneck Patterns and Their Signatures

Write amplification / CP storm: high CPU and high CP time in sysstat -x, low Cache hit, high WAFL_Ex, disk ut% elevated. Usually snapshots, SnapMirror, deduplication or an aggregate that is too full.
Aggregate IOPS exhaustion: uniformly high ut% across all disks in statit with rising usecs and normal CPU. The fix is capacity, not tuning.
Network saturation: one data or cluster column pinned at line rate in statistics show-periodic, high Nwk_Exmpt, and rising latency without disk pressure.
Single hot volume: one volume dominates in qos statistics volume performance show. Either apply QoS to bound it or move it to its own aggregate.
Hardware fault pretending to be performance: warnings in event log show for disk, shelf, or link errors. Always read the events before tuning anything.

Baselining and Alerting with Active IQ Unified Manager

The commands above are snapshots of a moment. Active IQ Unified Manager collects the same counters continuously, so it can compare today's behaviour with the object's own history rather than with a static threshold — which is the only reliable way to find the volume that is slow for its own baseline but still inside an arbitrary global limit. A practical division of labour: use the CLI for the incident, and use Unified Manager (or Cloud Insights) for trending, capacity and performance forecasting, and for the alert that wakes someone up. Two habits make the pair work: capture a statistics start/stop sample immediately after any alert so the moment is preserved, and record the before-and-after numbers for every change so the next engineer can see what was tried.