NetApp ONTAP Performance Monitoring Commands Guide

When a NetApp cluster starts feeling slow, the fastest way to separate storage latency from client-side issues is the ONTAP command line. This article walks through the official NetApp knowledge base set of performance-monitoring commands — statistics show-periodic, sysstat, statit and wafltop — and explains which columns to read for CPU, disk IOPS and per-drive latency problems in ONTAP 9.

Everything below is read-only. None of these commands change configuration, so they are safe to run on a production node while users are complaining. The goal is not to memorise every column, but to know the small set of counters that answer three questions: is the node CPU-bound, is the aggregate disk-bound, and is a single drive misbehaving?

Statistics Show-Periodic: Cluster-Wide View

statistics show-periodic -node node1
statistics show-periodic -node node1 -iterations 20
statistics show-periodic -node node1 -object cpu
statistics show-periodic -node node1 -object nfs

Watch the cpu busy column for CPU utilization across domains, the data columns for Ethernet throughput across all NICs, and the cluster columns for Cluster LIF traffic. This is the first command to run when throughput numbers look wrong.

The output is grouped by domain: cpu for busy percentages per CPU, data for the data LIFs, cluster for inter-node traffic, mgmt for the management interfaces. Reading it in one screen is the whole point — if the cpu busy figure is high but the data throughput is low, you are looking at a processing bottleneck (WAFL, protocol work, or a consistency-point storm) rather than a network bottleneck. If throughput is high and CPU is idle, the clients are simply pushing as much as they can and the array is not the constraint.

Adding -object cpu narrows the output to the CPU table only, which is convenient when you are tailing the command in a terminal with limited height. Adding -object nfs, -object cifs or -object iscsi switches the view to protocol-level operation counts, which is how you prove that a particular workload — not the whole cluster — is driving the load. Because the default is a one-second refresh with no end, remember to press Ctrl+C to stop it; if you are collecting data for a case, prefer -iterations so the command terminates by itself.

Sysstat: Per-Second CPU and Consistency Points

sysstat -x 1
sysstat -M 1
sysstat -u 1

sysstat -x 1 refreshes every second: CPU shows utilization, CP time shows time flushing consistency points (CPs) from RAM to disk, and CP_Ty is worth watching when you see back-to-back CPs — the classic sign of dirty-buffer pressure. sysstat -M 1 adds the Nwk_Exmpt (network CPU processing) and WAFL_Ex (data processing/WAFL) columns. Note that sysstat, statit and wafltop are node shell commands requiring diagnostic privilege (priv set diag).

The single most useful pattern in sysstat is the pair of CP rows. A healthy node produces CPs at a lazy, fairly regular rhythm and spends a small fraction of each second in CP time. When write pressure builds, ONTAP is forced to flush dirty buffers more often: CPs become back-to-back, the CP interval shrinks, and CP time climbs. By the time you see back-to-back CPs the array is already throttling incoming writes, and clients will report latency that has nothing to do with their own servers.

sysstat -u 1 is the condensed form many administrators prefer: it collapses the per-CPU detail into utilisation and service-time figures, which makes it easier to read on a wide terminal or over a slow SSH session. Whichever variant you use, capture the output to a file (sysstat -x 1 > /tmp/sysstat.txt) so the numbers can be attached to a support case instead of being retyped.

Statit: Per-Drive IOPS and Latency Breakdown

statit
statit -b
statit -e

Statit shows ut% (percentage of time the disk had I/O commands outstanding), xfers (average IOPS per drive), chain (I/O size in 4 kB units — 1 chain = 4 kB, 2 chains = 8 kB) and usecs (microsecond latency per IOP type per drive). A single drive with high usecs while others stay low points at a failing disk; uniform high latency points at a CP or queueing problem.

The disk section is where a bad drive announces itself long before it is marked failed. Look for one row whose usecs value is an order of magnitude higher than its neighbours in the same RAID group, or a disk whose ut% is pinned near 100 while the rest of the group sits far lower. That asymmetry is the signature of a drive doing internal retries; the array has not failed it yet, but the RAID group is paying for it on every write.

The disk section also carries per-drive read/write split columns (rd and wr), which tell you whether a particular workload is read-heavy or write-heavy at the physical layer. This matters when sizing Flash Pool or FabricPool later: a workload that shows a few hundred read IOPS per drive and almost no writes is a good SSD caching candidate, whereas a write-dominated pattern needs write cache and free space to absorb CP flushes.

Wafltop and QoS Statistics

wafltop -i 5
qos statistics volume performance show -interval 1
qos statistics volume latency show -interval 1
qos statistics workload performance show -interval 1

For workload-level insight use wafltop (per-workload CPU/disk activity) and qos statistics for Adaptive QoS counters. When performance is degraded, correlate these with aggregate space usage — a nearly full aggregate can silently throttle writes — and review ONTAP network configuration best practices if the bottleneck looks like it is on the wire instead of the disk.

wafltop -i 5 samples every five seconds and ranks the top consumers of WAFL processing time. Because it aggregates by workload rather than by volume, it answers the "who is hurting me" question directly: you can see that a single backup stream, a virus scan, or one NFS client is responsible for the majority of the node's processing time. Pair that output with qos statistics volume latency show for the same interval and you can state, with evidence, both the culprit and the damage.

A Practical Triage Order

Running all of the above at once produces more numbers than most people can read. A reliable order is:

1. sysstat -x 1 — confirm CPU and CP behaviour and decide whether the node is CPU-bound.
2. statistics show-periodic -node <node> — check whether traffic is arriving at all, and on which interfaces.
3. statit — confirm the disks are not the bottleneck, and look for one outlier drive.
4. qos statistics volume latency show -interval 1 — attribute what is left to a specific volume or workload.
5. wafltop — attribute the remaining CPU time to a specific workload.

Working in that order means each step eliminates a layer. If sysstat shows back-to-back CPs, there is little point studying per-drive latency: the disks are simply being asked to flush too much, and the fix is write-side (more free space, more spindles, or better caching), not read-side.

Collecting Data for NetApp Support

perfstat -cstat -d 2 -v -t 5
perf archive
node run -node node1 -command sysstat -x 1

When you open a case, NetApp support will normally ask for a perfstat and a perf archive bundle rather than a transcript pasted into a ticket. perfstat runs natively in the cluster shell and produces a compact, engineer-readable summary of CPU, WAFL, disk and network activity; perf archive then packages the underlying performance logs into a tarball that you can upload. Collecting these before you start changing anything is important — once you reboot, reconfigure or fail over, the interesting backlog is gone.

For node-shell-only commands such as sysstat, statit and wafltop, the cluster shell wrapper node run -node <nodename> -command "<command>" lets you stay in one session without manually entering the node shell first. This is the same mechanism ONTAP uses internally for the statistics subsystem, and it is the reason your diagnostic privilege may need to be escalated before those commands return anything.

Common Misreadings

• High CPU is not automatically a problem. A storage controller that is 60% busy while serving 80% of its rated throughput is doing exactly what it should; the metric matters only in relation to what the hardware can absorb.
• Low latency does not mean fast. A QoS counter can look healthy while the client is capped by a policy limit, because the limit is enforced before the request reaches the disks.
• chain is in 4 kB units, not kilobytes. A chain of 2 is an 8 kB I/O, not a 2 kB I/O; misreading this inverts the entire sequential-versus-random analysis.
• One slow drive is not always a failing drive. Confirm with a second sample before raising a hardware case; a transient rebuild or media scan can produce the same pattern for a few minutes.

Counter Reference Table

The following table summarises the counters that matter most when you are reading the commands above. Treat the "act when" column as a prompt to investigate, not as a hard threshold — every cluster has its own normal.

Two habits make this table far more useful. First, always capture a baseline on a quiet day; a number is only alarming relative to what the same cluster does when it is healthy. Second, record the sample interval alongside every number you save. A usecs figure averaged over an hour hides the three-second storm that users actually feel, and a support engineer who receives an unlabelled number will ask for the interval before anything else.

Performance Versus Space: The Connection People Miss

Performance investigations on NetApp frequently end not in the performance toolset but in the space toolset. An aggregate that is approaching full has to work harder for every write, because free space is what gives WAFL room to place new blocks and to satisfy snapshots before reclamation. The symptom appears in sysstat as growing CP time and in statit as rising write latency, which is why the same administrators who run statit also keep deduplication space errors and aggregate capacity on their watch list. If the counters here tell you the disks are working harder than the offered load justifies, check capacity before you order hardware.

Related Reading

Related: ONTAP dedup space errors explained.

原文链接：https://kb.netapp.com/on-prem/ontap/Perf/Perf-KBs/What_commands_are_useful_to_monitor_the_performance_in_ONTAP_9