SR-IOV Virtual Functions for KVM Guests

SR-IOV lets a single physical NIC present multiple independent virtual functions, each of which can be handed to a KVM guest as if it were a PCIe device. The performance gain over a virtual bridge is real — the packet path skips the host's software switch entirely — but so is the operational cost: VFs are statically provisioned, the host loses visibility into traffic once a VF is assigned, and MAC/VLAN behaviour is configured at the PF level from the host. This guide covers the enablement, the per-VF configuration that matters, the persistence problem and the verification steps.

Enabling VFs on the physical function

# check current state
ip -br link show ens785f0
cat /sys/class/net/ens785f0/device/sriov_totalvfs

# enable 8 VFs at runtime
echo 8 > /sys/class/net/ens785f0/device/sriov_numvfs
lspci | grep -i "virtual function"

Runtime enablement disappears on reboot. For persistence, create a systemd unit or a driver option; on Mellanox adapters the alternative is a module parameter, and both approaches should be validated after a reboot rather than assumed. Note that VF probing consumes adapter resources, so environments that do not monitor VFs can disable autoprobe via the PCI sriov_drivers_autoprobe interface on modern kernels.

Per-VF MAC, VLAN and rate limiting

# assign a MAC and VLAN (VST mode) to VF 0
ip link set dev ens785f0 vf 0 mac 52:54:00:aa:bb:01
ip link set dev ens785f0 vf 0 vlan 100 qos 3
ip link set dev ens785f0 vf 0 spoofchk on
ip link set dev ens785f0 vf 0 trust off
ip link set dev ens785f0 vf 0 max_tx_rate 2000

# back to VGT (VM controls VLAN tagging itself)
ip link set dev ens785f0 vf 0 vlan 0

# inspect what the PF thinks is configured
ip link show ens785f0
cat /sys/class/net/ens785f0/device/sriov/0/stats

Three settings do most of the work in production. VLAN (VST): with a VLAN set, the host tags and untags traffic for the guest, so the VM never sees or sets tags — ideal for untrusted guests and for consistent policy. Setting VLAN 0 returns control to the guest (VGT), which is what you want for trunk-style workloads that tag their own traffic. Spoof check: keeps the guest from transmitting frames with a MAC it does not own; turn it off only for legitimate bridging inside the guest, and understand that you are disabling an anti-spoofing control. Trust: controls whether the guest's own QoS markings are honoured — with trust off, the host's policy applies, which is the safer default on multi-tenant hosts.

Attaching a VF to a libvirt guest

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x41' slot='0x10' function='0x1'/>
  </source>
  <driver name='vfio'/>
</hostdev>

Identify the VF's PCI address with lspci or virsh nodedev-list --cap pci | grep and hand the exact address to the domain XML. Inside the guest, the interface appears as a normal NIC with the MAC you assigned on the host (check with lspci | grep -i mellanox inside the VM as well). If the guest does not see it, confirm vfio-pci is bound on the host rather than the netdev driver.

Verification and diagnostics

# host side
ip link show ens785f0 | grep -A1 "vf 0"
cat /sys/class/net/ens785f0/device/sriov/0/stats
ethtool -S ens785f0 | grep -i vf

# guest side
ip -br link
ethtool -i ens5
ethtool -S ens5 | head

# throughput sanity check
iperf3 -c 10.0.0.20 -P 4 -t 30

The host's sriov/<n>/stats file is the only place you can see per-VF counters once the VF is assigned — traffic no longer traverses the host stack, so tcpdump on the host interface will show nothing. That is the trade-off to plan for operationally: monitoring, firewalling and QoS that relied on the host now have to move to the switch port or into the guest. Rate limiting per VF (max_tx_rate) is the replacement for host-based shaping, and it is enforced in hardware, which means a guest cannot evade it.

Sizing: how many VFs, and which ones

The number of virtual functions is a hardware property, and it is not a free resource: each VF consumes adapter resources and consumes a PCIe function number on the host. The right number is driven by what you intend to attach, not by the maximum the card can present. Turning on sixty-four VFs to give out six creates a host with a large inventory of idle PCI devices, more lspci noise, and slightly more memory reserved for device structures.

Keep a small reserve. Reassigning a VF requires touching the running guest configuration, and having two unassigned VFs means a replacement VM can be brought up without reconfiguring an existing one. On multi-tenant hosts, decide the isolation model up front: SR-IOV gives the guest a hardware device with no hypervisor-mediated filtering (unless you enable it in hardware), so it belongs on hosts where you trust the tenants, or you need to enable the switch-side policy that enforces it.

Making the configuration survive a reboot

Everything you set with ip link set or by writing to sysfs is lost on reboot, which means a host can come back from maintenance with no VFs at all and a failed guest start. Choose one place to define the state and make it idempotent.

# option 1: systemd unit that recreates VFs and their attributes
# /etc/systemd/system/sriov-vfs.service
[Unit]
Description=Configure SR-IOV virtual functions for ens785f0
After=network-pre.target
Before=libvirtd.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/sriov-setup.sh
[Install]
WantedBy=multi-user.target

# /usr/local/sbin/sriov-setup.sh
#!/bin/bash
set -e
echo 8 > /sys/class/net/ens785f0/device/sriov_numvfs
ip link set dev ens785f0 vf 0 mac 52:54:00:aa:bb:01
ip link set dev ens785f0 vf 0 vlan 100
ip link set dev ens785f0 vf 0 spoofchk on
ip link set dev ens785f0 vf 0 trust off
ip link set dev ens785f0 vf 0 max_tx_rate 2000
systemctl enable sriov-vfs.service

Option two is a NetworkManager or distribution-specific mechanism, and option three is a driver parameter for the HCA itself. Whichever you use, validate it the only way that counts: reboot the host and check the VF count, the MAC assignments and the guest's ability to start. A configuration that was never tested across a reboot is a future outage scheduled for somebody else's maintenance window.

NUMA, performance and the checks that matter

# which NUMA node hosts the card, and where the guest's memory and vCPUs live
cat /sys/class/net/ens785f0/device/numa_node
lscpu | grep -i numa
virsh dumpxml guest1 | grep -A6 "numatune\|topology"

# confirm the VF landed on the right adapter and is bound to vfio
lspci -nn -s 41:10.1
lspci -k -s 41:10.1 | grep -i driver

Cross-NUMA traffic is the classic silent performance problem with SR-IOV: the guest runs on socket 0, the NIC sits on socket 1, and every packet crosses the interconnect. Pin the guest's vCPUs and memory to the socket that hosts the card, and verify rather than assume — the difference shows up as a consistent latency penalty that no amount of tuning elsewhere removes. If the guest sees the interface but throughput is far below the line rate, check the negotiated link speed and the VF's rate limit before looking at the guest's own networking: a max_tx_rate left at a low value from a test is a common leftover.

FAQ

Can a VF be shared between guests? A VF is a PCIe function and belongs to one guest at a time. Sharing requires a virtual switch (bridge, OVS) or a switchdev-based representation, which changes the performance characteristics you chose SR-IOV for.

Can I do live migration with an assigned VF? Not with a plain VF assignment — the destination host must present an equivalent device, which means matching hardware and configuration. For migratable workloads, use a paravirtual interface in the migration path and VF attachment for the performance-critical ones.

What about bonding inside the guest? Bonding two VFs inside a guest is useful for redundancy across two physical cards, but note that the host cannot see or influence the failover. Test it under load; some bonding modes behave poorly when the underlying device is a VF.

How do I monitor a VF's traffic from the host? Use the per-VF statistics under /sys/class/net/<pf>/device/sriov/<n>/stats. Packet capture on the host will not see VF traffic, so build monitoring around those counters or scrape them into your metric system before the workload goes live.

Quick reference: the sysfs and ip-link surface

# inventory
cat /sys/class/net/ens785f0/device/sriov_totalvfs     # maximum VFs
cat /sys/class/net/ens785f0/device/sriov_numvfs       # currently enabled
ip link show ens785f0 | grep -c "vf "                 # per-VF view from the host

# per-VF controls (host side)
ip link set ens785f0 vf 0 mac 52:54:00:aa:bb:01
ip link set ens785f0 vf 0 vlan 100 qos 3
ip link set ens785f0 vf 0 spoofchk on|off
ip link set ens785f0 vf 0 trust on|off
ip link set ens785f0 vf 0 max_tx_rate 2000
ip link set ens785f0 vf 0 rate 2000 min_tx_rate 500

# per-VF statistics once assigned to a guest
cat /sys/class/net/ens785f0/device/sriov/0/stats

The controls you set here are the only ones the host retains visibility into, so treat them as your monitoring surface: if a guest misbehaves and you have neither per-VF counters scraped into metrics nor switch-side telemetry, you have no way to attribute the problem. Set them deliberately at provisioning time rather than adjusting them during an investigation.

Related Reading

Related reading: KVM and libvirt: VM Management with virsh, SR-IOV, Multus and DPDK: CNF Networking on Kubernetes, Linux NIC Ring Buffer and Coalescing Tuning

原文链接：https://docs.nvidia.com/networking/display/mlnxofedv451010/single+root+io+virtualization+(sr-iov)