VFIO GPU How To series, part 3 - Host configuration

For my setup I'm using a Fedora 21 system with the virt-preview yum repos to get the latest QEMU and libvirt support along with Gerd Hoffmann's firmware repo for the latest EDK2 OVMF builds.  I hope though that the majority of the setup throughout this howto series is mostly distribution agnostic, just make sure you're running a newer distribution with current kernels and tools.  Feel free to add comments for other distributions if something is markedly different.  

The first thing we need to do on the host is enable the IOMMU.  To do this, verify that IOMMU support is enabled in the host BIOS.  How to do this will be specific to your hardware/BIOS vendor.  If you can't find an option, don't fret, it may be tied to processor virtualization support.  If you're using an Intel processor, check http://ark.intel.com to verify that your processor supports VT-d before going any further.

Next we need to modify the kernel commandline to allow the kernel to enable IOMMU support.  This will be similar between distributions, but not identical.  On Fedora we need to edit /etc/sysconfig/grub.  Find the GRUB_CMDLINE_LINUX line and within the quotes add either intel_iommu=on or amd_iommu=on, depending on whether your platform is Intel or AMD.  You may also want to add the option iommu=pt, which sets the IOMMU into passthrough mode for host devices.  This reduces the overhead of the IOMMU for host owned devices, but also removes any protection the IOMMU may have provided again errant DMA from devices.  If you weren't using the IOMMU before, there's nothing lost.  Regardless of passthrough mode, the IOMMU will provide the same degree of isolation for assigned devices.

Save the system grub configuration file and use your distribution provided update scrips to apply this configuration to the boot-time grub config file.  On Fedora, the command is:

# grub2-mkconfig -o /etc/grub2.cfg

If your host system boots via UEFI, the correct target file is /etc/grub2-efi.cfg.

With these changes, reboot the system and verify that the IOMMU is enabled.  To do this, first verify that the kernel booted with the desired updates to the commandline.  We can check this using:

# cat /proc/cmdline

If the changes are not there, verify that you've booted the correct kernel or double check instructions specific to your distribution.  If they are there, then we next need to check that the IOMMU is actually functional.  The easiest way to do this is to check for IOMMU groups, which are setup by the IOMMU and will be used by VFIO for assignment.  To do this, run the following:

# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/4/devices/0000:00:1a.0
/sys/kernel/iommu_groups/5/devices/0000:00:1b.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.5
/sys/kernel/iommu_groups/8/devices/0000:00:1c.6
/sys/kernel/iommu_groups/9/devices/0000:00:1c.7
/sys/kernel/iommu_groups/9/devices/0000:05:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.2
/sys/kernel/iommu_groups/11/devices/0000:00:1f.3
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.1
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:04:00.0

If you get output like above, then the IOMMU is working.  If you do not get a list of devices, then something is wrong with the IOMMU configuration on your system, either not properly enabled or not supported by the hardware and you'll need to figure out the problem before moving forward.

This is also a good time to verify that we have the desired isolation via the IOMMU groups.  In the above example, there's a separate group per device except for the following groups: 1, 9, 11, and 12.  Group 1 includes:

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)

This includes the processor root port and my GeForce card.  This is a case where the processor root port does not provide isolation and is therefore included in the IOMMU group.  The host driver for the root port should remain in place, with only the two endpoint devices, the GPU itself and its companion audio function bound to vfio-pci.

Group 9 has a similar constraint, though in this case device 0000:00:1c.7 is not a root port, but a PCI bridge.  Since this is conventional PCI, the bridge and all of the devices behind it are grouped together.  Device 0000:05:00.0 is another bridge, so there's nothing assignable in the IOMMU group anyway.

Group 11 is composed of internal components, an ISA bridge, SATA controller, and SMBus device.  These are grouped because there's not ACS between the devices and therefore no isolation.  I don't plan to assign any of these devices anyway, so it's not an issue.

Group 12 includes only the functions of my second graphics card, so the grouping here is also reasonable and perfectly usable for our purposes.

If your grouping is not reasonable, or usable, you may be able to "fix" this by using the ACS override patch, but carefully consider the implications of doing this.  There is a potential for putting your data at risk.  Read my IOMMU groups article again to make sure you understand the issue.

Next we need to handle the problem that we only intend to use the discrete GPUs for guests, we do not want host drivers attaching to them.  This avoids issues with the host driver unbinding and re-binding to the device.  Generally this is only necessary for graphics cards, though I also throw in the companion audio function to keep the host desktop from getting confused which audio device to use.  We have a couple options for doing this.  The most common option is to use the pci-stub driver to claim these devices before native host drivers have the opportunity.  Fedora builds the pci-stub driver statically into the kernel, giving it loading priority over any loadable modules, simplifying this even further.  If your distro doesn't keep reading, we'll cover a similar scenario with vfio-pci.

The first step is to determine the PCI vendor and device IDs we need to bind to pci-stub.  For this we use lspci:

$ lspci -n -s 1:
01:00.0 0300: 10de:1381 (rev a2)
01:00.1 0403: 10de:0fbc (rev a1)
$ lspci -n -s 2:
02:00.0 0300: 1002:6611
02:00.1 0403: 1002:aab0

The Vendor:Device IDs for my GPUs and audio functions are therefore 10de:1381, 10de:0fbc, 1002:6611, and 1002:aab0.  From this, we can craft a new option to add to our kernel commandline using the same procedure as above for the IOMMU.  In this case the commandline addition looks like this:

pci-stub.ids=10de:1381,10de:0fbc,1002:6611,1002:aab0

After adding this to our grub configuration, using grub2-mkconfig, and rebooting, lspci -nnk for these devices should list pci-stub for the kernel driver in use.

A further trick we can use is to craft an ids list using the advanced parsing of PCI vendor and class attributes to create an option list that will claim any Nvidia or AMD GPU or audio device:

pci-stub.ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

If you're using kernel v4.1 or newer, the vfio-pci driver supports the same ids option so you can directly attach devices to vfio-pci and skip pci-stub.  vfio-pci is not generally built statically into the kernel, so we need to force it to be loaded early.  To do this on Fedora we need to setup the module options we want to use with modprobe.d.  I typically use a file named /etc/modprobe.d/local.conf for local, ie. system specific, configuration.  In this case, that file would include:

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

Next we need to ensure that dracut includes the necessary modules to load vfio-pci.  I therefore create /etc/dracut.conf.d/local.conf with the following:

add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"

(Note, the vfio_virqfd module only exists in kernel v4.1+)

Finally, we need to tell dracut to load vfio-pci first.  This is done by once again editing our grub config file and adding the option: rd.driver.pre=vfio-pci  Note that in this case we no longer use a pci-stub.ids option from grub, since we're replacing it with vfio-pci.  Regenerate the dracut initramfs with dracut -f --kver `uname -r` and reboot to see the effect (The --regenerate-all dracut option is also sometimes useful).

Another issue that users encounter when sequestering devices is what to do when there are multiple devices with the same vendor:device ID and some are intended to be used for the host.  Some users have found the xen-pciback module to be a suitable stand-in for pci-stub with the additional feature that the "hide" option for this module takes device addresses rather than device IDs.  I can't load this module on Fedora, so here's my solution that I like a bit better.

Create a small script, I've named mine /sbin/vfio-pci-override-vga.sh  It contains:

#!/bin/sh

for i in $(find /sys/devices/pci* -name boot_vga); do
        if [ $(cat $i) -eq 0 ]; then
                GPU=$(dirname $i)
                AUDIO=$(echo $GPU | sed -e "s/0$/1/")
                echo "vfio-pci" > $GPU/driver_override
                if [ -d $AUDIO ]; then
                        echo "vfio-pci" > $AUDIO/driver_override
                fi
        fi
done

modprobe -i vfio-pci

This script will find every non-boot VGA device in the system, use the driver_override feature introduced in kernel v3.16, and make vfio-pci the exclusive driver for that device.  If there's a companion audio device at function 1, it also gets a driver override.  We then modprobe the vfio-pci module, which will automatically bind to the devices we've specified.  Don't forget to make the script executable with chmod 755.  Now, in place of the options line in our modprobe.d file, we use the following:

install vfio-pci /sbin/vfio-pci-override-vga.sh

So we specify that to install the vfio-pci module, run the script we just wrote, which sets up our driver overrides and then loads the module, ignoring the install option (-i) to prevent a loop.  Finally, we need to tell dracut to include this script in the initramfs, so in addition to the add_drivers+= that we added above, add the following to /etc/dracut.conf.d/local.conf:

install_items+="/sbin/vfio-pci-override-vga.sh /usr/bin/find /usr/bin/dirname"

Note that the additional utilities required were found using lsinitrd and iteratively added to make the script work.  Regenerate the initramfs with dracut again and you should now have all the non-boot VGA devices and their companion audio functions bound to vfio-pci after reboot.  The primary graphics should load with the native host driver normally.  This method should work for any kernel version, and I think I'm going to switch my setup to use it since I wrote it up here.

Obviously a more simple script can be used to pick specific devices.  Here's an example that achieves the same result on my system:

#!/bin/sh

DEVS="0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1"

for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

(In this case the find and dirname binaries don't need to be included in the intramfs)

Another couple other bonuses for v4.1 and newer kernels is that by binding devices statically to vfio-pci, they will be placed into a low power state when not in use.  Before you get your hopes too high, this generally only saves a few watts and does not stop the fan.  v4.1 users with exclusively OVMF guests can also add an "options vfio-pci disable_vga=1" line to their modprobe.d which will cause vfio-pci to opt-out devices from vga arbitration if possible.  This prevents VGA arbitration from interfering with host devices, even in configurations like mine with multiple assigned GPUs.

If you're in the unfortunate situation of needing to use legacy VGA BIOS support for your assigned graphics cards and you have Intel host graphics using the i915 driver, this is also the point where you need to patch your host kernel for the i915 VGA arbitration fix.  Don't forget that to enable this patch you also need to pass the enable_hd_vgaarb=1 option to the i915 driver.  This is typically done via a modprobe.d options entry as discussed above.

At this point your system should be ready to use.  The IOMMU is enabled, the IOMMU groups have been verified, the VGA and audio functions for assignment have been bound to either vfio-pci or pci-stub for later use by libvirt, and we've enabled proper VGA arbitration support in the i915 driver if needed.  In the next part we'll actually install a VM, and maybe even attach a GPU to it.  Stay tuned.