libvirt 1.2.19 - session mode device assignment

Just a quick note to point out a feature, actually bug fix, in the new libvirt 1.2.19 release:

hostdev: skip ACS check when using VFIO for device assignment (Laine Stump)

Why is this noteworthy?  Well, that ACS checking required access to PCI config space beyond the standard header, which is privileged.  That means that session (ie. user) mode libvirt couldn't do it and failed trying to support <hostdev> entries.  Now that libvirt recognizes that vfio enforces device isolation and a userspace ACS test is unnecessary, session mode libvirt can support device assignment!  Thanks Laine!

Note that a user still can't just pluck a device from the host and start using it, that's still privileged.  There's also the problem that a VM making use of device assignment needs to lock all of the VM memory into RAM, which is typically quite a lot more than the standard user locked memory limit of 64kB.  But these can be resolved by enabling the (trusted) user to lock memory sufficient for their VM and preparing the device for the user.  The keys to doing this are:

1. Use /etc/security/limits.conf to increase memlock for the desired user
2. Pre-bind the desired device to vfio-pci, either by the various mechanisms provided in other posts or simply using virsh nodedev-detach.
3. Change the ownership of the vfio group to that of the (trusted) user.  To determine the group, follow the links in sysfs or use virsh nodedev-dumpxml, for example:

      $ virsh nodedev-dumpxml pci_0000_00_19_0
      <device>
        <name>pci_0000_00_19_0</name>
        <path>/sys/devices/pci0000:00/0000:00:19.0</path>
        <parent>computer</parent>
        <driver>
          <name>e1000e</name>
        </driver>
        <capability type='pci'>
          <domain>0</domain>
          <bus>0</bus>
          <slot>25</slot>
          <function>0</function>
          <product id='0x1502'>82579LM Gigabit Network Connection</product>
          <vendor id='0x8086'>Intel Corporation</vendor>
          <iommuGroup number='4'>
            <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
          </iommuGroup>
        </capability>
      </device>

The iommuGroup sections tells us that this is group number 4, so permissions need to be set on /dev/vfio/4.  As always, also note the set of devices within this group and ensure that all endpoints listed are either bound to vfio-pci or pci-stub, the former will allow the user access to the device, the latter will allow the group to be usable without explicitly allowing the user access.

Enjoy!