KVM Forum 2014

The schedule is up for KVM Forum 2014.  It looks like I'll be talking about VFIO GPU assignment with OVFM on the afternoon of Tuesday, October 14th.  If you're in or around Düsseldorf, register for the conference and come see.  There's also a talk just before mine on KvmGT that should be interesting.

Upstream updates for August 28th 2014

qemu.git now includes the MTRR fixes that eliminate the long delay in guest reboot when using OVMF with an assigned device on Intel hardware that does not support IOMMU snoop control.

Fixes for Linux Radeon with 440FX guests

The DRM and Radeon drivers in Linux assume that there's always a parent device to the GPU.  We can break this assumption easily with either the 440FX or Q35 QEMU machine modules by attaching the GPU to the root bus.  This has been one of the problems drawing users to more complicated Q35 models which more accurately reflect the host hardware.  We can also fix the driver to avoid such assumptions:

[PATCH] drm: Test for PCI root bus to avoid NULL pointer dereference

[PATCH] radeon: Test for PCI root bus before assuming bus->self

Upstream updates for August 26th 2014

A couple updates relevant to Nvidia GeForce assignment:

QEMU

fe08275d is now in qemu.git, decoupling the primary Nvidia GPU device quirk from the x-vga=on option.  This means that an Nvidia GPU assigned to a legacy-free OVMF VM will now enable this quirk automatically.

libvirt

d0711642 is now in libvirt.git enabling libvirt support for the kvm=off QEMU cpu option.  To enable this in your XML, add this to your VM definition:

<domain type='kvm'...>
  <features>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  ...
</domain>

Dual VGA assignment, GeForce + Radeon

This is a little old, but it's a nice bit of eye candy to start things off.  On the left we have an Nvidia 8400GS and on the right an AMD HD5450.  These are both very, very low-end cards (but they're quite; fanless).  A big bottleneck here is the hard disk.  Two VMs exercising an old spinning rust drive can get pretty slow.  In future posts I'll show how we can improve this.

The script I used has evolved from what's shown in the video, so this will differ from what you can read off the screen.  Here's the current version:

#!/bin/sh -x

#PASS_AUDIO="yes"

GPU=$(lspci -D | grep VGA | grep -i geforce | awk '{print $1}')
AUDIO=$(lspci -D -s $(echo $GPU | colrm 12 1) | grep -i audio | awk '{print $1}')
LIBVIRT_GPU=pci_$(echo $GPU | tr ':' '_' | tr '.' '_')
LIBVIRT_AUDIO=pci_$(echo $AUDIO | tr ':' '_' | tr '.' '_')

virsh nodedev-detach $LIBVIRT_GPU
virsh nodedev-detach $LIBVIRT_AUDIO

DISK=/mnt/store/vm/win7-geforce

if [ -v PASS_AUDIO ]; then
AUDIO_CMD="-device vfio-pci,host=$AUDIO,addr=9.1"
fi

MEM=4096

HUGE="-mem-path /dev/hugepages"

NEED=$(( $MEM / 2 ))
TOTAL=$(grep HugePages_Total /proc/meminfo | awk '{print $NF}')
WANT=$(( $TOTAL + $NEED ))

echo $WANT > /proc/sys/vm/nr_hugepages

AVAIL=$(grep HugePages_Free /proc/meminfo | awk '{print $NF}')
if [ $AVAIL -lt $NEED ]; then
echo $TOTAL > /proc/sys/vm/nr_hugepages
HUGE=
fi

QEMU=/home/alwillia/local/bin/qemu-system-x86_64
#QEMU=/bin/qemu-kvm

QEMU_PA_SAMPLES=8192 QEMU_AUDIO_DRV=pa \
$QEMU \
-enable-kvm -rtc base=localtime \
-m $MEM $HUGE -smp sockets=1,cores=2 -cpu host,hv-time,kvm=off \
-vga none -nographic \
-monitor stdio -serial none -parallel none \
-nodefconfig \
-device intel-hda -device hda-output \
-netdev tap,id=br0,vhost=on \
-device virtio-net-pci,mac=02:12:34:56:78:91,netdev=br0 \
-drive file=$DISK,cache=none,if=none,id=drive0,aio=threads \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \
-device vfio-pci,host=$GPU,multifunction=on,addr=9.0,x-vga=on \
$AUDIO_CMD

TOTAL=$(grep HugePages_Total /proc/meminfo | awk '{print $NF}')
RELEASE=$(( $TOTAL - $NEED ))
echo $RELEASE > /proc/sys/vm/nr_hugepages

virsh nodedev-reattach $LIBVIRT_GPU
virsh nodedev-reattach $LIBVIRT_AUDIO

Much of this is similar to what you'll find in the ArchLinux BBS thread, but there are some differences.  First notice that I'm not using the vfio bind scripts found there, I'm using libvirt through virsh for that.  Next, note that I'm not using the QEMU Q35 chipset model.  The importance of Q35 has been largely exaggerated.  If you are mostly concerned with assigning GPUs to Windows guests, Windows seems to be perfectly happy using the default 440FX chipset model.  Linux guests won't like this, particularly with Radeon cards because the driver blindly attempts to poke at the upstream PCIe root port.

On the QEMU side of things, everything should be included in QEMU 2.1.  The last required piece was the kvm=off cpu option, which is necessary for Nvidia 340+ guest drivers.  On the kernel side, this setup requires only the i915 patch since IGD is my host graphics.  This means DRI is disabled on the center, host monitor and using i915.enable_hd_vgaarb=1 on the kernel commandline.  The GeForce and Radeon devices are bound to pci-stub using kernel commandline options, ex. pci-stub.ids=10de:0e0f,1002:aab0,10de:1280,1002:6611  The only other kernel option necessary is intel_iommu=on.  I do not require the PCIe ACS override patch because one card is installed in a slot from the processor-based (PEG) root port and the other is installed in a slot from the PCH root port.  I find many users on the ArchLinux thread using the option vfio_iommu_type1.allow_unsafe_interrupts=1.  In most cases this is entirely unnecessary since most new processors are going to support VT-d2 and therefore have interrupt remapping support.  AMD IOMMU users have always had hardware support for interrupt remapping and any recent kernel can be configured to enable it.

For mouse and keyboard in the guest, I use synergy.