Monday, August 25, 2014

Primary graphics assignment without VGA

We really have a love-hate relationship with VGA when talking about graphics assignment.  We love that it initializes our cards and provides a standard interface, but we hate all the baggage that it brings along.  There is however an alternative emerging, UEFI by way of OVMF.  UEFI is a legacy-free firmware for PCs (among other things) that aims to replace the BIOS.  Let me repeat that, "legacy-free".  It doesn't get any more legacy than VGA.

So how do we get primary graphics without VGA?  Well, assuming your graphics card isn't too terribly old, it probably already contains support for both UEFI and VBIOS in the ROM and OVMF will use that new entry point to initialize the card.  There are some however some additional restrictions in going this route.  First, the guest operating system needs to support UEFI.  This means a relatively recent version of Linux, Windows 8+, or some of the newer Windows Server versions.  A gaming platform is often the target for "enthusiast" use of GPU assignment, so Windows 8/8.1 is probably a good target if you can bear the user interface long enough to start a game.  AFAIK, Windows 7 does not support UEFI natively, requiring the CSM (Compatibility Support Module) which I believe defeats the purpose of using UEFI.  If one of these guests does not meet your needs, turn away now.

Next up, UEFI doesn't (yet) support the Q35 chipset model.  In a previous post I showed Windows 7 happily running on a 440FX QEMU machine with both GeForce and Radeon graphics.  The same is true for Windows 8/8.1 here.  Linux isn't so happy about this though.  The radeon driver in particular will oops blindly looking for an downstream port above the graphics card.  You may or may not have better luck with Nvidia, neither nouveau or nvidia play nice with my GT635.  The radeon driver problem may be fixable without Q35, but it needs further investigation.  fglrx is untested.  Therefore, if Windows 8/8.1 is an acceptable guest or you're willing to help test or make Linux guests work, let's move on, otherwise turn back now.

Ok, you're still reading, let's get started.  First you need an OVMF binary.  You can build this from source using the TianoCore EDK2 tree, but it is a massive pain.  Therefore, I recommend using a pre-built binary, like the one Gerd Hoffmann provides.  With Gerd's repo setup (or one appropriate to your distribution), you can install the edk2.git-ovmf-x64 package, which gives us the OVMF-pure-efi.fd OVMF image.

Next create a basic libvirt VM using virt-manage or your favorite setup too.  We'll need to edit the domain XML before we install, so just point it to a pre-existing (empty) image or whatever gets it to the point that you can have the VM saved without installing it yet.  Also, don't assign any devices just yet.  Once there, edit the domain XML with virsh edit.  To start, we need to add a tag on the first line to make libvirt accept QEMU commandline options.  libvirt does not yet have support for natively handling the OVMF image, but it is under active development upstream, so these instructions may change quickly.  Make the first line of the XML look like this:

<domain type='kvm' xmlns:qemu=''>

The xmlns specification is the part that needs to be added.  Next we need to add OVMF, just before the </domain> close at the end of the file, add something like this:

    <qemu:arg value='-drive'/>
    <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/>

Adjust the path as necessary if you're using a different installation.  If you're using selinux, you may with to copy this file to /usr/share/qemu/ and run restorecon on it to setup permissions for QEMU to use it.

Save the XML and you should now be able to start the VM with OVMF.  From here, we can do the rest of the VM management using virt-manager.  You can add the GPU and audio as a standard PCI assigned device.  If you remove the "Graphics" (ie. VNC/Spice) and "Video" (ie. VGA/QXL/Cirrus) devices from the VM, the assigned GPU will be the primary display.

If you are not assigning the audio function or otherwise need to use the host audio for the guest, I recommend using QXL+Spice.  When the guest is running you can configure the QXL display off and use the connection only for sound.

UEFI purists will also note that I'm not providing a second flash drive for EFI variable storage.  In simple configurations this is unnecessary, but if you find that you've lost your boot options, this is the reason why.

In the video below I'm running two separate Windows 8.1 VMs using OVMF and managed through libvirt.  The one on the left is assigned a GeForce GT635, the one on the right a Radeon HD8570.  The host processor is an i5-3470T (dual-core + threads) and each VM is given 2 vCPUs, exposed as a single core with 2 threads.

Astute viewers will note that I'm using the version of the Nvidia driver which requires the kvm=off QEMU options.  There are two ways around this currently.  The first is to add additional qemu:commandline options:

<qemu:arg value='-cpu'/>
<qemu:arg value='host,hv_time,kvm=off'/>

We can do this because QEMU doesn't explode at the second instance of -cpu on the commandline, but it's not an ideal option.  The preferred method, and what I'm using, is support that is just going into libvirt which creates a new feature element for this purpose.  The syntax there is:

    <hidden state='on'/>

This will add kvm=off to the set of parameters that libvirt uses.

Also note that while Radeon doesn't seem to need any of the devices quirks previously enabled with the x-vga=on option, Nvidia still does.  This commit, which will soon be available in qemu.git, enable the necessary quirk anytime an Nvidia VGA device is assigned.

To summarize, for Radeon no non-upstream patches are required.  For Nvidia, libvirt will soon be updated for the hidden feature above, until then use the additional commandline option and QEMU will soon be updated to always quirk Nvidia VGA devices, until then the referenced commit will need to be applied.  Both cases currently need the qemu:commandline OVMF support, which should also be in libvirt soon.

In the video below, I'm running a stock Fedora 20 kernel.  The IGD device is used by the host with full DRI support (as noted by glschool running in the backgroups... which I assume doesn't work when DRI is disabled by using the i915 patch).  QEMU is qemu.git plus the above referenced Nvidia quirk enablement patch and  libvirt is patched with this, both of which have been accepted upstream and should appear in the development tree at any moment.  If you're running OVMF from a source other than above, make sure it includes commit b0bc24af from mid-June of this year.  If it doesn't, OVMF will abort when it finds an assigned device with a PCIe capability.  Also, you'll likely experience a long (~1 minute) delay on guest reboot.  This is due to lack of MTRR initialization on reboot, patches have been accepted upstream and will be in qemu.git shortly.



  1. As of this morning, the update to enable Nvidia quirks on any assigned Nvidia GPU is in qemu.git and the libvirt support for kvm=off is in libvirt.git, therefore the above can be done with only upstream bits.

  2. And now qemu.git includes the MTRR fix to solve the guest reboot delay experienced on some Intel hardware.

  3. I'd like to point out, that I had disable vgaarb in kernel to get it working - otherwise Xorg would disable DRI on host (at least with DRI2).

    1. Hmm, I wonder why I don't need to do that, or maybe DRI is still getting disabled and I just don't know how to exercise it. What can you not do when DRI is disabled? Do we need a kernel option to disable the VGA arbiter?


    I've tested a bit (both logs were generated consecutive on same system). Launching xorg via sddm (running as root) disables DRI on host, while launching same environment via startx (with user privileges) doesn't.

  5. This comment has been removed by the author.

  6. This comment has been removed by the author.

  7. Im thinking about trying this on gentoo.

    Should I work from the most recent posts to the oldest posts.

    Id like to ask a question about feasibility. Not that I was step by step. Do you think its possible to use this setup on a laptop with (vt-x and vt-d) and intel integrate video as well as an nvidia card with ovmf/uefi/kvm. Im picturing using the guest in windowed mode for setup and light use, but switching to steam streaming (for 3d) from the nvidia guest to the intel on the host since there is only one screen but two display adapters. Do you think this feasible? Is there a more elegant solution?

  8. OVMF is the way to go if you want to avoid patching your kernel, start from the later posts if your GPU and guest OS support UEFI.

    Dual-graphics laptops are tricky. There are no guarantees that any of this will work, but especially custom graphics cards on laptops. The discrete GPU may not be directly connected to any of the outputs, so "remoting" the graphics internally may be the only way to get to the guest desktop. It's possible that the GPU does not have a discrete ROM, instead hiding it in ACPI or elsewhere to be extracted. Some hybrid graphics laptops require custom drivers from the vendor. The more integration it has into the system, probably the less likely that it will behave like a discrete desktop GPU.

    1. You've got me curious. My new laptop is hybrid graphics however the discrete GPU (AMD Bonaire M6100) is a mini PCIe card. The integrated GPU is an HD 4600 (Haswell). There's HDMI and DisplayPort a well a VGA port.

      Since the AMd GPU is not connected directly to the ports. How likely would it be to "remote" this?

      Its too bad Haswell doesn't contain GVT-x to do this the "easy" way.

    2. Ideally, if I could start a KVM guest and pass the AMD GPU to the VM and see the display in a window - no VNC or Spice - that would be great, but I don't know if we can do this yet?

  9. Do you think this might be able to work with Bumblebee on a laptop?

    When you say UEFI support is required for the OS and the video card, do all the Video cards need to support UEFI or just the one(s) you will be attempting to use in the OVMF guest?

  10. This is the first time i am reading your post and admire that you posted article which gives users lot of information regarding particular topic thanks for this share.
    Gaming laptops South Africa

  11. During windows 8.1 installation setup crashes right after selecting the disk to install to. The next screen shows up and starts then crashed. Any ideas? Thanks!

    1. Install without an assigned GPU, add it later. I've posted an entire guide series for setting this up.

    2. I'm having the same issue without having any GPU passed through. I could also reproduce the issue without libvirt.

      Here's my qemu-call:

      qemu-system-x86_64 \
      -enable-kvm \
      -m 4096 \
      -cpu host,kvm=off \
      -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
      -hda /mnt/vms/hvm-win81.img \
      -cdrom /mnt/vms/win81.iso \

      Any ideas?

    3. File a bug with your distribution or upstream.

    4. not distribution-related (reproduced on F22 and Arch Linux)

      my guess would be latest OVMF (i'm using 2.5).

    5. yeah, definitely upstream. will file a bug there

      @Joe: I've just been successful by not using the latest OVMF but a previous build. You might want to give it a try.

      @Alex: BIG THANKS for this blog series! The Arch Forum has become to chaotic and your posts really helped me understand the latest developments (UEFY/OVMF vs. SeaBIOS+patching).

  12. @Dante Could you please tell me where to get this previous build from?

  13. Do I need to use grub-efi for host system to boot?
    Passthrough of ASUS Radeon R5 230 failed. Guest is Debian GNU/Linux Jessie. Kernel fails to insert radeon module printing stacktrace, however module is listed in lsmod.

    Passthrough of this card using BIOS and VBIOS works perfectly. Host video card is nVidia GTX 760. I plan to use IGD for host and create second virtual machine with nVidia and use IGD for host. Do I understand correctly that the only way to do this without patching Linux kernel is to use OVMF instead of BIOS/VBIOS?


Comments are not a support forum. For help with problems, please try the vfio-users mailing list (