Tuesday, May 5, 2015

VFIO GPU How To series, part 5 - A VGA-mode, SeaBIOS VM

For this example I'm show how to setup a VGA-mode VM using a Windows 7 guest and SeaBIOS.  In this case we are not dependent on the guest or the graphics card supporting UEFI, but with Intel host graphics, we do need to work around the Linux i915 driver's broken participation in VGA arbitration.  If you intend to use IGD for the host, the typical solution for this is to apply the i915 patch to the host kernel and use the enable_hd_vgaarb=1 option for the i915 driver to make it correctly participate in VGA arbitration.  The side-effect of doing this is the loss of DRI support in the host Xorg server.

However, since this is just a demo, I'm going to be lazy and modify my script from part 4 of this series to comment out the test for boot_vga being non-zero such that vfio-pci will be set as the driver override for all VGA devices in the system.  This limits my system to text-mode on the primary head for this example.  For a long term solution, I would either be looking for a UEFI/OVMF setup as outlined in part 4, or disabling IGD and using discrete graphics for the host.  Continuing to patch kernels for i915 VGA arbitration support is of course an option too, if you enjoy that sort of thing.  If you're not using IGD on the host, or like my example, avoiding the i915 driver, you should be ok.

Windows 7 VM installation is largely the same as Windows 8 installation in part 4.  The difference is that we're not going to modify the VM firmware to use UEFI, we'll leave that set to BIOS.  The other difference is that BIOS won't drop to a shell and allow us to manually boot from the install CD after we've broken the boot order by adding the virtio drivers.  My brute force solution to this is to attempt to boot the VM, it will error and say no OS found, force the VM off, then modify the boot options to push the Windows CDROM above the virtio disk and start the VM again.  I expect that you could also enable the boot menu on the first pass and try to catch the F12 prompt to select the correct boot device.

After installation, update, and installing TightVNC, as in part 4, shut down the VM, prune and tune the VM using virsh edit, just as we did in the previous setup.  If you're using Nvidia also add the KVM hidden section to features, removing the Hyper-V section and disable the Hyper-V clock source.  Also return to virt-manager and add the GPU and audio function to the VM.  All of this is documented in part 4.

Before you start the VM with the GPU, we first need to make a wrapper script around our qemu-kvm binary to insert the x-vga=on option.  To do this, create a file named /usr/libexec/qemu-kvm.vga with the following:

#!/bin/sh
exec /usr/libexec/qemu-kvm \
`echo "\$@" | \
sed 's|01:00.0|01:00.0,x-vga=on|g' | \
sed 's|02:00.0|02:00.0,x-vga=on|g'`

We'll use this as the executable libvirt uses in place of qemu-kvm directly.  Any time the qemu-kvm command line contains 01:00.0 or 02:00.0, which are the PCI addresses of the graphics cards in my system, we'll add the option ",x-vga=on".  This makes it transparent to libvirt that this is happening.  Be sure to chmod 755 the file before proceeding.

If your system is using selinux, libvirt will get an audit error trying to execute this script, so we'll need to add a new selinux module to allow for this.  Red Hat documentation here provides instructions for doing this.  In summary:

Set the selinux permissions for the file:
# restorecon /usr/libexec/qemu-kvm.vga

Create an selinux module
# cat > qemukvmvga.te << EOF
policy_module(qemukvmvga, 1.0)

gen_require(\`
 attribute virt_domain;
 type qemu_exec_t;
')

can_exec(virt_domain, qemu_exec_t)
EOF

Build the selinux module
# make -f /usr/share/selinux/devel/Makefile

Install selinux module
# semodule -i qemukvmvga.pp

With selinux happy, we next need to run virsh edit on the domain again.  Find the <emulator> tag and update the executable to point to our new script.  Save and exit the configuration and you should now be able to start the VM from virt-manager with VGA-mode enabled.  The same driver installation guidelines from part 4 apply for the GeForce/Catalyst drivers.  Enjoy.

12 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. I'm sorry, but this section makes very little sense to me, and simply does not work as written (under a fresh install of Fedora 22).

    For example, if I try to update the emulator tag, I get:

    error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 127: /usr/libexec/qemu-kvm.vga: line 2: /usr/libexec/qemu-kvm: No such file or directory

    I assume the script should actually point to /usr/bin/qemu-kvm ?

    Also:

    [erik@localhost ~]$ make -f /usr/share/selinux/devel/Makefile
    make: /usr/share/selinux/devel/Makefile: No such file or directory
    make: *** No rule to make target '/usr/share/selinux/devel/Makefile'. Stop.

    But I guess I'm not running selinux.

    Failed. Try again? [y,n,i,f,?]:

    ReplyDelete
    Replies
    1. I think you need to install selinux-policy to be able to create your own selinux modules and get make -f /usr/share/selinux/devel/Makefile to work

      Delete
    2. i have selinux-policy intalled, and make -f /usr/share/selinux/devel/Makefile dont work...

      make -f /usr/share/selinux/devel/Makefile
      make: /usr/share/selinux/devel/Makefile: No such file or directory
      make: *** No rule to make target '/usr/share/selinux/devel/Makefile'. Stop.

      Delete
    3. you need selinux-policy-devel to provide the devel/ directory and Makefile

      Delete
  3. This section doesen't seem to work for me either, using the script like this:

    #!/bin/sh
    exec /usr/libexec/qemu-kvm \
    `echo "\$@" | \
    sed 's|04:00.0|04:00.0,x-vga=on|g'`

    throws the following error:

    error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 127: /usr/libexec/qemu-kvm.vga: line 2: /usr/libexec/qemu-kvm: No such file or directory

    Failed. Try again? [y,n,f,?]:

    changing to /usr/bin/qemu-kvm like erik suggested leads to this error instead:

    error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 1: qemu-system-x86_64: -machine accel=kvm: could not open disk image : Could not open ' ': No such file or directory

    Failed. Try again? [y,n,f,?]:

    Am I missing something here ? :S

    ReplyDelete
    Replies
    1. The wrapper script must behave identically to the emulator you're replacing when run with -help

      Delete
  4. On ARCH, a user has reported success with a wrapper script placed in /sbin with the following content:

    #!/bin/bash
    exec /sbin/qemu-system-x86_64 \
    `/bin/echo "\$@" | \
    /bin/sed 's|01:00.0|01:00.0,x-vga=on|g'`

    The way ARCH invokes libvirt seems to require the explicit full paths on the helper binaries.

    ReplyDelete
  5. Hi! Thanks for this awesome tutorial series. I am THIS close to getting rid of windows as my main driver. However I still have some issues when exposing more than one core to the guest.

    I tried the current Arch kernel 4.2.1-1 as well its zen pendant. Base system is a 4790k (devil's canyon) with a gtx980. The system works just fine when giving it one core. Adding a second can lead to instabilities in some situations. Exposing 3 is already incredibly wonky and 4 will lead to a definite reset on windows boot. From what I read over the web this might be related to having to disable all the HyperV stuff (thanks NVIDIA). I can also stabilize the situation by throwing out my passthrough device (said 980 + its audio). I also experimented with different core/thread pinnings trying to take one thread from each core or 2 treads from 2 of the cores each but nothing seems to work. I am unsure if this is down to hardware problems on the processor or the mainboard (z97 based Asus Maximus VII Hero), or if there is some configuration option I might have missed, but right now it is not quite where it needs to be to ditch windows for gamins since one core is unfortunately not enough.
    In any case thank you again for this series, from all the documentation I absorbed on the web yours is definitely the most comprehensive one.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Is it possible to use x-vga=on with OVMF?

    ReplyDelete

Comments are not a support forum. For help with problems, please try the vfio-users mailing list (https://www.redhat.com/mailman/listinfo/vfio-users)