GPU Passthrough

GPU passthrough refers to configurations where a graphics card is reserved for use by a virtual machine. Making use of IOMMU, the graphics card can be used at native performance.

This page first describes the process in general and then my specific system configurations.

Information

Hardware Requirements

The focus will be on Intel + NVIDIA systems, because I am unfamiliar with how GPU passthrough works with AMD processors and GPUs.

VT-d

The CPU, motherboard, and BIOS must support VT-d. Check ARK for CPU support. Note that K series (unlocked multiplier) CPUs historically did not have VT-d support, but Haswell Refresh and Skylake do.

IOMMU Groups and ACS

The guest graphics card needs proper IOMMU isolation from other devices on the PCIe bus. It should be alone in its own IOMMU group. This ensures no DMA transfers will occur between devices without being translated by the IOMMU.

If you have a Xeon E5 processor, it will probably support ACS. Your PCIe devices will be isolated in their own IOMMU groups. Some socket 2011 CPUs also support ACS (according to the datasheets).

If your processor doesn't support ACS, but your chipset is affected by the root port isolation quirk, then your graphics cards might still be isolated. You can use find /sys/kernel/iommu_groups/ -type l or the iommu_groups.sh script to view what devices are in each group.

iommu_groups.sh

#!/bin/sh
 
# List the devices in each IOMMU group, from AW at
# https://bbs.archlinux.org/viewtopic.php?id=162768&p=29
 
BASE="/sys/kernel/iommu_groups"
 
for i in $(find $BASE -maxdepth 1 -mindepth 1 -type d); do
        GROUP=$(basename $i)
        echo "### Group $GROUP ###"
        for j in $(find $i/devices -type l); do
                DEV=$(basename $j)
                echo -n "    "
                lspci -s $DEV
        done
done

If your guest graphics card is not isolated, you can try moving it to a different slot on your motherboard to see if that slot has better isolation. Some slots are connected to the processor PCI lanes, which only have isolation if your processor supports ACS. Other slots are connected to the PCH PCI lanes, which may have isolation on root ports.

If you don't have isolation on any PCI slot and you don't need the devices that share the guest graphics card's IOMMU group, you can bind the extra device to either of the vfio-pci or pci-stub drivers.

As a last resort, you can try using the ACS override kernel patch. However, this does not create isolation. It merely hides the fact that isolation doesn't exist. Devices in the same real IOMMU group could still perform DMA transfers within another's address space.

Graphics Cards

You must have at least one GPU for the host and one GPU for the guest.

You may use an IGP (CPU integrated graphics) for the host, but you cannot pass it through to the guest. If you use an IGP for the host and a guest graphics card that doesn't have a UEFI firmware, then you must apply the vga arbitration kernel patch. If you use a guest graphics card that has a UEFI firmware, then you can use OVMF virtual machine firmware and not apply the vga arbitration patch.

Host Device	Guest Device	Guest Device UEFI-enabled	Result
GPU	GPU	Yes or No	OK
GPU	IGP	Yes or No	Invalid
IGP	GPU	No	Requires VGA arbitration kernel patch; DRI will be disabled
IGP	GPU	Yes	Use OVMF firmware; can still use DRI

Virtualization is fully supported by NVIDIA Quadro cards. GeForce cards will work, but require that the virtualization be hidden from NVIDIA drivers.

GeForce 600 series cards probably did not ship with a UEFI firmware, but one can be flashed onto them. Check the manufacturer's site for an update, or use GOPupd.

Xeon E3-1275

Hardware

Hardware Configuration:

Asus P8B WS
Intel E3-1275 (Sandy Bridge)
32GB Kingston ECC Unbuffered
EVGA NVIDIA GTX 570
[Testing: NVIDIA 8800 GT]
[Purchase: Gigabyte 750 Ti GV-N75TOC-2GI]
Asus Xonar DX

iommu_groups.sh

Shows pci devices grouped by IOMMU group

#!/bin/sh

BASE="/sys/kernel/iommu_groups"

for i in $(find $BASE -maxdepth 1 -mindepth 1 -type d); do
  GROUP=$(basename $i)
  echo "### GROUP $GROUP ###"
  for j in $(find $i/devices -type l); do
    DEV=$(basename $j)
    echo -n "  "
    lspci -s $DEV
  done
done

iommu_groups.sh output

This output corresponds to the following pci card configuration:

Slot 1 (16x): Nvidia GeForce 570
Slot 2 (16x): empty
Slot 3 (1x): empty
Slot 4 (16x): Asus Xonar DX
Slot 5 (PCI): Silicon Image SATA Controller
Slot 6 (16x): empty

iommu_groups.sh output

### GROUP 0 ###
  00:00.0 Host bridge: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller (rev 09)
### GROUP 1 ###
  00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
  01:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 570] (rev a1)
  01:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1)
### GROUP 2 ###
  00:02.0 Display controller: Intel Corporation Xeon E3-1200 Processor Family Integrated Graphics Controller (rev 09)
### GROUP 3 ###
  00:06.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
  02:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa)
  03:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
### GROUP 4 ###
  00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
### GROUP 5 ###
  00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
### GROUP 6 ###
  00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
### GROUP 7 ###
  00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
### GROUP 8 ###
  00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
### GROUP 9 ###
  00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
### GROUP 10 ###
  00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
### GROUP 11 ###
  00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
### GROUP 12 ###
  00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
### GROUP 13 ###
  00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
  09:00.0 SATA controller: Silicon Image, Inc. SiI 3512 [SATALink/SATARaid] Serial ATA Controller (rev 01)
  09:03.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)
### GROUP 14 ###
  00:1f.0 ISA bridge: Intel Corporation C206 Chipset Family LPC Controller (rev 05)
  00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
  00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
### GROUP 15 ###
  06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
### GROUP 16 ###
  07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
### GROUP 17 ###
  08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

Kernel

Because the C206 chipset is affected by the Intel root port isolation quirk (1, 2), both graphics cards are isolated in their own IOMMU group. The pcie_acs_override patch is not needed. (To check if the root port isolation quirk exists, grep dmesg for “Intel PCH root port ACS workaround enabled”.)

If we wanted to use the integrated graphics of the E3-1275, then we would need to either:

Use the i914.enable_hd_vgaarb patch (and lose DRI).
Use a guest GPU that supports UEFI and use the OVMF boot firmware to start the virtual machine. (OVMF can initialize the GPU using legacy-free UEFI, keeping DRI and bypassing VGA and the need for arbitration.)