This is an old revision of the document!
Table of Contents
GPU Passthrough
GPU passthrough refers to configurations where a graphics card is reserved for use by a virtual machine. Making use of VT-d, IOMMU, and related technologies, the graphics card can be used at native performance.
This page first describes the process in general and then my specific system configurations.
Information
Hardware Requirements
The focus will be on Intel + NVIDIA systems, because I am unfamiliar with how GPU passthrough works with AMD processors and GPUs.
VT-d
The CPU, motherboard, and BIOS must support VT-d. Check ARK for CPU support. Note that K series (unlocked multiplier) CPUs historically did not have VT-d support, but Haswell Refresh and Skylake do.
IOMMU Groups and ACS
The guest graphics card needs proper IOMMU isolation from other devices on the PCIe bus. It should be alone in its own IOMMU group. This ensures no DMA transfers will occur between devices without being translated by the IOMMU.
If you have a Xeon E5 processor, it will probably support ACS. Your PCIe devices will be isolated in their own IOMMU groups. Some socket 2011 CPUs also support ACS (according to the datasheets).
If your processor doesn't support ACS, but your chipset is affected by the root port isolation quirk, then your graphics cards might still be isolated. You can use the iommu_groups.sh script to view what devices are in each group:
Graphics Cards
You must have at least one GPU for the host and one GPU for the guest.
You may use an IGP (CPU integrated graphics) for the host, but you cannot pass it through to the guest. If you use an IGP for the host and a guest graphics card that doesn't have a UEFI firmware, then you must apply the vga arbitration kernel patch. If you use a guest graphics card that has a UEFI firmware, then you can use OVMF virtual machine firmware and not apply the vga arbitration patch.
| Host Device | Guest Device | Guest Device UEFI-enabled | Result |
|---|---|---|---|
| GPU | GPU | Yes or No | OK |
| GPU | IGP | Yes or No | Invalid |
| IGP | GPU | No | Requires VGA arbitration kernel patch |
| IGP | GPU | Yes | Use OVMF firmware |
Virtualization is fully supported by NVIDIA Quadro cards. GeForce cards will work, but require that the virtualization be hidden from NVIDIA drivers.
GeForce 600 series cards probably did not ship with a UEFI firmware, but one can be flashed onto them. Check the manufacturer's site for an update, or use GOPupd.
Xeon E3-1275
Hardware
Hardware Configuration:
- Asus P8B WS
- Intel E3-1275 (Sandy Bridge)
- 32GB Kingston ECC Unbuffered
- EVGA NVIDIA GTX 570
- [Testing: NVIDIA 8800 GT]
- [Purchase: Gigabyte 750 Ti GV-N75TOC-2GI]
- Asus Xonar DX
iommu_groups.sh
Shows pci devices grouped by IOMMU group
#!/bin/sh
BASE="/sys/kernel/iommu_groups"
for i in $(find $BASE -maxdepth 1 -mindepth 1 -type d); do
GROUP=$(basename $i)
echo "### GROUP $GROUP ###"
for j in $(find $i/devices -type l); do
DEV=$(basename $j)
echo -n " "
lspci -s $DEV
done
done
iommu_groups.sh output
This output corresponds to the following pci card configuration:
- Slot 1 (16x): Nvidia GeForce 570
- Slot 2 (16x): empty
- Slot 3 (1x): empty
- Slot 4 (16x): Asus Xonar DX
- Slot 5 (PCI): Silicon Image SATA Controller
- Slot 6 (16x): empty
Kernel
Because the C206 chipset is affected by the Intel root port isolation quirk (1, 2), both graphics cards are isolated in their own IOMMU group. The pcie_acs_override patch is not needed. (To check if the root port isolation quirk exists, grep dmesg for “Intel PCH root port ACS workaround enabled”.)
If we wanted to use the integrated graphics of the E3-1275, then we would need to either:
- Use the i914.enable_hd_vgaarb patch (and lose DRI).
- Use a guest GPU that supports UEFI and use the OVMF boot firmware to start the virtual machine. (OVMF can initialize the GPU using legacy-free UEFI, keeping DRI and bypassing VGA and the need for arbitration.)