GPU Passthrough on Proxmox for AI VMs

Why passthrough instead of containers

If you only need one AI workload at a time, run it on bare metal, VMs add nothing. The reason to use Proxmox + GPU passthrough is when you want strong isolation between multiple AI environments on the same hardware. Examples: a stable production Ollama VM that you never touch + an experimental VM where you break things weekly, or one VM for a model you trust and another for an untrusted model that gets unrestricted internet access.

Unlike Docker, a VM with PCIe passthrough has true hardware isolation. The CUDA driver inside the VM doesn't know it's virtualized. Performance loss vs bare metal is essentially zero. The cost is RAM (each VM needs its own kernel) and complexity (configuring IOMMU groups is fiddly).

Step 1: Enable IOMMU in BIOS and Proxmox

Passthrough requires the CPU's IOMMU (Intel calls it VT-d, AMD calls it AMD-Vi). Both are off by default in most consumer BIOSes. Turn it on, then enable IOMMU in the Linux kernel command line by editing /etc/default/grub. Reboot. Confirm with dmesg.

# In Proxmox shell, edit /etc/default/grub. Find GRUB_CMDLINE_LINUX_DEFAULT and change to:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
# For AMD: amd_iommu=on iommu=pt

# Apply and reboot
update-grub
reboot

# After reboot, confirm:
dmesg | grep -e DMAR -e IOMMU
# You want to see "IOMMU enabled" or similar.

Step 2: Check IOMMU groups

The IOMMU groups things together that share isolation boundaries, typically a PCIe slot and its connected devices. You can only pass through an entire group, not individual devices. Consumer boards often put the GPU in the same group as USB controllers or audio chips, which forces you to pass those through too (or use the ACS override patch). Run this script to inspect.

for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU group %s ' "$n"
  lspci -nns "${d##*/}"
done

# Look for your target GPU. The 'group N' number is what matters.
# Note all devices in that group, you'll bind them all to vfio-pci.

Step 3: Bind the GPU to vfio-pci

vfio-pci is the kernel module that lets a device be claimed by a VM. We tell the host kernel to grab the GPU's PCI IDs at boot, before nvidia.ko or amdgpu.ko can. The IDs come from lspci -nn (look for the hex like [10de:2684]).

# Find the IDs:
lspci -nn | grep -i nvidia
# 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation [10de:2684]
# 01:00.1 Audio device [0403]: NVIDIA Corporation [10de:22ba]

# Tell vfio-pci to claim them, edit /etc/modprobe.d/vfio.conf:
options vfio-pci ids=10de:2684,10de:22ba disable_vga=1

# Blacklist the proprietary driver on the HOST:
echo 'blacklist nvidia' >> /etc/modprobe.d/blacklist.conf
echo 'blacklist nouveau' >> /etc/modprobe.d/blacklist.conf

# Apply
update-initramfs -u -k all
reboot

# After reboot, verify:
lspci -nnk -d 10de:2684
# Kernel driver in use: vfio-pci   <- success

Step 4: Create a VM and attach the GPU

In Proxmox's web UI create a new VM, Ubuntu 22.04 ISO, q35 machine type, OVMF (UEFI) firmware. Boot Ubuntu and install normally without the GPU attached. After install, shut down the VM and add the GPU under Hardware → Add → PCI Device.

VM Hardware settings:
- Machine: q35
- BIOS: OVMF (UEFI)
- CPU: host (passes through CPU features the model needs)
- Memory: 32+ GB (no balloon)
- PCI Device: your GPU, with 'All Functions' and 'PCI-Express' boxes ticked

Then boot the VM. Install NVIDIA driver inside the VM normally:
  sudo ubuntu-drivers autoinstall
  reboot
  nvidia-smi   # GPU should show up just like bare metal

Step 5: Validate performance is bare-metal

After install, run a benchmark inside the VM (Ollama with llama3.1:8b is fine) and compare tokens/sec to the same workload on bare metal. They should be within 1-2%. If the VM is 10-20% slower, something is wrong, usually CPU pinning or vCPU type. Set CPU type to 'host' and pin vCPUs to physical cores.

# Inside the VM
ollama run llama3.1:8b --verbose "Write 200 words about anything."
# Note the eval rate (tokens/sec)

# On a 4090 you should see 75+ tok/s on llama3.1:8b Q4_K_M
# If you're getting 60-65, check Proxmox CPU type = host and pinning

Step 6: Snapshot before every experiment

The killer feature of VMs vs bare metal: snapshots. Before you try a sketchy install (custom CUDA build, experimental driver, untrusted model), snapshot the VM. If anything breaks, rollback is 5 seconds. This is the real reason to virtualize, not isolation per se but the ability to break things fearlessly. Treat your AI VMs as disposable; treat the Proxmox host as sacred.

Bonus: multiple VMs sharing one GPU (vGPU)

NVIDIA only allows true GPU virtualization (vGPU) on enterprise cards: A100, H100, and the older Tesla/Quadro line. Consumer RTX cards cannot do vGPU, one GPU = one VM at a time. There are unofficial drivers that enable vGPU on consumer cards, but their legality is questionable and they break with every driver update. For homelabs the cleaner pattern is one GPU per VM, with multiple GPUs in the host if you need multiple VMs running simultaneously.