Linux Kernel & Driver Matrix for AI Workstations

Why the Kernel Matters for AI Workloads

Most ML engineers treat the kernel as a black box, but when you are stitching together four RTX 5090s, an NPU, and a Sapphire Rapids host, the kernel becomes the single point of failure. AI inference is memory-bandwidth-bound, not compute-bound, so even a 5% overhead from interrupt handling or TLB misses translates to lost tokens per second. Worse, a misconfigured IOMMU group can prevent you from passing a GPU to a VM, or a missing firmware blob can silently downgrade your MI300X to CPU fallback. This article builds a driver matrix covering NVIDIA, AMD, Intel, and emerging NPU accelerators, with specific kernel versions, firmware requirements, and tuning parameters you need to know before building a production AI workstation.

The kernel driver stack for AI has four layers: the core kernel module (e.g., nvidia.ko, amdgpu.ko), the user-space runtime (CUDA, ROCm, Level Zero), the firmware (VBIOS, PSP, SMU), and the IOMMU/VFIO layer. Each layer has its own version dependencies. For example, NVIDIA's open kernel module (nvidia-open) requires kernel 5.13+ but is only production-ready for datacenter GPUs as of driver 545. AMD's amdgpu driver requires firmware from linux-firmware 20240220 for MI300X support. Intel's Xe driver for Battlemage is still experimental and lacks full AI compiler support.

We will cover each vendor's driver stack, common pitfalls like missing firmware or wrong kernel config, and provide a decision matrix for choosing between open and closed drivers. We'll also cover VFIO passthrough, hugepages, and NUMA tuning, because those are the knobs that separate a 50 token/s setup from a 120 token/s one.

NVIDIA: Proprietary vs. Open Kernel Module Showdown

NVIDIA's driver stack has bifurcated. The proprietary nvidia.ko (closed source) has been the gold standard for years, supporting all features (CUDA, NVLink, GPUDirect, MIG) and all GPUs from Kepler to Blackwell. The open kernel module (nvidia-open), first released in 2022, is now the default for datacenter GPUs (H100, A100, H200, B200) starting with R550 drivers. For consumer GPUs (RTX 4090, RTX 5090), NVIDIA still recommends the proprietary module, though the open module works for basic compute.

Why does this matter for AI? The open module integrates better with the upstream kernel (no tainted flag, better kABI compatibility) and allows custom kernel patches (e.g., for IOMMU groups). However, the open module lacks support for MIG (Multi-Instance GPU) on A100/H100, which is a dealbreaker if you want to partition a GPU for multi-tenant inference. The proprietary module also has better power management on consumer cards: the open module on RTX 4090 can cause higher idle power draw (30W vs 15W) due to missing PMU firmware calls.

For a practical recommendation: if you are building a multi-GPU workstation with H100s or B200s, use the open module with driver 550.120 or later. If you are using RTX 4090s or 5090s, stick with the proprietary module (driver 545 or 550) until NVIDIA certifies the open module for consumer cards. In both cases, ensure you have the correct firmware: the VBIOS for each card must support resizable BAR (ReBAR) for optimal PCIe bandwidth. On a 4x RTX 4090 setup with PCIe Gen 4, enabling ReBAR can improve throughput by 8-12% in memory-bound kernels like FlashAttention.

One more gotcha: the open module requires kernel 5.13+ with CONFIG_DRM_KMS_HELPER and CONFIG_DRM_FBDEV_EMULATION. If you build a custom kernel, do not forget those, or the module will fail to load with cryptic errors.

# Check if open kernel module is available and loaded
modinfo nvidia | grep version
lsmod | grep nvidia
# For open module, look for nvidia-open in /lib/modules/
# Recommended kernel parameters for multi-GPU:
# nvidia.NVreg_EnablePCIeGen3=1 nvidia.NVreg_UsePageAttributeTable=1
# Enable resizable BAR (check with dmesg | grep BAR)
sudo dmesg | grep -i 'resizable bar'

AMD: amdgpu, ROCm, and the Firmware Maze

AMD's amdgpu driver is fully open source (kernel side), but the firmware blobs are proprietary. This creates a split: the kernel driver is upstream, but you must install linux-firmware from a specific date to get support for MI300X, MI250X, or the upcoming Strix Halo NPU. For example, MI300X requires firmware from linux-firmware 20240220 or later. Without it, amdgpu will fall back to CPU compute for ROCm, silently degrading performance by 100x.

The amdgpu driver supports three modes: amdgpu.dc=1 (display), amdgpu.dc=0 (compute only), and amdgpu.ppfeaturemask=0xffffffff (enable all power features). For AI workstations without a display attached to AMD GPUs, set amdgpu.dc=0 to reduce power overhead and avoid display-related interrupts that can jitter inference latency. For multi-GPU setups, you must also set amdgpu.gpu_recovery=1 and amdgpu.lockup_timeout=10000 to prevent a single hung GPU from freezing the entire system.

ROCm 6.0+ supports MI300X and MI250X with full FP8 and FP16 matrix cores. However, the kernel driver must be compiled with CONFIG_HSA_AMD enabled. Most distro kernels have this, but custom kernels often miss it. Check /sys/kernel/debug/dri/0/amdgpu_gpu_recovery to verify.

For consumer Radeon GPUs (RX 7900 XTX), ROCm support is partial: you get HIP but not the full MIOpen library for LLM inference. Most AI builders avoid AMD for LLM inference because TensorRT-LLM and vLLM lack full ROCm support (as of 2026, vLLM has experimental ROCm backend but at 60-70% of CUDA throughput). If you must use AMD, use the MI series or wait for the CDNA 4 architecture.

The Strix Halo NPU (XDNA 2) uses a separate driver: amdxdna.ko, which landed in linux-next for kernel 6.10. This driver is still in staging and requires firmware from AMD's NPU firmware package. For AI workloads on NPU, expect limited support for PyTorch: only ONNX Runtime with QNN EP works reliably.

# Check amdgpu firmware version
sudo cat /sys/kernel/debug/dri/0/amdgpu_firmware_info
# Verify ROCm is using GPU not CPU fallback
rocminfo | grep 'Name:'
# Set kernel parameters for compute-only mode
# Add to /etc/default/grub:
# GRUB_CMDLINE_LINUX="amdgpu.dc=0 amdgpu.ppfeaturemask=0xffffffff"
sudo update-grub

Intel: i915, Xe, and the NPU Driver Saga

Intel's GPU driver stack is the most fragmented. The legacy i915 driver supports integrated graphics and Arc Alchemist (A770, A580) but is not optimized for AI compute. The newer Xe driver (for Battlemage and beyond) is designed for discrete GPUs but still lacks a complete compiler stack for oneAPI. For AI inference on Intel GPUs, most users still rely on i915 + Level Zero + oneAPI, but performance is 2-3x behind NVIDIA on LLM inference due to missing tensor core support.

The Intel NPU (e.g., Meteor Lake, Arrow Lake) uses a separate driver: intel_npu.ko, which landed in kernel 6.7. This driver is minimal: it exposes the NPU as a character device but requires user-space libraries (OpenVINO, intel-npu-accel-library) to run models. The NPU is only useful for lightweight ONNX models (< 1B parameters); for LLMs, it is not competitive.

For AI workstations, Intel GPUs are rarely chosen for primary compute, but they are common as integrated graphics on Sapphire Rapids and Granite Rapids hosts. If you have an Intel dGPU (Arc A770), you can run llama.cpp with the SYCL backend, but expect 30-40 tokens/s for Llama 3 8B Q4, compared to 100+ tokens/s on an RTX 4090. The Xe driver may improve this with future firmware updates, but as of 2026, Intel is not a serious contender for LLM inference.

One critical issue: the i915 driver has a bug with large BAR allocations (> 256 MB) on some Arc cards, causing system hangs during model loading. The workaround is to set i915.enable_guc=0 and i915.disable_display=1 in kernel parameters. This disables the GuC firmware and display engine, freeing memory for compute.

IOMMU Groups and VFIO Passthrough: The Multi-GPU Nightmare

If you plan to run multiple GPUs in a single workstation, IOMMU groups will be your biggest headache. Each PCIe device belongs to an IOMMU group, and VFIO passthrough requires that all devices in a group be passed together. On many platforms (especially consumer chipsets like Z790, X670E), GPUs are grouped with their PCIe root port, audio controller, and USB controller. This means you cannot pass a single GPU to a VM without also passing the entire group, which may include the NVMe drive or network card.

The solution is the ACS (Access Control Services) override patch or kernel parameter pci=acs_override. This forces each device into its own IOMMU group, but it is not upstream due to security concerns (it breaks isolation guarantees). For a single-user workstation, the security risk is acceptable. On Ubuntu, you can add pci=acs_override to GRUB_CMDLINE_LINUX. On Proxmox, you need to add it to /etc/default/grub and update-initramfs.

For multi-GPU inference servers (e.g., 4x RTX 5090), you also need to ensure each GPU has its own NUMA node. On dual-socket Sapphire Rapids, GPUs on socket 0 should be passed to processes pinned to socket 0, and GPUs on socket 1 to socket 1. Use numactl --cpunodebind=0 --membind=0 to bind inference processes. Failure to do so can cause 20-30% bandwidth loss due to cross-socket memory access.

VFIO also requires hugepages. For a 70B model in FP16 (140 GB VRAM across 4 GPUs), you need at least 64 GB of 1 GB hugepages for the host to avoid TLB misses. Set vm.nr_hugepages=65536 in /etc/sysctl.conf. Without hugepages, the kernel's page table walk for 4 KB pages adds 5-10% overhead to every memory access, which is catastrophic for bandwidth-bound kernels.

# Check IOMMU groups
for d in /sys/kernel/iommu_groups/*/devices/*; do
  echo $(basename $(dirname $d)) $(basename $d)
done
# Add ACS override to kernel parameters
echo 'GRUB_CMDLINE_LINUX="pci=acs_override"' >> /etc/default/grub
sudo update-grub
# Allocate 1 GB hugepages
echo 65536 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
# Verify hugepages
cat /proc/meminfo | grep HugePages

NUMA Balancing and Memory Management Tuning

The kernel's NUMA balancer (numa_balancing) is designed for workloads with unpredictable memory access patterns, but for AI inference, it is harmful. When you load a model into GPU VRAM, the host memory used for KV cache and intermediate buffers should stay pinned to the NUMA node closest to the GPU. NUMA balancing can migrate pages to other sockets, causing cross-socket bandwidth penalties. Disable it with sysctl kernel.numa_balancing=0.

Similarly, the kernel's OOM killer can kill your inference server if memory pressure is high. Use cgroup v2 memory limits to protect critical processes. For example, if you are running vLLM with 4 GPUs, create a cgroup with memory.max set to 80% of total RAM, and memory.high set to 70%. This prevents the OOM killer from targeting vLLM when other processes (e.g., a Python script that leaks memory) consume too much.

Another critical tunable is vm.swappiness. Set it to 0 to avoid swapping, which would destroy inference latency. Also set vm.vfs_cache_pressure=50 to reduce inode cache pressure, which can cause jitter when loading large model files.

For systems with 512 GB+ RAM (common for multi-GPU workstations), consider using ZONE_MOVABLE to prevent kernel allocations from fragmenting memory. Add kernel parameter movablecore=64G to reserve 64 GB for movable allocations. This is especially important when using hugepages, as the kernel's buddy allocator can fail to allocate contiguous 1 GB pages if memory is fragmented.

Firmware and Microcode: The Silent Performance Killers

Beyond kernel modules, firmware versions for GPUs, CPUs, and PCIe switches significantly affect AI performance. For NVIDIA GPUs, the VBIOS controls power limits, clock speeds, and resizable BAR. On RTX 4090, many aftermarket cards ship with a conservative VBIOS that limits power to 350W. Flashing a 450W VBIOS (e.g., from a water-cooled card) can boost token throughput by 10-15% for compute-bound models, but increases power draw and thermal load.

For AMD MI300X, the firmware includes PSP (Platform Security Processor) and SMU (System Management Unit) blobs. Outdated SMU firmware can cause the GPU to throttle at 80°C instead of 100°C, losing 20% performance. Check /sys/kernel/debug/dri/0/amdgpu_smu_info for current firmware version. Update firmware by installing the latest linux-firmware package from kernel.org.

Intel CPUs (Sapphire Rapids, Granite Rapids) have microcode updates that fix errata related to PCIe bandwidth and memory latency. For example, a 2024 microcode update fixed a bug where PCIe Gen 5 links would train at Gen 4 speeds on some motherboards. Use iucode_tool to check your microcode version and update via the intel-microcode package.

PCIe switch firmware (e.g., on PLX switches used in multi-GPU motherboards) is often overlooked. Outdated switch firmware can cause packet loss under heavy P2P traffic, leading to NCCL timeouts during multi-GPU training. Check the switch vendor's support page for firmware updates.

# Check NVIDIA VBIOS version
nvidia-smi --query-gpu=vbios_version --format=csv
# Check AMD SMU firmware
sudo cat /sys/kernel/debug/dri/0/amdgpu_smu_info | grep 'firmware'
# Check Intel microcode version
iucode_tool -l | grep 'microcode'
# Update linux-firmware (Ubuntu/Debian)
sudo apt update && sudo apt install linux-firmware

Decision Matrix: Which Driver Stack for Your Build?

Here is a quick decision matrix based on your GPU and use case:

- NVIDIA H100/H200/B200: Use nvidia-open (R550+). Enable MIG only if using proprietary module. Required kernel 5.13+ with DRM helpers. Firmware: VBIOS 94.02.xx.xx for H100. - NVIDIA RTX 4090/5090: Use proprietary nvidia.ko (R545 or R550). Avoid open module until NVIDIA certifies it. Enable ReBAR in VBIOS. Use nvidia-smi to lock clocks: nvidia-smi -lgc 2500 for stable inference. - AMD MI300X/MI250X: Use amdgpu with ROCm 6.0+. Required kernel 6.5+ for MI300X. Firmware: linux-firmware 20240220+. Set amdgpu.dc=0 for compute-only. - AMD Strix Halo NPU: Use amdxdna.ko (kernel 6.10+). Firmware from AMD NPU package. Only ONNX Runtime supported. - Intel Arc A770: Use i915 with Level Zero. Kernel 6.2+. Set i915.enable_guc=0. Expect 30-40 tok/s for 8B models. - Intel NPU (Meteor Lake): Use intel_npu.ko (kernel 6.7+). OpenVINO runtime only. Not for LLMs. - Apple M3 Ultra: No kernel driver needed; Metal uses IOKit. MLX framework bypasses kernel for GPU compute. Performance is competitive for memory-bound inference (400 GB/s bandwidth).

For multi-GPU setups, prioritize IOMMU group isolation and NUMA binding over driver version. A perfectly tuned kernel with a slightly older driver will outperform a bleeding-edge driver with misconfigured memory management.

Testing and Validating Your Driver Stack

After configuring your kernel and drivers, validate with a suite of tests. First, check that all GPUs are visible and have correct memory bandwidth. Use bandwidthTest from CUDA samples: for an RTX 4090, expect 1008 GB/s (HBM2E). For H100, expect 3.35 TB/s (HBM3). If you see half that, check PCIe link speed (nvidia-smi -q -d PCI) and ensure Gen 4 or Gen 5 is negotiated.

Second, run a multi-GPU all-reduce benchmark using NCCL (nccl-tests). For 4x RTX 4090 on PCIe Gen 4 x16, expect ~50 GB/s per GPU for all-reduce. If you see < 30 GB/s, check IOMMU groups and NUMA binding. Use nsys profile to see if there are cross-socket transfers.

Third, run an inference benchmark with vLLM or llama.cpp. For Llama 3 70B Q4 on 4x RTX 4090, expect 30-40 tokens/s. If you get < 20 tokens/s, check hugepages (cat /proc/meminfo | grep HugePages) and NUMA balancing (sysctl kernel.numa_balancing). Also check GPU power limits: nvidia-smi -pl 350 for RTX 4090.

Finally, stress test with a continuous inference loop for 24 hours. Monitor dmesg for GPU recovery events or PCIe errors. If you see amdgpu: GPU reset or nvidia: NVLink error, reduce PCIe link speed or update firmware.

# Validate PCIe link speed
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv
# Run NCCL all-reduce benchmark
mpirun -np 4 ./build/all_reduce_perf -b 128M -e 8G -f 2 -g 1
# Check hugepages usage
cat /proc/meminfo | grep -E 'HugePages|Hugepagesize'
# Monitor GPU power and temp
watch -n 1 nvidia-smi --query-gpu=power.draw,temperature.gpu --format=csv

Further reading

• NVIDIA Open Kernel Module Documentation
• ROCm Installation Guide for MI300X
• Intel NPU Driver Kernel Documentation
• VFIO and IOMMU Groups Explained
• NCCL Tests Repository