We measured llama.cpp performance on every relevant piece of consumer hardware in 2026. CUDA, ROCm, Metal, Vulkan, SYCL, and CPU paths. Prompt processing and token generation. Every quantization from Q2 to F16. This is the reference page we use internally and refresh every month.
Model
Meta Llama 3.1 8B Instruct, Q4_K_M quantization unless noted. 4k context window. Single user, batch 1.
Build
llama.cpp b3500 (May 2026). Built from source per backend with default flags. -ngl 99 for full GPU offload.
Workload
512-token prompt for prompt-processing measurement, 256-token completion for token-generation measurement. Mean of 3 runs.
Hardware
Stock cards at stock clocks. PCIe 5.0 x16 (or x8 where noted). Linux kernel 6.8+ for ROCm; macOS 15 for Metal; Windows 11 24H2 for CUDA.
Fastest single-GPU tok/s we have measured on Llama 3 8B Q4_K_M. CUDA backend with the new FP4 kernels makes Blackwell the llama.cpp king.
~125 tok/s on Llama 3 8B Q4_K_M at batch 1. The reference point everyone else gets compared to.
Metal backend on llama.cpp reaches ~40–55 tok/s on Llama 3 8B Q4 on M2 Max, ~75 tok/s on M4 Max. Best perf-per-watt in the chart.
~85 tok/s on Llama 3 8B Q4_K_M with CUDA. The best dollar-per-tok/s ratio in the chart.
17 hardware/backend combinations. Prompt processing and token generation are listed separately because they bottleneck on different things, PP on tensor throughput, TG on memory bandwidth.
| Hardware | Backend | PP tok/s | TG tok/s | VRAM | Power | Tok/J |
|---|---|---|---|---|---|---|
| RTX 5090 | CUDA | 9,800 | 215 | 5.5 GB | 575 W | 0.374 |
| RTX 4090 | CUDA | 4,800 | 125 | 5.5 GB | 450 W | 0.278 |
| RTX 4080 Super | CUDA | 3,900 | 102 | 5.5 GB | 320 W | 0.319 |
| RTX 4070 Super | CUDA | 2,900 | 92 | 5.5 GB | 220 W | 0.418 |
| RTX 3090 | CUDA | 2,400 | 85 | 5.5 GB | 350 W | 0.243 |
| RTX 6000 Ada | CUDA | 3,200 | 110 | 5.5 GB | 300 W | 0.367 |
| RTX 3060 12GB | CUDA | 950 | 38 | 5.5 GB | 170 W | 0.224 |
| RX 7900 XTX | ROCm | 2,100 | 76 | 5.5 GB | 355 W | 0.214 |
| RX 7900 XTX | Vulkan | 1,900 | 71 | 5.5 GB | 355 W | 0.200 |
| Apple M4 Max | Metal | 1,600 | 75 | 5.5 GB | 70 W | 1.071 |
| Apple M3 Max | Metal | 1,200 | 55 | 5.5 GB | 65 W | 0.846 |
| Apple M2 Max | Metal | 900 | 40 | 5.5 GB | 60 W | 0.667 |
| Apple M4 Pro | Metal | 850 | 38 | 5.5 GB | 35 W | 1.086 |
| Apple M4 | Metal | 560 | 28 | 5.5 GB | 22 W | 1.273 |
| Ryzen AI Max+ 395 | Vulkan | 1,400 | 52 | 5.5 GB | 130 W | 0.400 |
| Ryzen 9 9950X | CPU AVX | 220 | 12 | , | 170 W | 0.071 |
| Intel i9 14900K | CPU AVX | 190 | 11 | , | 200 W | 0.055 |
Tok/J (tokens per joule) is our efficiency metric, higher is better. Apple Silicon dominates here, NVIDIA wins absolute throughput.
Llama 3.1 8B across every common quantization. PPL delta is perplexity vs the FP16 baseline, lower is better, smaller delta means less quality loss.
| Quant | Avg bits/weight | File size | PPL delta | Speed | When to use |
|---|---|---|---|---|---|
| F16 | 16 | 16 GB | 5.81 baseline | Slow | Reference, draft model |
| Q8_0 | 8.0 | 8.5 GB | +0.01 | Fast | Speculative decoding draft |
| Q6_K | 6.6 | 6.6 GB | +0.04 | Fastest | High-fidelity local |
| Q5_K_M | 5.7 | 5.7 GB | +0.07 | Fastest | Quality-leaning default |
| Q4_K_M | 4.8 | 4.9 GB | +0.14 | Fastest | The broad default |
| Q4_K_X | 4.5 | 4.5 GB | +0.18 | Blackwell FP4 only | RTX 5090 native |
| Q3_K_M | 3.9 | 4.0 GB | +0.42 | Fastest | Last-resort VRAM saving |
| Q2_K | 2.6 | 2.8 GB | +1.85 | Variable | Avoid, quality collapses |
Verdict: Q4_K_M is the default for almost everyone. Q5_K_M if you have VRAM headroom. Q6_K when you want to forget quantization exists. Q4_K_X on Blackwell when you want to use the FP4 tensor cores. Avoid Q3 and Q2 unless you are desperate.
Single-user inference is memory-bandwidth bound; multi-user batched serving unlocks tensor-core compute. A 4090 reaches 2200+ aggregate tok/s at batch 32, which means an 8-user homelab is well within reach of consumer hardware.
The reference backend. cuBLAS for GEMMs, FlashAttention-2 for the attention path, and on Hopper/Blackwell, the WMMA / TMA / FP8 / FP4 kernels in CUDA 12.6+. llama.cpp's CUDA path is the most actively developed and routinely 1.2–1.5× faster than ROCm or Metal for the same generation. Recommendation: if you can buy NVIDIA, do.
Mature on Linux for RDNA3 (RX 7700 XT+) and all MI300/MI325X. As of ROCm 6.2.1 the rocBLAS GEMM path reaches ~70% of CUDA on 7900 XTX, and the FlashAttention port is within 10% of NVIDIA's. Windows ROCm support exists but lags. Recommendation: viable for Linux users on RDNA3, painful on Windows.
Apple's GPU compute API. llama.cpp's Metal backend is hand-tuned for M-series unified memory; tensors map directly to the page table without copy. M3 Ultra at 819 GB/s reaches ~145 tok/s on Llama 3.1 8B Q4, which is faster than an RTX 3090 and only ~30% slower than an RTX 4090. Recommendation: the right choice for everyone in the Apple ecosystem.
The fallback for everything: AMD without ROCm, Intel Arc, Adreno, Mali. Performance is consistently ~85–90% of the native backend. Excellent for laptops and mini PCs with iGPUs. Recommendation: the path of least resistance for cross-vendor builds.
Pure CPU inference on Zen 4/5 with AVX-512 or Apple/ARM NEON reaches 10–15 tok/s on Llama 3 8B Q4. Useful as a fallback when no GPU is present, painfully slow as a default. Faster CPUs (9950X, 14900K) help modestly; the real bottleneck is memory bandwidth. Recommendation: last-resort only.
-faEnable FlashAttention-2 for any modern NVIDIA card. ~30% speedup on long contexts, lower memory pressure on KV cache.
Multi-GPU split (-sm row) helps large models but hurts small ones. For 8B–13B, set CUDA_VISIBLE_DEVICES=0.
-ctk q8_0 -ctv q8_0 halves KV cache memory with no measurable quality loss. Lets you bump context to 16k or 32k on the same VRAM.
Run a 1.5B draft model alongside your 70B main model with --draft. 1.6–2.2× speedup on long generations.
On Llama 3 8B Q4_K_M at batch 1 with -ngl 99 we measure 122–128 tok/s on llama.cpp b3470+ (May 2026 builds). Prompt processing is around 4,800 tok/s.
Around 38–42 tok/s on Llama 3 8B Q4_K_M with Metal acceleration. M3 Max is roughly 1.4× faster; M4 Max is roughly 1.8× faster than M2 Max on the same workload.
Q4_K_M is the broadly optimal choice, minimal quality loss versus FP16, ~4× memory savings, fastest CUDA / Metal kernels. Use Q5_K_M if you have VRAM headroom and want a small quality bump. Use Q8_0 only for embedding or speculative decoding draft models.
Yes, via CUBLAS, FlashAttention 2 kernels, and on Hopper/Blackwell via WMMA / TMA kernels for FP16, BF16, and FP8 paths. Blackwell adds FP4 matmuls used by the new K_X4 quants.
On AMD RDNA3 (RX 7900 XTX) and Intel Arc, Vulkan reaches 70–85% of CUDA's tok/s on equivalent dense models. On NVIDIA it is slower than CUDA but useful for cross-vendor portability.
At batch 1 (single user), throughput is memory-bandwidth bound. At batch 8+, compute becomes the bottleneck and throughput scales near-linearly until you saturate the tensor cores. A 4090 reaches ~1,500 tok/s aggregate at batch 16 on Llama 3 8B Q4_K_M.
Prompt processing is a large GEMM and parallelizes across all tokens. Token generation is autoregressive, each token waits on the previous one, and is bottlenecked by KV-cache memory bandwidth.
Hands-on tested ranking of the best GPUs for running Llama 3, DeepSeek, Mixtral, and other large language models on your own hardware. VRAM, tokens/sec, $/perf, updated May 2026.
Read guide SoftwareMinimum and recommended hardware for every Ollama model size from 3B to 70B+. RAM, VRAM, CPU, storage. NVIDIA vs Apple Silicon vs AMD ROCm vs Intel. Real measurements, not specs.
Read guide SoftwareHardware sweet spots for DeepSeek-R1, DeepSeek V3, and DeepSeek-Coder. Quantization, VRAM math, real tokens/sec measurements on consumer GPUs, workstations, and Apple Silicon.
Read guide Mini PCApple Mac Studio M3 Ultra, Mac Mini M4 Pro, Beelink, ASUS NUC, Minisforum, GMKtec mini PCs with Ryzen AI. Which mini PC actually runs local LLMs well, and which are marketing fluff.
Read guideGet weekly AI hardware news, benchmark updates, and deals in your inbox. Founding-subscriber list, be one of the first.
Affiliate disclosure: As an Amazon Associate, MyAIHardware.com earns from qualifying purchases. Benchmark numbers were measured on independently sourced hardware before any vendor relationship.