Llama 3.1 8B is Meta's best-in-class small dense model — competitive with much larger 13B-class models on reasoning and instruction following. It's fast enough for single-GPU local chat on 8 GB cards at Q4 and ships with full 128K context support. The weakness is that it still trails 70B-class models on multi-step reasoning and tool use.
Llama 3.1 8B needs at minimum 4.8 GB of VRAM at Q4_K_M quantization (17.6 GB at FP16). The cheapest GPU that comfortably fits with KV-cache headroom is the Intel Arc A580 8GB (8 GB VRAM, $179 MSRP). Measured throughput hits 1980 tok/s on 8× NVIDIA H100 SXM5 80GB. This model fits a single consumer GPU under 48 GB, so a one-card build works.
Weights only. Add ~20-30% for KV cache at typical context lengths.
| Quant | VRAM | Quality | Notes |
|---|---|---|---|
| FP16 | 17.6 GB | Reference | Training-precision weights |
| Q8_0 | 8.8 GB | Near-lossless | 8-bit, ~0.1% perplexity hit |
| Q6_K | 7.1 GB | Very high | 6-bit, near Q8 quality |
| Q5_K_M | 6.1 GB | High | Strong middle ground |
| Q4_K_M | 4.8 GB | Balanced (recommended) | Default for local deployments |
| Q4_0 | 5.0 GB | Legacy 4-bit | Older GGUF, kept for compatibility |
| Q3_K_M | 3.8 GB | Lossy | When VRAM is very tight |
| Q2_K | 2.8 GB | Extreme | Rescue option, quality degrades visibly |
Filtered from MyAIHardware's GPU database. FP16 needs 17.6 GB; Q4_K_M needs 4.8 GB.
85 measurement(s) for this model from MyAIHardware's benchmark database.
| Device | Speed | Quant | Context | Power |
|---|---|---|---|---|
| 8× NVIDIA H100 SXM5 80GB | 1980 tok/s | FP16 | 8192 | 5600W |
| Cerebras WSE-3 (CS-3) | 1850 tok/s | FP16 | 8192 | 23000W |
| Groq LPU (cloud) | 1850 tok/s | FP16 | 8192 | 350W |
| AMD Instinct MI300X 192GB | 920 tok/s | Q4_K_M | 4096 | 750W |
| Groq LPU (per chip) | 750 tok/s | FP16 | 8192 | 215W |
| NVIDIA GeForce RTX 5090 32GB | 720 tok/s | Q4_K_M | 4096 | 575W |
| NVIDIA GeForce RTX 4090 24GB | 540 tok/s | Q4_K_M | 4096 | 450W |
| NVIDIA B200 192GB | 520 tok/s | FP16 | 8192 | 1000W |
| Google TPU v5p | 410 tok/s | FP16 | 8192 | 700W |
| NVIDIA GH200 Grace Hopper 480GB | 380 tok/s | FP16 | 8192 | 1000W |
| NVIDIA B200 192GB | 360 tok/s | Q4_K_M | 131072 | 1000W |
| NVIDIA H200 141GB | 340 tok/s | FP16 | 4096 | 700W |
| AWS Trainium2 | 320 tok/s | FP16 | 8192 | 500W |
| NVIDIA H100 SXM5 80GB | 282 tok/s | FP16 | 4096 | 700W |
| AMD Instinct MI300X 192GB | 260 tok/s | FP16 | 8192 | 750W |
| Google TPU v5e | 240 tok/s | FP16 | 4096 | 170W |
| NVIDIA A100 80GB SXM | 215 tok/s | FP16 | 4096 | 400W |
| AWS Trainium2 | 215 tok/s | Q4_K_M | 8192 | 500W |
| NVIDIA H200 141GB | 215 tok/s | Q4_K_M | 131072 | 700W |
| NVIDIA A100 40GB | 195 tok/s | FP16 | 4096 | 400W |
Official weights + popular runtime tags.
ollama pull llama3.1:8blms get meta-llama/Meta-Llama-3.1-8B-Instruct-GGUFStep-by-step guides that use this model.
Models in a similar size or capability class.
Use these in papers, blog posts, or internal docs.
@misc{llama-3-1-8b-2024,
title={Llama 3.1 8B},
author={Meta AI},
year={2024},
url={https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct}
}Meta AI (2024). Llama 3.1 8B [Model card]. Hugging Face. https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct
Llama 3.1 8B (Llama 3.1, Meta AI, 2024). Available at https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct.