Mistral 7B v0.3 remains the reference small open model — first to popularize sliding-window attention and grouped-query attention. It's been surpassed on raw benchmarks by Llama 3.1 8B and Qwen 2.5 7B, but its Apache 2.0 license and massive ecosystem of fine-tunes keep it relevant. Excellent base for custom fine-tunes.
Mistral 7B needs at minimum 4.2 GB of VRAM at Q4_K_M quantization (15.4 GB at FP16). The cheapest GPU that comfortably fits with KV-cache headroom is the Intel Arc A380 6GB (6 GB VRAM, $139 MSRP). Measured throughput hits 1850 tok/s on Cerebras WSE-3. This model fits a single consumer GPU under 48 GB, so a one-card build works.
Weights only. Add ~20-30% for KV cache at typical context lengths.
| Quant | VRAM | Quality | Notes |
|---|---|---|---|
| FP16 | 15.4 GB | Reference | Training-precision weights |
| Q8_0 | 7.7 GB | Near-lossless | 8-bit, ~0.1% perplexity hit |
| Q6_K | 6.2 GB | Very high | 6-bit, near Q8 quality |
| Q5_K_M | 5.3 GB | High | Strong middle ground |
| Q4_K_M | 4.2 GB | Balanced (recommended) | Default for local deployments |
| Q4_0 | 4.3 GB | Legacy 4-bit | Older GGUF, kept for compatibility |
| Q3_K_M | 3.3 GB | Lossy | When VRAM is very tight |
| Q2_K | 2.5 GB | Extreme | Rescue option, quality degrades visibly |
Filtered from MyAIHardware's GPU database. FP16 needs 15.4 GB; Q4_K_M needs 4.2 GB.
98 measurement(s) for this model from MyAIHardware's benchmark database.
| Device | Speed | Quant | Context | Power |
|---|---|---|---|---|
| Cerebras WSE-3 | 1850 tok/s | FP16 | 4096 | 23000W |
| Cerebras WSE-3 (CS-3) | 1620 tok/s | FP16 | 8192 | 23000W |
| Groq LPU (cloud) | 1280 tok/s | FP16 | 8192 | 350W |
| Groq LPU Inference Engine | 750 tok/s | FP8 | 4096 | 215W |
| Groq LPU (per chip) | 540 tok/s | FP16 | 8192 | 215W |
| NVIDIA B200 192GB | 415 tok/s | Q4_K_M | 4096 | 1000W |
| Google TPU v5p (Trillium) | 410 tok/s | INT8 | 4096 | 300W |
| NVIDIA H200 141GB | 305 tok/s | Q4_K_M | 8192 | 700W |
| NVIDIA GH200 480GB | 305 tok/s | Q4_K_M | 4096 | 1000W |
| NVIDIA H100 SXM5 80GB | 295 tok/s | Q4_K_M | 16384 | 700W |
| AWS Trainium2 Trn2 | 280 tok/s | INT8 | 4096 | 350W |
| AMD Instinct MI300X 192GB | 268 tok/s | Q4_K_M | 4096 | 750W |
| NVIDIA H200 141GB | 268 tok/s | Q4_K_M | 4096 | 700W |
| NVIDIA H100 SXM5 80GB | 215 tok/s | Q4_K_M | 4096 | 700W |
| NVIDIA GeForce RTX 5090 32GB | 198 tok/s | Q4_K_M | 4096 | 575W |
| AMD Instinct MI300X 192GB | 195 tok/s | Q4_K_M | 4096 | 750W |
| Google TPU v5e | 195 tok/s | INT8 | 4096 | 170W |
| NVIDIA GeForce RTX 5090 32GB | 182 tok/s | Q4_K_M | 8192 | 575W |
| NVIDIA GeForce RTX 5090 32GB | 178 tok/s | Q4_K_M | 4096 | 575W |
| NVIDIA A100 SXM4 80GB | 168 tok/s | Q4_K_M | 4096 | 400W |
Official weights + popular runtime tags.
ollama pull mistral:7blms get mistralai/Mistral-7B-Instruct-v0.3-GGUFStep-by-step guides that use this model.
Models in a similar size or capability class.
Use these in papers, blog posts, or internal docs.
@misc{mistral-7b-2023,
title={Mistral 7B},
author={Mistral AI},
year={2023},
url={https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3}
}Mistral AI (2023). Mistral 7B [Model card]. Hugging Face. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3
Mistral 7B (Mistral, Mistral AI, 2023). Available at https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3.