Mixtral 8x7B was the first widely successful open MoE — 47 B total weights but only 12.9 B activated per token. It needs all weights in VRAM (~26 GB at Q4_K_M), so practically a single 32 GB GPU (RTX 5090, A6000) or 2x 16 GB cards. The Apache 2.0 license and competitive Llama 2 70B-class quality made it a homelab favorite, though Llama 3.1 70B has largely superseded it.
Mixtral 8x7B needs at minimum 28.3 GB of VRAM at Q4_K_M quantization (102.7 GB at FP16). The cheapest GPU that comfortably fits with KV-cache headroom is the Apple Mac mini (M4 Pro, 48 GB) (48 GB VRAM, $1,799 MSRP). Community benchmark submissions are open. This model fits a single consumer GPU under 48 GB, so a one-card build works.
Weights only. Add ~20-30% for KV cache at typical context lengths.
| Quant | VRAM | Quality | Notes |
|---|---|---|---|
| FP16 | 102.7 GB | Reference | Training-precision weights |
| Q8_0 | 51.4 GB | Near-lossless | 8-bit, ~0.1% perplexity hit |
| Q6_K | 41.6 GB | Very high | 6-bit, near Q8 quality |
| Q5_K_M | 35.4 GB | High | Strong middle ground |
| Q4_K_M | 28.3 GB | Balanced (recommended) | Default for local deployments |
| Q4_0 | 28.9 GB | Legacy 4-bit | Older GGUF, kept for compatibility |
| Q3_K_M | 22.1 GB | Lossy | When VRAM is very tight |
| Q2_K | 16.4 GB | Extreme | Rescue option, quality degrades visibly |
Filtered from MyAIHardware's GPU database. FP16 needs 102.7 GB; Q4_K_M needs 28.3 GB.
We don't have community benchmarks for this exact model yet.
Official weights + popular runtime tags.
ollama pull mixtral:8x7blms get TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUFStep-by-step guides that use this model.
Models in a similar size or capability class.
Use these in papers, blog posts, or internal docs.
@misc{mixtral-8x7b-2023,
title={Mixtral 8x7B},
author={Mistral AI},
year={2023},
url={https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1}
}Mistral AI (2023). Mixtral 8x7B [Model card]. Hugging Face. https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
Mixtral 8x7B (Mistral, Mistral AI, 2023). Available at https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1.