Phi-3.5 Mini is Microsoft's curated-synthetic-data 3.8 B model — rivals 7-13 B models on reasoning despite the small size. Particularly notable for the 128K context window in such a small footprint. Runs at 100+ tok/s on midrange consumer GPUs at Q4_K_M (only ~2.3 GB VRAM). Weakness: limited world knowledge versus naturally-trained peers.
Phi-3.5 Mini 3.8B needs at minimum 2.3 GB of VRAM at Q4_K_M quantization (8.4 GB at FP16). The cheapest GPU that comfortably fits with KV-cache headroom is the Intel Arc A380 6GB (6 GB VRAM, $139 MSRP). Measured throughput hits 612 tok/s on NVIDIA H100 SXM5 80GB. This model fits a single consumer GPU under 48 GB, so a one-card build works.
Weights only. Add ~20-30% for KV cache at typical context lengths.
| Quant | VRAM | Quality | Notes |
|---|---|---|---|
| FP16 | 8.4 GB | Reference | Training-precision weights |
| Q8_0 | 4.2 GB | Near-lossless | 8-bit, ~0.1% perplexity hit |
| Q6_K | 3.4 GB | Very high | 6-bit, near Q8 quality |
| Q5_K_M | 2.9 GB | High | Strong middle ground |
| Q4_K_M | 2.3 GB | Balanced (recommended) | Default for local deployments |
| Q4_0 | 2.4 GB | Legacy 4-bit | Older GGUF, kept for compatibility |
| Q3_K_M | 1.8 GB | Lossy | When VRAM is very tight |
| Q2_K | 1.3 GB | Extreme | Rescue option, quality degrades visibly |
Filtered from MyAIHardware's GPU database. FP16 needs 8.4 GB; Q4_K_M needs 2.3 GB.
68 measurement(s) for this model from MyAIHardware's benchmark database.
| Device | Speed | Quant | Context | Power |
|---|---|---|---|---|
| NVIDIA H100 SXM5 80GB | 612 tok/s | Q4_K_M | 4096 | 700W |
| NVIDIA GeForce RTX 5090 32GB | 348 tok/s | Q4_K_M | 4096 | 575W |
| NVIDIA GeForce RTX 5090 32GB | 320 tok/s | Q4_K_M | 4096 | 575W |
| Google TPU v5e | 320 tok/s | INT8 | 4096 | 170W |
| NVIDIA GeForce RTX 4090 24GB | 285 tok/s | Q4_K_M | 4096 | 450W |
| NVIDIA GeForce RTX 4090 24GB | 245 tok/s | Q4_K_M | 4096 | 450W |
| NVIDIA GeForce RTX 4080 Super 16GB | 215 tok/s | Q4_K_M | 4096 | 320W |
| AMD Radeon RX 7900 XTX 24GB | 195 tok/s | Q4_K_M | 4096 | 355W |
| AMD Instinct MI210 64GB | 175 tok/s | Q4_K_M | 4096 | 300W |
| NVIDIA GeForce RTX 4070 Ti 12GB | 165 tok/s | Q4_K_M | 4096 | 285W |
| NVIDIA GeForce RTX 5070 12GB | 165 tok/s | Q4_K_M | 4096 | 250W |
| NVIDIA A40 48GB | 162 tok/s | Q4_K_M | 4096 | 300W |
| NVIDIA GeForce RTX 5060 Ti 8GB | 158 tok/s | Q4_K_M | 4096 | 180W |
| NVIDIA GeForce RTX 5060 8GB | 145 tok/s | Q4_K_M | 4096 | 145W |
| NVIDIA GeForce RTX 4070 12GB | 138 tok/s | Q4_K_M | 4096 | 200W |
| NVIDIA GeForce RTX 5060 Ti 16GB | 132 tok/s | Q4_K_M | 4096 | 180W |
| AMD Radeon RX 7800 XT 16GB | 132 tok/s | Q4_K_M | 4096 | 263W |
| NVIDIA RTX A5000 24GB | 122 tok/s | Q4_K_M | 4096 | 230W |
| NVIDIA GeForce RTX 5060 Ti 8GB | 118 tok/s | Q4_K_M | 4096 | 165W |
| NVIDIA GeForce RTX 5060 8GB | 98 tok/s | Q4_K_M | 4096 | 145W |
Official weights + popular runtime tags.
ollama pull phi3.5lms get bartowski/Phi-3.5-mini-instruct-GGUFStep-by-step guides that use this model.
Models in a similar size or capability class.
Use these in papers, blog posts, or internal docs.
@misc{phi-3-5-mini-2024,
title={Phi-3.5 Mini 3.8B},
author={Microsoft Research},
year={2024},
url={https://huggingface.co/microsoft/Phi-3.5-mini-instruct}
}Microsoft Research (2024). Phi-3.5 Mini 3.8B [Model card]. Hugging Face. https://huggingface.co/microsoft/Phi-3.5-mini-instruct
Phi-3.5 Mini 3.8B (Phi, Microsoft Research, 2024). Available at https://huggingface.co/microsoft/Phi-3.5-mini-instruct.