llm workload · 36 runs on record
DeepSeek-R1 Distill 7B (Q4)
DeepSeek-R1-Distill-Qwen-7B, Q4_K_M, batch 1, 8K context.
Primary metric: Tokens / sec (tok/s)
Reference prompts
Representative prompts for this workload. Exact prompts and harness settings still depend on the cited source for each record.
- Prompt 1
A train leaves station A at 9:00 going 60 mph. Another leaves station B (180 miles away) at 9:30 going 80 mph toward A. When do they meet? Show your work.
- Prompt 2
If a 70B-parameter model uses ~140GB at FP16, what is its likely memory footprint at Q4_K_M? Show the arithmetic.
- Prompt 3
Walk through which of these is cheaper for 10M tokens/day at 30 days: H100 cloud at $2/hr vs RTX 4090 owned at $1600 + $0.10/kWh.
- Prompt 4
Explain step-by-step why FP8 inference can be 2x faster than FP16 on H100 but not on A100.
- Prompt 5
Given a 4-bit quantized Llama 3 70B model at ~40GB, how much VRAM headroom is needed for KV cache at 8K context, batch 1?
Reference runtime command
A representative invocation for reproducing this workload class. Source-specific runs may use adjacent runtimes unless the record says otherwise.
shell
llama-server -m DeepSeek-R1-Distill-Qwen-7B.Q4_K_M.gguf -c 8192 -ngl 999 --seed 42Reasoning workload. We use 8K context to give the chain-of-thought tokens room. Single batch.
Full leaderboard
Every record for DeepSeek-R1 7B.
Top 10 leaderboard
DeepSeek-R1 Distill 7B (Q4) · sorted by tokens / sec
36 entries
Value frontier, MSRP vs tokens / sec
Each dot is a device. Top-left is best value (cheap + fast).
Full leaderboard
Click any row for detailed breakdown. Click column headers to sort.
| # | Device | Verif | Buy | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Groq LPU (per chip)ASIC Groq·FP16·8K ctx | 6 | 620tok/s | 0.23 GB | 215 W | $20k | 2.88 tok/s/W | $0.34/M | Amazon | |
| 2 | NVIDIA B200 192GBDatacenter GPU NVIDIA·Q4_K_M·8K ctx | 6 | 480tok/s | 192 GB | 1000 W | $40k | 0.48 tok/s/W | $0.88/M | Amazon | |
| 3 | NVIDIA B200 192GBDatacenter GPU NVIDIA·Q4_K_M·8K ctx | 6 | 385tok/s | 192 GB | 1000 W | $40k | 0.39 tok/s/W | $0.0011/k | Amazon | |
| #4 | NVIDIA H200 141GBDatacenter GPU NVIDIA·Q4_K_M·8K ctx | 6 | 305tok/s | 141 GB | 700 W | $30k | 0.44 tok/s/W | $0.0010/k | Amazon | |
| #5 | NVIDIA H100 SXM5 80GBDatacenter GPU NVIDIA·Q4_K_M·16K ctx | 6 | 285tok/s | 80 GB | 700 W | $25k | 0.41 tok/s/W | $0.93/M | Amazon | |
| #6 | AMD Instinct MI300X 192GBDatacenter GPU AMD·Q4_K_M·16K ctx | 6 | 248tok/s | 192 GB | 750 W | $15k | 0.33 tok/s/W | $0.64/M | Amazon | |
| #7 | NVIDIA H100 SXM5 80GBDatacenter GPU NVIDIA·Q4_K_M·8K ctx | 6 | 245tok/s | 80 GB | 700 W | $25k | 0.35 tok/s/W | $0.0011/k | Amazon | |
| #8 | NVIDIA H200 141GBDatacenter GPU NVIDIA·Q4_K_M·8K ctx | 6 | 232tok/s | 141 GB | 700 W | $30k | 0.33 tok/s/W | $0.0014/k | Amazon | |
| #9 | AMD Instinct MI300X 192GBDatacenter GPU AMD·Q4_K_M·8K ctx | 6 | 215tok/s | 192 GB | 750 W | $18k | 0.29 tok/s/W | $0.89/M | Amazon | |
| #10 | NVIDIA H100 SXM5 80GBDatacenter GPU NVIDIA·Q4_K_M·8K ctx | 6 | 198tok/s | 80 GB | 700 W | $25k | 0.28 tok/s/W | $0.0013/k | Amazon | |
| #11 | NVIDIA GeForce RTX 5090 32GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 185tok/s | 32 GB | 575 W | $2.0k | 0.32 tok/s/W | $0.11/M | Amazon | |
| #12 | AMD Instinct MI300X 192GBDatacenter GPU AMD·Q4_K_M·8K ctx | 6 | 168tok/s | 192 GB | 750 W | $15k | 0.22 tok/s/W | $0.94/M | Amazon | |
| #13 | NVIDIA GeForce RTX 5090 32GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 165tok/s | 32 GB | 575 W | $2.0k | 0.29 tok/s/W | $0.13/M | Amazon | |
| #14 | NVIDIA DGX Spark (Project DIGITS, 128GB)Datacenter GPU NVIDIA·Q4_K_M·8K ctx | 6 | 145tok/s | 128 GB | 240 W | $3.0k | 0.60 tok/s/W | $0.22/M | Amazon | |
| #15 | NVIDIA GeForce RTX 5080 16GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 138tok/s | 16 GB | 360 W | $999 | 0.38 tok/s/W | $0.08/M | Amazon | |
| #16 | NVIDIA L40S 48GBDatacenter GPU NVIDIA·Q4_K_M·8K ctx | 6 | 138tok/s | 48 GB | 350 W | $7.8k | 0.39 tok/s/W | $0.60/M | Amazon | |
| #17 | NVIDIA GeForce RTX 4090 24GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 125tok/s | 24 GB | 450 W | $1.6k | 0.28 tok/s/W | $0.14/M | Amazon | |
| #18 | NVIDIA GeForce RTX 5080 16GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 122tok/s | 16 GB | 360 W | $999 | 0.34 tok/s/W | $0.09/M | Amazon | |
| #19 | NVIDIA GeForce RTX 4090 24GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 118tok/s | 24 GB | 450 W | $1.6k | 0.26 tok/s/W | $0.14/M | Amazon | |
| #20 | NVIDIA GeForce RTX 5070 Ti 16GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 108tok/s | 16 GB | 300 W | $749 | 0.36 tok/s/W | $0.07/M | Amazon | |
| #21 | NVIDIA GeForce RTX 5070 Ti 16GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 105tok/s | 16 GB | 300 W | $749 | 0.35 tok/s/W | $0.07/M | Amazon | |
| #22 | NVIDIA GeForce RTX 4080 Super 16GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 98tok/s | 16 GB | 320 W | $999 | 0.31 tok/s/W | $0.11/M | Amazon | |
| #23 | NVIDIA GeForce RTX 4080 Super 16GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 95tok/s | 16 GB | 320 W | $999 | 0.30 tok/s/W | $0.11/M | Amazon | |
| #24 | NVIDIA GeForce RTX 3090 24GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 82tok/s | 24 GB | 350 W | $1.5k | 0.23 tok/s/W | $0.19/M | Amazon | |
| #25 | AMD Radeon RX 7900 XTX 24GBConsumer GPU AMD·Q4_K_M·8K ctx | 6 | 78tok/s | 24 GB | 355 W | $999 | 0.22 tok/s/W | $0.14/M | Amazon | |
| #26 | NVIDIA GeForce RTX 5070 12GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 75tok/s | 12 GB | 250 W | $549 | 0.30 tok/s/W | $0.08/M | Amazon | |
| #27 | Apple M3 Ultra (80c GPU, 512GB)Apple Silicon Apple·Q4_K_M·32K ctx | 6 | 75tok/s | 512 GB | 80 W | $8.5k | 0.94 tok/s/W | $0.0012/k | Amazon | |
| #28 | AMD Radeon RX 7900 XTX 24GBConsumer GPU AMD·Q4_K_M·8K ctx | 6 | 70tok/s | 24 GB | 355 W | $999 | 0.20 tok/s/W | $0.15/M | Amazon | |
| #29 | NVIDIA GeForce RTX 5060 Ti 16GBConsumer GPU NVIDIA·Q4_K_M·8K ctx | 6 | 62tok/s | 16 GB | 180 W | $449 | 0.34 tok/s/W | $0.08/M | Amazon | |
| #30 | Apple M4 Max (40c GPU, 128GB)Apple Silicon Apple·Q4_K_M·16K ctx | 6 | 58tok/s | 128 GB | 65 W | $5.0k | 0.89 tok/s/W | $0.91/M | Amazon | |
| #31 | Intel Arc B580 12GBConsumer GPU Intel·Q4_K_M·8K ctx | 6 | 52tok/s | 12 GB | 190 W | $249 | 0.27 tok/s/W | $0.05/M | Amazon | |
| #32 | Apple M4 Max (40c GPU, 128GB)Apple Silicon Apple·Q4_K_M·8K ctx | 6 | 50tok/s | 128 GB | 65 W | $4.7k | 0.77 tok/s/W | $0.99/M | Amazon | |
| #33 | Apple M4 Max (40c GPU, 128GB)Apple Silicon Apple·Q4_K_M·8K ctx | 6 | 48tok/s | 128 GB | 70 W | $4.7k | 0.69 tok/s/W | $0.0010/k | Amazon | |
| #34 | Apple Mac mini M4 Pro 48GBApple Silicon Apple·Q4_K_M·16K ctx | 6 | 32tok/s | 48 GB | 35 W | $2.0k | 0.91 tok/s/W | $0.66/M | Amazon | |
| #35 | AMD Ryzen AI Max+ 395 (Strix Halo, 96GB)NPU AMD·Q4_K_M·8K ctx | 6 | 24tok/s | 96 GB | 120 W | $2.2k | 0.20 tok/s/W | $0.97/M | Amazon | |
| #36 | NVIDIA Jetson AGX Orin 64GBEdge NVIDIA·Q4_K_M·8K ctx | 6 | 22tok/s | 64 GB | 60 W | $2.0k | 0.37 tok/s/W | $0.96/M | Amazon |
Sorted by Tokens / sec (high → low)Tip: $/1k tok is a 3-year amortization of MSRP at sustained throughput.
Cite this benchmark
Use this in your paper, blog post, or comparison table.
BibTeX
@misc{myaihardware_deepseek-r1-7b-q4_2026,
title = {MyAI Bench: DeepSeek-R1 Distill 7B (Q4)},
author = {{MyAIHardware Contributors}},
year = {2026},
url = {https://www.myaihardware.com/benchmarks/workload/deepseek-r1-7b-q4},
note = {Version 1.3, accessed 2026-08-27}
}APA
MyAIHardware Contributors. (2026). MyAI Bench: DeepSeek-R1 Distill 7B (Q4). MyAIHardware. Retrieved 2026-08-27, from https://www.myaihardware.com/benchmarks/workload/deepseek-r1-7b-q4
MLA
MyAIHardware Contributors. "MyAI Bench: DeepSeek-R1 Distill 7B (Q4)." MyAIHardware, 2026, https://www.myaihardware.com/benchmarks/workload/deepseek-r1-7b-q4. Accessed 2026-08-27.
Plain text
MyAI Bench, DeepSeek-R1 Distill 7B (Q4). MyAIHardware Contributors, 2026. Version 1.3. https://www.myaihardware.com/benchmarks/workload/deepseek-r1-7b-q4 (accessed 2026-08-27).