llm workload · 44 runs on record

Qwen 2.5 14B (Q4)

Qwen 2.5 14B Instruct, Q4_K_M GGUF, batch 1, 8K context.

Primary metric: Tokens / sec (tok/s)

Reference prompts

Representative prompts for this workload. Exact prompts and harness settings still depend on the cited source for each record.

  • Prompt 1

    Explain the relationship between attention heads and KV cache memory usage for a 7B-parameter transformer at 4K context.

  • Prompt 2

    Write a 60-word product description for a mid-range AI workstation: 1× RTX 4090, 64GB DDR5, 2TB NVMe. Highlight one trade-off.

  • Prompt 3

    List five concrete differences between INT4 weight-only quantization (Q4_K_M) and INT8 quantization (Q8_0) for inference.

  • Prompt 4

    I have 12GB of VRAM and want to run a coding assistant locally. What model size and quantization fit, with what context length?

  • Prompt 5

    Summarize the difference between greedy decoding and nucleus sampling in 3 short bullet points.

Reference runtime command

A representative invocation for reproducing this workload class. Source-specific runs may use adjacent runtimes unless the record says otherwise.

shell
llama-server -m qwen2.5-14b-instruct.Q4_K_M.gguf -c 8192 -ngl 999 --seed 42

Single batch. Q4_K_M (~8GB). 8K context is the lowest sensible setting; bump on devices with VRAM headroom.

Full leaderboard

Every record for Qwen 2.5 14B.

Top 10 leaderboard

Qwen 2.5 14B (Q4) · sorted by tokens / sec

Bar chart: Top 10 devices ranked by Tokens / sec for Qwen 2.5 14B (Q4). 1. 8× NVIDIA H100 SXM5 80GB (DGX H100) at 1,080 tok/s. 2. Groq LPU (8-chip rack) at 480 tok/s. 3. NVIDIA B200 192GB at 285 tok/s. 4. NVIDIA B200 192GB at 268 tok/s. 5. Google TPU v5p (Trillium) at 195 tok/s.

Value frontier, MSRP vs tokens / sec

Each dot is a device. Top-left is best value (cheap + fast).

Scatter chart of MSRP versus Tokens / sec across 42 devices. Devices in the upper-left region offer the best price/performance ratio. Top 5 by primary metric: 8× NVIDIA H100 SXM5 80GB (DGX H100) at $200,000 delivering 1,080 tok/s; Groq LPU (8-chip rack) at $160,000 delivering 480 tok/s; B200 192GB at $39,999 delivering 285 tok/s; B200 192GB at $39,999 delivering 268 tok/s; H200 141GB at $30,000 delivering 178 tok/s.

Full leaderboard

Click any row for detailed breakdown. Click column headers to sort.

#DeviceVerifBuy
1
8× NVIDIA H100 SXM5 80GB (DGX H100)Datacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
1.1ktok/s
640 GB5600 W$200k0.19 tok/s/W$0.0020/kAmazon
2
Groq LPU (8-chip rack)ASIC
Groq·FP16·8K ctx
6
480tok/s
1.84 GB1720 W$160k0.28 tok/s/W$0.0035/kAmazon
3
NVIDIA B200 192GBDatacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
285tok/s
192 GB1000 W$40k0.28 tok/s/W$0.0015/kAmazon
#4
NVIDIA B200 192GBDatacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
268tok/s
192 GB1000 W$40k0.27 tok/s/W$0.0016/kAmazon
#5
Google TPU v5p (Trillium)ASIC
Google·INT8·8K ctx
6
195tok/s
95 GB300 W$00.65 tok/s/W$0.00/MAmazon
#6
NVIDIA H200 141GBDatacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
178tok/s
141 GB700 W$30k0.25 tok/s/W$0.0018/kAmazon
#7
NVIDIA H100 SXM5 80GBDatacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
168tok/s
80 GB700 W$25k0.24 tok/s/W$0.0016/kAmazon
#8
NVIDIA H100 SXM5 80GBDatacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
165tok/s
80 GB700 W$25k0.24 tok/s/W$0.0016/kAmazon
#9
AWS Trainium2 Trn2ASIC
AWS·INT8·8K ctx
6
158tok/s
96 GB350 W$00.45 tok/s/W$0.00/MAmazon
#10
NVIDIA H100 SXM5 80GBDatacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
145tok/s
80 GB700 W$25k0.21 tok/s/W$0.0018/kAmazon
#11
AMD Instinct MI300X 192GBDatacenter GPU
AMD·Q4_K_M·8K ctx
6
142tok/s
192 GB750 W$18k0.19 tok/s/W$0.0013/kAmazon
#12
AMD Instinct MI300X 192GBDatacenter GPU
AMD·Q4_K_M·8K ctx
6
142tok/s
192 GB750 W$15k0.19 tok/s/W$0.0011/kAmazon
#13
AMD Instinct MI300X 192GBDatacenter GPU
AMD·Q4_K_M·8K ctx
6
128tok/s
192 GB750 W$15k0.17 tok/s/W$0.0012/kAmazon
#14
NVIDIA GeForce RTX 5090 32GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
115tok/s
32 GB575 W$2.0k0.20 tok/s/W$0.18/MAmazon
#15
NVIDIA A100 SXM4 80GBDatacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
108tok/s
80 GB400 W$15k0.27 tok/s/W$0.0015/kAmazon
#16
NVIDIA GeForce RTX 5090 32GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
105tok/s
32 GB575 W$2.0k0.18 tok/s/W$0.20/MAmazon
#17
NVIDIA GeForce RTX 5090 32GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
92tok/s
32 GB575 W$2.0k0.16 tok/s/W$0.23/MAmazon
#18
NVIDIA L40S 48GBDatacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
92tok/s
48 GB350 W$7.8k0.26 tok/s/W$0.90/MAmazon
#19
2× NVIDIA RTX 4090 24GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
92tok/s
48 GB900 W$3.2k0.10 tok/s/W$0.37/MAmazon
#20
NVIDIA GeForce RTX 5080 16GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
86tok/s
16 GB360 W$9990.24 tok/s/W$0.12/MAmazon
#21
NVIDIA GeForce RTX 5080 16GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
82tok/s
16 GB360 W$9990.23 tok/s/W$0.13/MAmazon
#22
NVIDIA DGX Spark (Project DIGITS, 128GB)Datacenter GPU
NVIDIA·Q4_K_M·16K ctx
6
82tok/s
128 GB240 W$3.0k0.34 tok/s/W$0.39/MAmazon
#23
NVIDIA DGX Spark (Project DIGITS, 128GB)Datacenter GPU
NVIDIA·Q4_K_M·8K ctx
6
78tok/s
128 GB240 W$3.0k0.33 tok/s/W$0.41/MAmazon
#24
NVIDIA GeForce RTX 4090 24GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
78tok/s
24 GB450 W$1.6k0.17 tok/s/W$0.22/MAmazon
#25
4× NVIDIA RTX 3090 24GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
78tok/s
96 GB1400 W$4.5k0.06 tok/s/W$0.61/MAmazon
#26
NVIDIA RTX 6000 Ada 48GBPro GPU
NVIDIA·Q4_K_M·8K ctx
6
76tok/s
48 GB300 W$6.8k0.25 tok/s/W$0.95/MAmazon
#27
NVIDIA GeForce RTX 4090 24GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
68tok/s
24 GB450 W$1.6k0.15 tok/s/W$0.25/MAmazon
#28
NVIDIA GeForce RTX 4080 Super 16GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
68tok/s
16 GB320 W$9990.21 tok/s/W$0.15/MAmazon
#29
NVIDIA GeForce RTX 5070 Ti 16GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
65tok/s
16 GB300 W$7490.22 tok/s/W$0.12/MAmazon
#30
NVIDIA GeForce RTX 4080 Super 16GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
62tok/s
16 GB320 W$9990.19 tok/s/W$0.17/MAmazon
#31
NVIDIA GeForce RTX 3090 24GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
55tok/s
24 GB350 W$1.5k0.16 tok/s/W$0.29/MAmazon
#32
Apple M3 Ultra (80c GPU, 512GB)Apple Silicon
Apple·Q4_K_M·32K ctx
6
52tok/s
512 GB80 W$8.5k0.65 tok/s/W$0.0017/kAmazon
#33
AMD Radeon RX 7900 XTX 24GBConsumer GPU
AMD·Q4_K_M·8K ctx
6
52tok/s
24 GB355 W$9990.15 tok/s/W$0.20/MAmazon
#34
AMD Radeon RX 7900 XTX 24GBConsumer GPU
AMD·Q4_K_M·8K ctx
6
48tok/s
24 GB355 W$9990.14 tok/s/W$0.22/MAmazon
#35
NVIDIA GeForce RTX 3090 24GBConsumer GPU
NVIDIA·Q4_K_M·8K ctx
6
48tok/s
24 GB350 W$1.5k0.14 tok/s/W$0.33/MAmazon
#36
Apple M3 Ultra (80c GPU, 512GB)Apple Silicon
Apple·Q4_K_M·8K ctx
6
44tok/s
512 GB100 W$10.0k0.44 tok/s/W$0.0024/kAmazon
#37
Apple M3 Ultra (80c GPU, 512GB)Apple Silicon
Apple·Q4_K_M·8K ctx
6
38tok/s
512 GB270 W$9.5k0.14 tok/s/W$0.0026/kAmazon
#38
Apple M4 Max (40c GPU, 128GB)Apple Silicon
Apple·Q4_K_M·16K ctx
6
38tok/s
128 GB65 W$5.0k0.58 tok/s/W$0.0014/kAmazon
#39
Apple M4 Max (40c GPU, 128GB)Apple Silicon
Apple·Q4_K_M·8K ctx
6
34tok/s
128 GB65 W$4.7k0.52 tok/s/W$0.0015/kAmazon
#40
Apple M2 Ultra (76c GPU, 192GB)Apple Silicon
Apple·Q4_K_M·8K ctx
6
32tok/s
192 GB80 W$7.0k0.40 tok/s/W$0.0023/kAmazon
#41
Apple M4 Max (40c GPU, 128GB)Apple Silicon
Apple·Q4_K_M·8K ctx
6
28tok/s
128 GB70 W$4.7k0.40 tok/s/W$0.0018/kAmazon
#42
Apple Mac mini M4 Pro 48GBApple Silicon
Apple·Q4_K_M·8K ctx
6
18tok/s
48 GB35 W$2.0k0.51 tok/s/W$0.0012/kAmazon
#43
NVIDIA Jetson AGX Orin 64GBEdge
NVIDIA·Q4_K_M·4K ctx
6
16tok/s
64 GB60 W$2.0k0.27 tok/s/W$0.0013/kAmazon
#44
AMD Ryzen AI Max+ 395 (Strix Halo, 96GB)NPU
AMD·Q4_K_M·8K ctx
6
14tok/s
96 GB120 W$2.2k0.12 tok/s/W$0.0017/kAmazon
Sorted by Tokens / sec (high → low)

Cite this benchmark

Use this in your paper, blog post, or comparison table.

BibTeX
@misc{myaihardware_qwen2.5-14b-q4_2026,
  title  = {MyAI Bench: Qwen 2.5 14B (Q4)},
  author = {{MyAIHardware Contributors}},
  year   = {2026},
  url    = {https://www.myaihardware.com/benchmarks/workload/qwen2.5-14b-q4},
  note   = {Version 1.3, accessed 2026-08-27}
}
APA
MyAIHardware Contributors. (2026). MyAI Bench: Qwen 2.5 14B (Q4). MyAIHardware. Retrieved 2026-08-27, from https://www.myaihardware.com/benchmarks/workload/qwen2.5-14b-q4
MLA
MyAIHardware Contributors. "MyAI Bench: Qwen 2.5 14B (Q4)." MyAIHardware, 2026, https://www.myaihardware.com/benchmarks/workload/qwen2.5-14b-q4. Accessed 2026-08-27.
Plain text
MyAI Bench, Qwen 2.5 14B (Q4). MyAIHardware Contributors, 2026. Version 1.3. https://www.myaihardware.com/benchmarks/workload/qwen2.5-14b-q4 (accessed 2026-08-27).