llm workload · 63 runs on record
Llama 3 8B (Q4_K_M)
Meta Llama 3 8B Instruct, Q4_K_M GGUF, batch 1, 4K context.
Primary metric: Tokens / sec (tok/s)
Reference prompts
Representative prompts for this workload. Exact prompts and harness settings still depend on the cited source for each record.
- Prompt 1
Explain the relationship between attention heads and KV cache memory usage for a 7B-parameter transformer at 4K context.
- Prompt 2
Write a 60-word product description for a mid-range AI workstation: 1× RTX 4090, 64GB DDR5, 2TB NVMe. Highlight one trade-off.
- Prompt 3
List five concrete differences between INT4 weight-only quantization (Q4_K_M) and INT8 quantization (Q8_0) for inference.
- Prompt 4
I have 12GB of VRAM and want to run a coding assistant locally. What model size and quantization fit, with what context length?
- Prompt 5
Summarize the difference between greedy decoding and nucleus sampling in 3 short bullet points.
Reference runtime command
A representative invocation for reproducing this workload class. Source-specific runs may use adjacent runtimes unless the record says otherwise.
shell
llama-server -m llama-3-8b-instruct.Q4_K_M.gguf -c 4096 -ngl 999 --seed 42Single batch. Q4_K_M GGUF (~5GB). Median of 5 runs, prompt eval excluded.
Full leaderboard
Every record for Llama 3 8B Q4.
Top 10 leaderboard
Llama 3 8B (Q4_K_M) · sorted by tokens / sec
63 entries
Value frontier, MSRP vs tokens / sec
Each dot is a device. Top-left is best value (cheap + fast).
Full leaderboard
Click any row for detailed breakdown. Click column headers to sort.
| # | Device | Verif | Buy | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | AMD Instinct MI300X 192GBDatacenter GPU AMD·Q4_K_M·4K ctx | 6 | 920tok/s | 192 GB | 750 W | $15k | 1.23 tok/s/W | $0.17/M | Amazon | |
| 2 | NVIDIA GeForce RTX 5090 32GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 720tok/s | 32 GB | 575 W | $2.0k | 1.25 tok/s/W | $0.03/M | Amazon | |
| 3 | NVIDIA GeForce RTX 4090 24GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 540tok/s | 24 GB | 450 W | $1.6k | 1.20 tok/s/W | $0.03/M | Amazon | |
| #4 | NVIDIA B200 192GBDatacenter GPU NVIDIA·Q4_K_M·128K ctx | 6 | 360tok/s | 192 GB | 1000 W | $40k | 0.36 tok/s/W | $0.0012/k | Amazon | |
| #5 | AWS Trainium2ASIC AWS·Q4_K_M·8K ctx | 6 | 215tok/s | 96 GB | 500 W | $28k | 0.43 tok/s/W | $0.0014/k | Amazon | |
| #6 | NVIDIA H200 141GBDatacenter GPU NVIDIA·Q4_K_M·128K ctx | 6 | 215tok/s | 141 GB | 700 W | $30k | 0.31 tok/s/W | $0.0015/k | Amazon | |
| #7 | NVIDIA GeForce RTX 5090 32GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 182tok/s σ 5.6 | 32 GB | 575 W | $2.0k | 0.32 tok/s/W | $0.12/M | Amazon | |
| #8 | NVIDIA H100 SXM5 80GBDatacenter GPU NVIDIA·Q4_K_M·128K ctx | 6 | 178tok/s | 80 GB | 700 W | $25k | 0.25 tok/s/W | $0.0015/k | Amazon | |
| #9 | NVIDIA GeForce RTX 5080 16GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 165tok/s σ 4.8 | 16 GB | 360 W | $999 | 0.46 tok/s/W | $0.06/M | Amazon | |
| #10 | NVIDIA GeForce RTX 5090 32GBConsumer GPU NVIDIA·Q4_K_M·32K ctx | 6 | 162tok/s | 32 GB | 575 W | $2.0k | 0.28 tok/s/W | $0.13/M | Amazon | |
| #11 | NVIDIA GeForce RTX 4090 24GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 132tok/s σ 3.8 | 24 GB | 450 W | $1.6k | 0.29 tok/s/W | $0.13/M | Amazon | |
| #12 | NVIDIA GeForce RTX 5070 Ti 16GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 128tok/s | 16 GB | 300 W | $749 | 0.43 tok/s/W | $0.06/M | Amazon | |
| #13 | AMD Instinct MI300X 192GBDatacenter GPU AMD·Q4_K_M·128K ctx | 6 | 122tok/s | 192 GB | 750 W | $15k | 0.16 tok/s/W | $0.0013/k | Amazon | |
| #14 | NVIDIA GeForce RTX 4090 24GBConsumer GPU NVIDIA·Q4_K_M·32K ctx | 6 | 112tok/s | 24 GB | 450 W | $1.6k | 0.25 tok/s/W | $0.15/M | Amazon | |
| #15 | NVIDIA GeForce RTX 4080 Super 16GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 102tok/s σ 3.4 | 16 GB | 320 W | $999 | 0.32 tok/s/W | $0.10/M | Amazon | |
| #16 | Apple M3 Ultra (80c GPU, 512GB)Apple Silicon Apple·Q4_K_M·8K ctx | 6 | 102tok/s | 512 GB | 270 W | $9.5k | 0.38 tok/s/W | $0.98/M | Amazon | |
| #17 | AMD Radeon RX 7900 XTX 24GBConsumer GPU AMD·Q4_K_M·4K ctx | 6 | 95tok/s | 24 GB | 355 W | $999 | 0.27 tok/s/W | $0.11/M | Amazon | |
| #18 | NVIDIA GeForce RTX 5070 12GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 92tok/s | 12 GB | 250 W | $549 | 0.37 tok/s/W | $0.06/M | Amazon | |
| #19 | NVIDIA GeForce RTX 3090 24GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 88tok/s | 24 GB | 350 W | $1.5k | 0.25 tok/s/W | $0.18/M | Amazon | |
| #20 | NVIDIA GeForce RTX 4070 Ti 12GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 78tok/s | 12 GB | 285 W | $799 | 0.27 tok/s/W | $0.11/M | Amazon | |
| #21 | NVIDIA GeForce RTX 5060 Ti 16GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 72tok/s | 16 GB | 180 W | $449 | 0.40 tok/s/W | $0.07/M | Amazon | |
| #22 | AMD Radeon RX 7800 XT 16GBConsumer GPU AMD·Q4_K_M·4K ctx | 6 | 64tok/s | 16 GB | 263 W | $499 | 0.24 tok/s/W | $0.08/M | Amazon | |
| #23 | NVIDIA A40 48GBPro GPU NVIDIA·Q4_K_M·4K ctx | 6 | 64tok/s | 48 GB | 300 W | $5.5k | 0.21 tok/s/W | $0.91/M | Amazon | |
| #24 | NVIDIA GeForce RTX 4070 12GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 62tok/s | 12 GB | 200 W | $599 | 0.31 tok/s/W | $0.10/M | Amazon | |
| #25 | NVIDIA RTX A5000 24GBPro GPU NVIDIA·Q4_K_M·4K ctx | 6 | 58tok/s | 24 GB | 230 W | $2.2k | 0.25 tok/s/W | $0.40/M | Amazon | |
| #26 | NVIDIA GeForce RTX 5060 Ti 8GBConsumer GPU NVIDIA·Q4_K_M·2K ctx | 6 | 58tok/s | 8 GB | 165 W | $379 | 0.35 tok/s/W | $0.07/M | Amazon | |
| #27 | Apple M4 Max (40c GPU, 128GB)Apple Silicon Apple·Q4_K_M·8K ctx | 6 | 56tok/s σ 1.8 | 128 GB | 70 W | $4.7k | 0.80 tok/s/W | $0.89/M | Amazon | |
| #28 | AMD Radeon RX 7700 XT 12GBConsumer GPU AMD·Q4_K_M·4K ctx | 6 | 52tok/s | 12 GB | 245 W | $449 | 0.21 tok/s/W | $0.09/M | Amazon | |
| #29 | Apple M3 Max (40c GPU, 64GB)Apple Silicon Apple·Q4_K_M·8K ctx | 6 | 50tok/s | 64 GB | 65 W | $3.5k | 0.77 tok/s/W | $0.74/M | Amazon | |
| #30 | NVIDIA GeForce RTX 4060 Ti 16GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 48tok/s | 16 GB | 165 W | $499 | 0.29 tok/s/W | $0.11/M | Amazon | |
| #31 | NVIDIA GeForce RTX 5060 8GBConsumer GPU NVIDIA·Q4_K_M·2K ctx | 6 | 48tok/s | 8 GB | 145 W | $299 | 0.33 tok/s/W | $0.07/M | Amazon | |
| #32 | Apple M4 Max (40c GPU, 128GB)Apple Silicon Apple·Q4_K_M·32K ctx | 6 | 48tok/s | 128 GB | 65 W | $4.7k | 0.74 tok/s/W | $0.0010/k | Amazon | |
| #33 | NVIDIA GeForce RTX 3060 12GBConsumer GPU NVIDIA·Q4_K_M·4K ctx | 6 | 42tok/s | 12 GB | 170 W | $329 | 0.25 tok/s/W | $0.08/M | Amazon | |
| #34 | NVIDIA RTX A4000 16GBPro GPU NVIDIA·Q4_K_M·4K ctx | 6 | 42tok/s | 16 GB | 140 W | $1.0k | 0.30 tok/s/W | $0.25/M | Amazon | |
| #35 | NVIDIA GeForce RTX 4060 8GBConsumer GPU NVIDIA·Q4_K_M·2K ctx | 6 | 42tok/s | 8 GB | 115 W | $299 | 0.36 tok/s/W | $0.07/M | Amazon | |
| #36 | Intel Arc B580 12GBConsumer GPU Intel·Q4_K_M·4K ctx | 6 | 38tok/s | 12 GB | 190 W | $249 | 0.20 tok/s/W | $0.07/M | Amazon | |
| #37 | Apple M2 Max (38c GPU, 96GB)Apple Silicon Apple·Q4_K_M·4K ctx | 6 | 38tok/s | 96 GB | 60 W | $3.3k | 0.63 tok/s/W | $0.92/M | Amazon | |
| #38 | Apple M4 Pro (20c GPU, 48GB)Apple Silicon Apple·Q4_K_M·8K ctx | 6 | 38tok/s | 48 GB | 50 W | $2.5k | 0.76 tok/s/W | $0.69/M | Amazon | |
| #39 | Apple M3 Ultra (80c GPU, 512GB)Apple Silicon Apple·Q4_K_M·128K ctx | 6 | 38tok/s | 512 GB | 100 W | $10.0k | 0.38 tok/s/W | $0.0028/k | Amazon | |
| #40 | Intel Arc A770 16GBConsumer GPU Intel·Q4_K_M·4K ctx | 6 | 36tok/s | 16 GB | 225 W | $329 | 0.16 tok/s/W | $0.10/M | Amazon | |
| #41 | NVIDIA GeForce RTX 4060 8GBConsumer GPU NVIDIA·Q4_K_M·2K ctx | 6 | 32tok/s | 8 GB | 115 W | $299 | 0.28 tok/s/W | $0.10/M | Amazon | |
| #42 | Apple Mac mini M4 Pro 24GBApple Silicon Apple·Q4_K_M·8K ctx | 6 | 32tok/s | 24 GB | 35 W | $1.4k | 0.91 tok/s/W | $0.46/M | Amazon | |
| #43 | AMD Radeon RX 7600 8GBConsumer GPU AMD·Q4_K_M·2K ctx | 6 | 28tok/s | 8 GB | 165 W | $269 | 0.17 tok/s/W | $0.10/M | Amazon | |
| #44 | AMD Ryzen AI Max+ 395 (Strix Halo, 96GB)NPU AMD·Q4_K_M·8K ctx | 6 | 28tok/s | 96 GB | 120 W | $2.2k | 0.23 tok/s/W | $0.83/M | Amazon | |
| #45 | Intel Arc A750 8GBConsumer GPU Intel·Q4_K_M·2K ctx | 6 | 24tok/s | 8 GB | 225 W | $249 | 0.11 tok/s/W | $0.11/M | Amazon | |
| #46 | NVIDIA Jetson AGX Orin 64GBEdge NVIDIA·Q4_K_M·4K ctx | 6 | 22tok/s | 64 GB | 60 W | $2.0k | 0.37 tok/s/W | $0.96/M | Amazon | |
| #47 | Apple M3 Pro (18c GPU, 36GB)Apple Silicon Apple·Q4_K_M·4K ctx | 6 | 22tok/s | 36 GB | 50 W | $2.5k | 0.44 tok/s/W | $0.0012/k | Amazon | |
| #48 | Apple M2 Pro (19c GPU, 32GB)Apple Silicon Apple·Q4_K_M·4K ctx | 6 | 20tok/s | 32 GB | 45 W | $2.0k | 0.44 tok/s/W | $0.0011/k | Amazon | |
| #49 | Apple M4 (10c GPU, 24GB)Apple Silicon Apple·Q4_K_M·4K ctx | 6 | 18tok/s | 24 GB | 22 W | $1.6k | 0.82 tok/s/W | $0.94/M | Amazon | |
| #50 | NVIDIA Jetson Orin NX 16GBEdge NVIDIA·Q4_K_M·4K ctx | 6 | 18tok/s | 16 GB | 25 W | $599 | 0.72 tok/s/W | $0.35/M | Amazon | |
| #51 | Apple Mac mini M4 24GBApple Silicon Apple·Q4_K_M·8K ctx | 6 | 17tok/s | 24 GB | 22 W | $999 | 0.77 tok/s/W | $0.62/M | Amazon | |
| #52 | Apple Mac mini M4 16GBApple Silicon Apple·Q4_K_M·4K ctx | 6 | 16tok/s | 16 GB | 18 W | $599 | 0.89 tok/s/W | $0.40/M | Amazon | |
| #53 | NVIDIA Jetson Orin Nano Super 8GBEdge NVIDIA·Q4_K_M·2K ctx | 6 | 14tok/s | 8 GB | 25 W | $249 | 0.56 tok/s/W | $0.19/M | Amazon | |
| #54 | Intel Xeon 6980P (128c Granite Rapids)CPU-only Intel·Q4_K_M·4K ctx | 6 | 14tok/s | , | 500 W | $18k | 0.03 tok/s/W | $0.0134/k | Amazon | |
| #55 | AMD Threadripper PRO 7995WX (96-core)CPU-only AMD·Q4_K_M·4K ctx | 6 | 12tok/s | , | 350 W | $10.0k | 0.03 tok/s/W | $0.0088/k | Amazon | |
| #56 | Qualcomm Snapdragon X Elite (X1E-84-100)NPU Qualcomm·INT4·4K ctx | 6 | 12tok/s | , | 23 W | $1.2k | 0.52 tok/s/W | $0.0011/k | Amazon | |
| #57 | Qualcomm Snapdragon X Elite (X1E-84-100)NPU Qualcomm·INT4·2K ctx | 6 | 11tok/s | , | 23 W | $1.2k | 0.48 tok/s/W | $0.0012/k | Amazon | |
| #58 | Intel Core Ultra 9 288V (Lunar Lake) NPUNPU Intel·INT4·2K ctx | 6 | 9tok/s | , | 17 W | $549 | 0.53 tok/s/W | $0.64/M | Amazon | |
| #59 | NVIDIA Jetson Orin Nano 8GBEdge NVIDIA·Q4_K_M·2K ctx | 6 | 8tok/s | 8 GB | 15 W | $499 | 0.53 tok/s/W | $0.66/M | Amazon | |
| #60 | AMD Ryzen AI 9 HX 370 NPUNPU AMD·INT4·2K ctx | 6 | 7tok/s | , | 28 W | $1.5k | 0.25 tok/s/W | $0.0023/k | Amazon | |
| #61 | AMD Ryzen AI 9 365 NPUNPU AMD·INT4·2K ctx | 6 | 6tok/s | , | 28 W | $1.2k | 0.21 tok/s/W | $0.0021/k | Amazon | |
| #62 | Intel Core Ultra 7 258V (Lunar Lake)NPU Intel·INT4·2K ctx | 6 | 6tok/s | , | 17 W | $1.3k | 0.32 tok/s/W | $0.0025/k | Amazon | |
| #63 | Raspberry Pi 5 + AI HAT+ (Hailo-8)Edge Raspberry Pi·INT8·1K ctx | 6 | 1tok/s | 8 GB | 12 W | $150 | 0.10 tok/s/W | $0.0013/k | Amazon |
Sorted by Tokens / sec (high → low)Tip: $/1k tok is a 3-year amortization of MSRP at sustained throughput.
Cite this benchmark
Use this in your paper, blog post, or comparison table.
BibTeX
@misc{myaihardware_llama3-8b-q4_2026,
title = {MyAI Bench: Llama 3 8B (Q4_K_M)},
author = {{MyAIHardware Contributors}},
year = {2026},
url = {https://www.myaihardware.com/benchmarks/workload/llama3-8b-q4},
note = {Version 1.3, accessed 2026-08-27}
}APA
MyAIHardware Contributors. (2026). MyAI Bench: Llama 3 8B (Q4_K_M). MyAIHardware. Retrieved 2026-08-27, from https://www.myaihardware.com/benchmarks/workload/llama3-8b-q4
MLA
MyAIHardware Contributors. "MyAI Bench: Llama 3 8B (Q4_K_M)." MyAIHardware, 2026, https://www.myaihardware.com/benchmarks/workload/llama3-8b-q4. Accessed 2026-08-27.
Plain text
MyAI Bench, Llama 3 8B (Q4_K_M). MyAIHardware Contributors, 2026. Version 1.3. https://www.myaihardware.com/benchmarks/workload/llama3-8b-q4 (accessed 2026-08-27).