VRAM
24 GB
TDP
450 W
MSRP
$1.6k
Perf/W
0.29 emb/s/W
Cost/1K tok
$0.13/M
Tested
2025-10-02
NVIDIA GeForce RTX 4090 24GB produces approximately 14.0 tok/s on Llama 3 70B at Q4_K_M quantization (batch 1, 4096-token context, 24GB VRAM, 450W TDP). That figure comes from 1 measured run on llama.cpp. The 70B model needs roughly 40GB of VRAM at Q4, so headroom and KV cache budget matter as much as raw throughput.
NVIDIA GeForce RTX 4090 is the Ada Lovelace flagship released in 2022. 24GB GDDR6X at ~1 TB/s bandwidth, 450W TDP, 4th-gen Tensor Cores with FP8 acceleration. The community-default high-end local-AI card from 2022-2025 with universal CUDA software support.
24GB VRAM comfortably runs 32B-class models at Q4 (Qwen 3 32B, Qwen 2.5 Coder 32B) at 70+ tok/s. 70B Q4 requires partial offload to system RAM, dropping to ~22-28 tok/s. The sweet-spot card for solo coding agents and 32B-class daily drivers. Used market pricing makes it compelling vs MSRP 5090.
Buy if you want the best single-card experience for 32B-class models and either bought at MSRP or found a clean used unit. Skip if 70B is your daily target or if 16GB cards cover your needs.
32B-class Q4 full-GPU at 70+ tok/s — Qwen 3 32B, Qwen 2.5 Coder 32B, QwQ 32B all fit comfortably
VRAM at 32K+ context — 70B Q4 at 32K context uses ~20GB for KV cache alone. Budget context length carefully.
Ubuntu 24.04 (excellent), Windows 11 (excellent), WSL2 (excellent), macOS (unsupported)
450W TGP, ~320-380W sustained decode. 850W PSU minimum, 1000W recommended. ~$8-10/month electricity at US average for 4hrs/day. Undervolt -100mV for zero perf cost thermal improvement.
NVIDIA GeForce RTX 4090 24GB with 24GB VRAM at 450W TDP, scored across 12 workloads with 23 benchmark records.
Best workload
Embedding throughput
8500 emb/s
Quantization
FP16
1K context · batch 1
| Workload | Score | Quant | Context | σ | Status | Tested |
|---|---|---|---|---|---|---|
| Llama 3 8B Q4 llm | 540.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2024-08-21 |
| Llama 3 8B FP16 llm | 72.0tok/s | FP16 | 4K | , | Curated Aggregate | 2024-12-22 |
| Llama 3 70B Q4 llm | 14.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2026-03-08 |
| Mistral 7B llm | 145.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2026-03-19 |
| SDXL image gen image | 24.0img/min | FP16 | , | , | Curated Aggregate | 2024-06-22 |
| Whisper transcription audio | 78.0x RT | FP16 | , | , | Curated Aggregate | 2025-03-08 |
| Embedding throughput embedding | 8500.0emb/s | FP16 | 1K | , | Curated Aggregate | 2024-07-15 |
| DeepSeek-R1 7B llm | 125.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2026-05-25 |
| Gemma 2 9B llm | 108.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2024-11-02 |
| Qwen 2.5 14B llm | 78.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2024-11-19 |
| Phi-3 Mini llm | 285.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2025-08-12 |
| Asure 12B llm | 60.0tok/s | Q4_K_M | 8K | ±3.2 | Curated Aggregate | 2025-05-01 |
NVIDIA GeForce RTX 4090 24GB clears a 9.1/10 based on workload-normalized throughput, memory headroom, efficiency, value, trust, and coverage.
What models fit this 24GB card at different quantization levels.
Trust score
6/10
MyAI rating
9.1
Runs
1
Freshness
Stale
Source-linked row with explicit verification status.
Tested on 2024-07-15; 773 days old.
Best place to start
Retailer we'd check first
Amazon
For mainstream AI hardware, Amazon usually updates street pricing, seller availability, and shipping speed faster than most comparison sites.
View current Amazon listingThis is one of the highest-scoring parts in our database, so it is worth tracking when inventory lands near fair-market pricing.
Affiliate note: this button opens the mapped Amazon product listing for this device. We may earn from qualifying purchases.