VRAM
24 GB
TDP
350 W
MSRP
$1.5k
Perf/W
0.25 emb/s/W
Cost/1K tok
$0.18/M
Tested
2025-02-25
NVIDIA GeForce RTX 3090 24GB produces approximately 9.0 tok/s on Llama 3 70B at Q4_K_M quantization (batch 1, 4096-token context, 24GB VRAM, 350W TDP). That figure comes from 1 measured run on llama.cpp. The 70B model needs roughly 40GB of VRAM at Q4, so headroom and KV cache budget matter as much as raw throughput.
NVIDIA GeForce RTX 3090 24GB — Ampere flagship from 2020. 24GB GDDR6X at ~936 GB/s, 350W TDP. Same VRAM capacity as RTX 4090 but ~60% the throughput on AI workloads. No FP8 acceleration.
The value king for VRAM-hungry local AI. 24GB runs 32B Q4 full-GPU at ~35-45 tok/s. Used market pricing (~$600-800) makes it the best $/VRAM in consumer hardware. NVLink support on some models enables dual-3090 rigs with 48GB combined VRAM — the budget workstation builder's secret weapon. Slower on prompt processing than Ada/Blackwell but decode speed is bandwidth-bound and competitive.
Buy used if you can find a clean unit under $700. The value-for-VRAM proposition is unmatched — nothing else gives you 24GB at this price. Pair with a second used 3090 for 48GB combined VRAM at ~$1,400 total.
32B Q4 full-GPU at 35-45 tok/s — or dual 3090s for 70B Q4 at 25-35 tok/s with 48GB combined
VRAM thermals under sustained load — GDDR6X runs hot. Repadding recommended for used cards. Fan bearings on mining cards typically the first mechanical failure.
Ubuntu 24.04 (excellent), Windows 11 (excellent), WSL2 (excellent), macOS (unsupported)
350W TGP. Used cards may draw more due to degraded thermal paste. Undervolt recommended for 24/7 operation. ~$10/month electricity for single card at 4hrs/day US average.
NVIDIA GeForce RTX 3090 24GB with 24GB VRAM at 350W TDP, scored across 10 workloads with 15 benchmark records.
Best workload
Embedding throughput
4900 emb/s
Quantization
FP16
1K context · batch 1
| Workload | Score | Quant | Context | σ | Status | Tested |
|---|---|---|---|---|---|---|
| Llama 3 8B Q4 llm | 88.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2025-02-25 |
| Llama 3 70B Q4 llm | 9.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2025-04-25 |
| SDXL image gen image | 12.0img/min | FP16 | , | , | Curated Aggregate | 2024-12-10 |
| Whisper transcription audio | 56.0x RT | FP16 | , | , | Curated Aggregate | 2024-08-17 |
| Qwen 2.5 14B llm | 55.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2025-03-08 |
| Gemma 2 9B llm | 65.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2024-06-12 |
| DeepSeek-R1 7B llm | 82.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2025-01-29 |
| Embedding throughput embedding | 4900.0emb/s | FP16 | 1K | , | Curated Aggregate | 2024-05-18 |
| Mistral 7B llm | 78.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2024-04-22 |
| Asure 12B llm | 38.0tok/s | Q4_K_M | 8K | ±2.3 | Curated Aggregate | 2025-05-01 |
NVIDIA GeForce RTX 3090 24GB clears a 8.6/10 based on workload-normalized throughput, memory headroom, efficiency, value, trust, and coverage.
What models fit this 24GB card at different quantization levels.
Trust score
6/10
MyAI rating
8.6
Runs
1
Freshness
Stale
Source-linked row with explicit verification status.
Tested on 2024-05-18; 831 days old.
Best place to start
Retailer we'd check first
Amazon search
For mainstream AI hardware, Amazon usually updates street pricing, seller availability, and shipping speed faster than most comparison sites.
Search Amazon listingsThis is a strong buy when the listing stays close to reference pricing and matches the workload you actually run.
Affiliate note: we do not have a device-level ASIN yet, so this opens tagged Amazon search results for the exact product name.