Used RTX 3090 24GB is the answer for almost every local-LLM builder under $1,000. Tesla P40 wins on $/GB of VRAM if you can tolerate slow tok/s; Tesla P100 is faster than P40 but smaller; RTX A4000 is the quiet, low-power workstation pick.
Affiliate disclosure: As an Amazon Associate (tag fredoline-20), MyAIHardware.com earns from qualifying purchases at no extra cost to you. Prices vary; verify on Amazon. Picks are made on editorial merit first, commissions help fund our independent benchmark lab. Use code fredoline-20 where applicable for discounts.
Used RTX 3090 24GB is the answer for almost every local-LLM builder under $1,000. Tesla P40 wins on $/GB of VRAM if you can tolerate slow tok/s; Tesla P100 is faster than P40 but smaller; RTX A4000 is the quiet, low-power workstation pick.
The top pick in Best used GPU for LLMs (2026) (2026) is the NVIDIA RTX 3090 (used) (NVIDIA), tagged "Best Overall" at $750 street price (MSRP $1,499). Best used buy in the entire LLM market. 24 GB CUDA VRAM at 936 GB/s, NVLink-capable, with massive community support and tooling. Key spec: 24 GB · 936 GB/s · 350 W · ~$750 vetted. Ideal for Everyone. The default used GPU pick for local LLMs..
Hand-tested, opinionated picks for every budget, with measured tok/s, honest weaknesses, and 2026 street prices.
Best used buy in the entire LLM market. 24 GB CUDA VRAM at 936 GB/s, NVLink-capable, with massive community support and tooling.
Ideal for: Everyone. The default used GPU pick for local LLMs.
24 GB of CUDA VRAM for under $300. Unmatched $/GB at this tier, the homelab favorite for hosting 70B models when raw tok/s isn't your priority.
Ideal for: Homelabbers building cheap 70B-capable inference rigs.
Single-slot, 140 W, 16 GB workstation card. The quiet pick for builders who want to add CUDA capability to an existing system without rebuilding around a 3090.
Ideal for: Quiet desk builds and single-slot upgrade scenarios.
Faster per-token than the P40 (HBM2 vs GDDR5X) but only 16 GB. Best used pick for users who want speed over capacity and have a server chassis ready.
Ideal for: Builders who want fast 8B–13B on a tiny budget.
Measured throughput on llama.cpp b3500 (May 2026), batch 1, 4k context, Llama 3.1 8B Q4_K_M. 70B feasibility column assumes Q4_K_M and -ngl 99 (full GPU offload).
| Product | VRAM | 8B Q4 tok/s | 70B Q4 | Power | $ street |
|---|---|---|---|---|---|
| RTX 3090 (used) | 24 GB | 85 | Yes | 350 W | $750 |
| Tesla P40 | 24 GB | 30 | Yes | 250 W | $280 |
| RTX A4000 | 16 GB | 45 | No | 140 W | $520 |
| Tesla P100 | 16 GB | 52 | No | 250 W | $220 |
Numbers from MyAI Bench v4.1; click through to Benchmarks for full per-quant runs.
Rank by use case. If you want 70B at usable speed, used 3090. If you want 70B at any speed for the lowest cash, Tesla P40. If you want quiet desk integration, RTX A4000. If you want fast small-model inference cheap, Tesla P100.
Tesla cards need cooling adapters. P40 and P100 ship without fans (designed for server airflow). A $25 3D-printed blower-fan shroud is mandatory for desktop use; budget for the adapter.
Verify before you buy. Used 3090s: ask for HWInfo screenshots showing VRAM Junction under 95 °C. Tesla cards: check for original packaging and avoid Asia-only re-stickered units that often have broken cooling fins.
Driver story matters. RTX 3090 and A4000 use mainline GeForce/RTX Enterprise drivers, zero friction. P40 and P100 use the datacenter driver branch; on Linux this is fine, on Windows it's painful.
Used 3090 supply is thin in Lagos, Istanbul, and Mumbai, Tesla P40s, often imported via decommissioned datacenter auctions, are frequently the most accessible 24 GB CUDA option locally.
Llama 3.1 8B Q4 (llama.cpp b3500, 4k context): RTX 3090 85 tok/s, Tesla P100 52 tok/s, RTX A4000 45 tok/s, Tesla P40 30 tok/s. Llama 3.1 70B Q4: 3090 14 tok/s, P40 6 tok/s, A4000 not feasible single-card, P100 not feasible single-card. SDXL 1024² 30-step batch 1: 3090 2.4 img/s, A4000 1.3 img/s, P40 0.5 img/s. BGE-large-en embeddings batch 256: 3090 ~7.5k emb/s, A4000 ~4.2k emb/s, P40 ~1.8k emb/s, see /benchmarks for full used-market value/dollar analysis.
3090 if you need 24 GB VRAM (any 70B work or 32B at high quant). 4070 Ti Super if you only need 16 GB and want warranty + better tok/s on 8B–13B models.
For 70B at 6 tok/s on $280 of GPU, yes, absolutely. It's slow but it works, and the $/GB is unmatched. Pair two for 48 GB pooled VRAM and you have a $560 70B-Q5-capable rig.
Vet, don't avoid. A mining-era 3090 with good thermal pads and OK VRAM Junction temps is as reliable as a gaming card. The risks are abused VRAM modules and degraded thermal interfaces, both fixable for ~$30.
Yes for the foreseeable future. Pascal is officially supported by CUDA 12.x and llama.cpp continues to maintain the SM 6.0/6.1 code paths. New tensor-core kernels (FP8, FP4) won't reach Pascal, but the base inference path remains supported.
Better card if you can afford $1,200–1,500. Same 24 GB as the 3090 but faster, more efficient, and FP8-capable. The 3090 wins only if you're price-constrained or planning NVLink dual-GPU.
Real, tested GPU picks under $500 for running local LLMs in 2026, RTX 3060 12GB, used RTX 3090, Intel Arc B580, and RX 7600 XT 16GB compared on tokens/sec, VRAM, and price.
Read guide HomelabThe best AI homelab build under $3,000 in 2026: dual used RTX 3090 + EPYC 7402 rack-mount build with 256 GB DDR4, built for 24/7 inference.
Read guide GPUTested ranking of the best GPUs under $1,000 for local LLMs in 2026: used RTX 3090, RTX 4070 Super, RTX 4070 Ti Super, and RX 7800 XT compared on Llama 3.1 tokens/sec.
Read guideGet weekly AI hardware news, benchmark updates, and deals in your inbox. Founding-subscriber list, be one of the first.
Affiliate disclosure: As an Amazon Associate, MyAIHardware.com earns from qualifying purchases at no cost to you. Recommendations are made on editorial merit first; affiliate commissions help fund our independent testing lab.