VRAM
80 GB
TDP
700 W
MSRP
$25k
Perf/W
0.40 emb/s/W
Cost/1K tok
$0.94/M
Tested
2025-01-05
NVIDIA H100 SXM5 80GB produces approximately 66.0 tok/s on Llama 3 70B at Q4_K_M quantization (batch 1, 4096-token context, 80GB VRAM, 700W TDP). That figure comes from 5 measured runs on TensorRT-LLM 0.10. The 70B model needs roughly 40GB of VRAM at Q4, so headroom and KV cache budget matter as much as raw throughput.
NVIDIA H100 SXM5 80GB — Hopper-architecture datacenter GPU. 80GB HBM3 at 3.35 TB/s, 700W TDP, with Transformer Engine for FP8 acceleration. NVLink and NVSwitch for scale-out. The reference platform for production LLM serving through 2025.
Comfortably hosts 70B FP16 with substantial context. 405B at Q4 fits with headroom. Production-grade throughput at batch sizes 8-64 via TensorRT-LLM or vLLM. The default datacenter inference GPU — used by every major cloud provider. Overkill for single-user workloads; right answer for multi-tenant serving and training runs.
The default datacenter inference GPU through 2025. Right answer for production multi-tenant serving. Not for homelab — this is enterprise/cloud territory.
70B FP16 with substantial context, 405B Q4 with headroom. Production multi-tenant serving at batch 8-64 via TensorRT-LLM or vLLM.
Thermal throttling at sustained 700W without proper datacenter cooling. NVLink fabric errors under heavy multi-GPU load if interconnect cables aren't properly seated.
Ubuntu 22.04/24.04 LTS (reference), RHEL 9, Windows Server (limited)
700W SXM. Requires datacenter power infrastructure. Not suitable for residential electrical. Paired with 2kW+ per node when accounting for CPU, memory, and cooling overhead.
NVIDIA H100 SXM5 80GB with 80GB VRAM at 700W TDP, scored across 13 workloads with 21 benchmark records.
Best workload
Embedding throughput
12400 emb/s
Quantization
FP16
1K context · batch 1
| Workload | Score | Quant | Context | σ | Status | Tested |
|---|---|---|---|---|---|---|
| Llama 3 8B FP16 llm | 282.0tok/s | FP16 | 4K | ±11.3 | Vendor Claim | 2025-01-05 |
| Llama 3 70B Q4 llm | 410.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2024-08-30 |
| Llama 3 70B Q8 llm | 42.0tok/s | Q8_0 | 4K | , | Curated Aggregate | 2024-06-20 |
| SDXL image gen image | 28.0img/min | FP16 | , | , | Curated Aggregate | 2026-04-09 |
| Whisper transcription audio | 145.0x RT | FP16 | , | , | Curated Aggregate | 2024-08-16 |
| Embedding throughput embedding | 12400.0emb/s | FP16 | 1K | , | Curated Aggregate | 2025-01-12 |
| DeepSeek-R1 7B llm | 285.0tok/s | Q4_K_M | 16K | , | Curated Aggregate | 2025-04-25 |
| Gemma 2 9B llm | 235.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2025-05-22 |
| Qwen 2.5 14B llm | 168.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2024-10-21 |
| Mistral 7B llm | 295.0tok/s | Q4_K_M | 16K | , | Curated Aggregate | 2025-05-08 |
| Phi-3 Mini llm | 612.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2025-06-04 |
| Llama 3 8B Q4 llm | 178.0tok/s | Q4_K_M | 128K | , | Curated Aggregate | 2024-11-08 |
| Asure 12B llm | 195.0tok/s | Q4_K_M | 8K | ±7.8 | Vendor Claim | 2025-05-01 |
NVIDIA H100 SXM5 80GB clears a 9.4/10 based on workload-normalized throughput, memory headroom, efficiency, value, trust, and coverage.
What models fit this 80GB card at different quantization levels.
Trust score
6/10
MyAI rating
9.4
Runs
1
Freshness
Stale
Source-linked row with explicit verification status.
Tested on 2025-01-12; 592 days old.
Enterprise buying note
Retailer we'd check first
Amazon search plus reseller quotes
For datacenter and accelerator parts, Amazon is useful for spotting live listings, accessories, or used pulls, but serious procurement usually happens through integrators, brokers, or cloud partners.
Enterprise procurement, not retail
This silicon is typically acquired through an authorized OEM partner, system integrator, or hyperscaler reseller. For on-demand access, compare hourly rates at RunPod, Vast.ai, Lambda Labs, or your existing cloud provider before committing to capital expenditure.
This is one of the highest-scoring parts in our database, so it is worth tracking when inventory lands near fair-market pricing.
Affiliate note: we do not have a device-level ASIN yet, so this opens tagged Amazon search results for the exact product name.