VRAM
80 GB
TDP
700 W
MSRP
$25k
Perf/W
0.40 img/min/W
Cost/1K tok
$0.94/M
Tested
2025-01-05
NVIDIA H100 SXM5 80GB has a source-attributed result of 66.0 tok/s on Llama 3 70B Q4 (batch 1, 4096-token context, Q4_K_M; runtime not documented). This is a reference report, not an independently verified lab result. A 70B Q4_K_M artifact needs roughly 40GB for weights alone; smaller memory configurations require explicit offload or model splitting and do not establish full-GPU residency.
NVIDIA H100 SXM5 80GB — Hopper-architecture datacenter GPU. 80GB HBM3 at 3.35 TB/s, 700W TDP, with Transformer Engine for FP8 acceleration. NVLink and NVSwitch for scale-out. The reference platform for production LLM serving through 2025.
80GB can accommodate many 70B quantized artifacts with context-dependent headroom. 70B FP16 weights need about 140GB and 405B Q4 weights over 200GB; those require multiple devices or offload. Batch throughput must not be treated as individual response speed.
The default datacenter inference GPU through 2025. Right answer for production multi-tenant serving. Not for homelab — this is enterprise/cloud territory.
70B quantized inference. 70B FP16 and 405B Q4 require more memory than one 80GB device.
Thermal throttling at sustained 700W without proper datacenter cooling. NVLink fabric errors under heavy multi-GPU load if interconnect cables aren't properly seated.
Ubuntu 22.04/24.04 LTS (reference), RHEL 9, Windows Server (limited)
700W SXM. Requires datacenter power infrastructure. Not suitable for residential electrical. Paired with 2kW+ per node when accounting for CPU, memory, and cooling overhead.
NVIDIA H100 SXM5 80GB with 80GB VRAM at 700W TDP, scored across 13 workloads with 21 benchmark records.
Reference workload
SDXL image gen
28 img/min
Quantization
FP16
· batch 1
| Workload | Score | Quant | Context | σ | Status | Tested |
|---|---|---|---|---|---|---|
| Llama 3 8B FP16 llm | 282.0tok/s | FP16 | 4K | , | Curated Aggregate | 2025-01-05 |
| Llama 3 70B Q4 llm | 66.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2024-06-24 |
| Llama 3 70B Q8 llm | 42.0tok/s | Q8_0 | 4K | , | Curated Aggregate | 2024-06-20 |
| SDXL image gen image | 28.0img/min | FP16 | , | , | Curated Aggregate | 2026-04-09 |
| Whisper transcription audio | 145.0x RT | FP16 | , | , | Curated Aggregate | 2024-08-16 |
| Embedding throughput embedding | 12400.0emb/s | FP16 | 1K | , | Curated Aggregate | 2025-01-12 |
| DeepSeek-R1 7B llm | 285.0tok/s | Q4_K_M | 16K | , | Curated Aggregate | 2025-04-25 |
| Gemma 2 9B llm | 235.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2025-05-22 |
| Qwen 2.5 14B llm | 165.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2025-04-12 |
| Mistral 7B llm | 295.0tok/s | Q4_K_M | 16K | , | Curated Aggregate | 2025-05-08 |
| Phi-3 Mini llm | 612.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2025-06-04 |
| Llama 3 8B Q4 llm | 178.0tok/s | Q4_K_M | 128K | , | Curated Aggregate | 2024-11-08 |
| Asure 12B llm | 195.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2025-05-01 |
There is insufficient comparable evidence to score NVIDIA H100 SXM5 80GB. A score needs results for multiple devices with matching workload, runtime version, quantization, context and batch size.
Planning estimates for 80GB: weights plus at least 2GB or 10% overhead. Actual KV cache depends on model, context and cache format; confirm the artifact before buying.
Trust score
6/10
MyAI rating
Not scored
Runs
Not documented
Freshness
Aging
Source-linked row with explicit verification status.
Record date: 2026-04-09; 153 days old.
Enterprise buying note
Retailer we'd check first
Amazon search plus reseller quotes
For datacenter and accelerator parts, Amazon is useful for spotting live listings, accessories, or used pulls, but serious procurement usually happens through integrators, brokers, or cloud partners.
Enterprise procurement, not retail
This silicon is typically acquired through an authorized OEM partner, system integrator, or hyperscaler reseller. For on-demand access, compare hourly rates at RunPod, Vast.ai, Lambda Labs, or your existing cloud provider before committing to capital expenditure.
There is insufficient comparable evidence for a purchase recommendation. Check model fit, software support and a current seller quote.
Affiliate note: we do not have a device-level ASIN yet, so this opens tagged Amazon search results for the exact product name.