VRAM
40 GB
TDP
400 W
MSRP
$9.0k
Perf/W
0.10 tok/s/W
Cost/1K tok
$0.0023/k
Tested
2024-09-02
NVIDIA A100 40GB produces approximately 42.0 tok/s on Llama 3 70B at Q4_K_M quantization (batch 1, 4096-token context, 40GB VRAM, 400W TDP). That figure comes from 1 measured run on llama.cpp. The 70B model needs roughly 40GB of VRAM at Q4, so headroom and KV cache budget matter as much as raw throughput.
NVIDIA A100 SXM4 40GB — Ampere datacenter GPU with 40GB HBM2e at 1.6 TB/s, 400W TDP. The smaller-memory variant of the GPU that trained GPT-3. Still widely deployed.
40GB HBM2e hosts 70B Q4 with moderate context. ~130 tok/s on 12B Q4. The most accessible datacenter GPU tier — available on every major cloud. Best for: cloud-based inference up to 70B, training runs up to 13B, and organizations with existing A100 reservations. The 40GB ceiling is the constraint vs 80GB variant.
NVIDIA A100 40GB with 40GB VRAM at 400W TDP, scored across 4 workloads with 4 benchmark records.
Best workload
Llama 3 8B FP16
195 tok/s
Quantization
FP16
4K context · batch 1
| Workload | Score | Quant | Context | σ | Status | Tested |
|---|---|---|---|---|---|---|
| Llama 3 70B Q4 llm | 42.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2024-09-02 |
| Llama 3 8B FP16 llm | 195.0tok/s | FP16 | 4K | , | Curated Aggregate | 2024-09-09 |
| Mistral 7B llm | 162.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2024-04-22 |
| Gemma 2 9B llm | 130.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2024-07-20 |
NVIDIA A100 40GB clears a 8.5/10 based on workload-normalized throughput, memory headroom, efficiency, value, trust, and coverage.
What models fit this 40GB card at different quantization levels.
Trust score
6/10
MyAI rating
8.5
Runs
1
Freshness
Stale
Source-linked row with explicit verification status.
Tested on 2024-09-09; 717 days old.
Enterprise buying note
Retailer we'd check first
Amazon search plus reseller quotes
For datacenter and accelerator parts, Amazon is useful for spotting live listings, accessories, or used pulls, but serious procurement usually happens through integrators, brokers, or cloud partners.
Enterprise procurement, not retail
This silicon is typically acquired through an authorized OEM partner, system integrator, or hyperscaler reseller. For on-demand access, compare hourly rates at RunPod, Vast.ai, Lambda Labs, or your existing cloud provider before committing to capital expenditure.
This is a strong buy when the listing stays close to reference pricing and matches the workload you actually run.
Affiliate note: we do not have a device-level ASIN yet, so this opens tagged Amazon search results for the exact product name.