VRAM
192 GB
TDP
750 W
MSRP
$18k
Perf/W
0.10 emb/s/W
Cost/1K tok
$0.0026/k
Tested
2025-07-11
AMD Instinct MI300X 192GB produces approximately 72.0 tok/s on Llama 3 70B at Q4_K_M quantization (batch 1, 8192-token context, 192GB VRAM, 750W TDP). That figure comes from 5 measured runs on vLLM 0.6.3 ROCm 6.2. The 70B model needs roughly 40GB of VRAM at Q4, so headroom and KV cache budget matter as much as raw throughput.
AMD Instinct MI300X 192GB — CDNA3-architecture datacenter APU. 192GB HBM3 at 5.3 TB/s, 750W TDP. AMD's answer to the H100/H200 for AI workloads. ROCm software stack maturing through 2025-2026.
192GB VRAM at competitive bandwidth makes this a credible H200 alternative for inference. vLLM ROCm support is production-grade. Llama 3.1 70B Q4 runs at ~120+ tok/s. The software tax (ROCm vs CUDA) is the real consideration — fewer engines support it, and community tooling is thinner. Best for operators who already have AMD infrastructure or who are specifically chasing $/VRAM ratios.
Best for organizations with existing AMD infrastructure or those specifically chasing $/VRAM ratios at datacenter scale. The software maturity gap vs NVIDIA is real but narrowing.
Large-model inference where VRAM capacity matters more than peak compute — 405B Q4 with headroom, 70B FP16 comfortably
ROCm driver compatibility with specific kernel versions — pin your ROCm + kernel combination once stable. vLLM ROCm branch occasionally lags upstream.
Ubuntu 22.04 LTS (reference), RHEL 9, other Linux (ROCm-supported), Windows (unsupported), macOS (unsupported)
750W TDP. Datacenter power and cooling required. Not suitable for residential. ROCm power management less mature than NVIDIA's — expect higher idle draw.
AMD Instinct MI300X 192GB with 192GB VRAM at 750W TDP, scored across 12 workloads with 20 benchmark records.
Best workload
Embedding throughput
9800 emb/s
Quantization
FP16
1K context · batch 1
| Workload | Score | Quant | Context | σ | Status | Tested |
|---|---|---|---|---|---|---|
| Llama 3 70B Q4 llm | 380.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2024-09-12 |
| Llama 3 70B Q8 llm | 48.0tok/s | Q8_0 | 8K | , | Curated Aggregate | 2025-07-15 |
| Llama 3 8B FP16 llm | 260.0tok/s | FP16 | 8K | , | Curated Aggregate | 2025-07-30 |
| SDXL image gen image | 22.0img/min | FP16 | , | , | Curated Aggregate | 2024-12-22 |
| Embedding throughput embedding | 9800.0emb/s | FP16 | 1K | , | Curated Aggregate | 2024-11-30 |
| Qwen 2.5 14B llm | 142.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2024-12-04 |
| DeepSeek-R1 7B llm | 215.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2026-03-19 |
| Gemma 2 9B llm | 188.0tok/s | Q4_K_M | 8K | , | Curated Aggregate | 2025-12-08 |
| Mistral 7B llm | 268.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2025-05-19 |
| Llama 3 8B Q4 llm | 920.0tok/s | Q4_K_M | 4K | , | Curated Aggregate | 2024-12-08 |
| Whisper transcription audio | 95.0x RT | FP16 | , | , | Curated Aggregate | 2025-01-12 |
| Asure 12B llm | 130.0tok/s | Q4_K_M | 8K | ±6.5 | Curated Aggregate | 2025-05-01 |
AMD Instinct MI300X 192GB clears a 9.3/10 based on workload-normalized throughput, memory headroom, efficiency, value, trust, and coverage.
What models fit this 192GB card at different quantization levels.
Trust score
6/10
MyAI rating
9.3
Runs
1
Freshness
Stale
Source-linked row with explicit verification status.
Tested on 2024-11-30; 635 days old.
Enterprise buying note
Retailer we'd check first
Amazon search plus reseller quotes
For datacenter and accelerator parts, Amazon is useful for spotting live listings, accessories, or used pulls, but serious procurement usually happens through integrators, brokers, or cloud partners.
Enterprise procurement, not retail
This silicon is typically acquired through an authorized OEM partner, system integrator, or hyperscaler reseller. For on-demand access, compare hourly rates at RunPod, Vast.ai, Lambda Labs, or your existing cloud provider before committing to capital expenditure.
This is one of the highest-scoring parts in our database, so it is worth tracking when inventory lands near fair-market pricing.
Affiliate note: we do not have a device-level ASIN yet, so this opens tagged Amazon search results for the exact product name.