Economics of Running 70B, 405B, 671B On-Prem

The Real Cost of Running LLMs On-Prem

Every AI builder eventually hits the same question: should I rent tokens or buy iron? The answer depends on model size, usage volume, and your tolerance for latency. For a 70B parameter dense model like Llama 3.3, you need at least 140GB of VRAM for FP16 weights alone, plus 2–4GB for KV cache per 4K context. That means two RTX 4090s (48GB each) in a single node, or one H100 (80GB) with aggressive quantization. At $0.002 per token from OpenAI, a 70B model generating 100 tokens per second costs $0.20 per second. Over a year of continuous usage, that's $6.3M. On-prem, a dual 4090 rig costs $6,000, plus $200/month for power. The break-even happens at just 3M tokens per month. But that's the simple case. When you scale to 405B or 671B, the math changes drastically. A 405B model in FP16 requires 810GB of VRAM, forcing you into multi-node setups with H100s or MI300X. The interconnect becomes the bottleneck, not the compute. And the power draw of an 8-GPU H100 rack is 10kW, which at $0.12/kWh adds $864/month. This article walks through the hard numbers for three model sizes, with real benchmarks from vLLM and llama.cpp, so you can decide whether on-prem makes sense for your workload.

Hardware Requirements: VRAM, Bandwidth, and Interconnect

Let's start with the raw math. A 70B parameter model in FP16 requires 70e9 * 2 bytes = 140GB for weights. Add 2GB for KV cache per 4K context (assuming 4K tokens, 80 layers, 8KV heads, FP16). So total VRAM needed is ~145GB. With two RTX 4090s (48GB each) you get 96GB, not enough. You need FP8 quantization to fit: 70e9 * 1 byte = 70GB, plus cache, total ~75GB per GPU. Two 4090s with tensor parallelism can run this at ~30 tok/s, limited by PCIe Gen 4 x16 bandwidth (32 GB/s per direction). An H100 with 80GB runs the same model in FP8 entirely on one GPU, achieving ~80 tok/s thanks to 3.35 TB/s HBM3 bandwidth. For 405B, FP16 needs 810GB. With 8x H100s (80GB each) you get 640GB, not enough. You must use FP8: 405GB for weights, plus ~20GB KV cache, total ~425GB. Eight H100s provide 640GB, so you have headroom. But tensor parallelism across 8 GPUs requires NVLink (900 GB/s per GPU) to avoid pipeline bubbles. Without NVLink, PCIe Gen 5 x16 (64 GB/s) becomes the bottleneck, dropping throughput by 40%. For 671B MoE (e.g., DeepSeek-V3), the model has 671B total parameters but only ~37B active per token. In FP8, weights are 671GB, but active parameters are 37GB. However, you still need to store all experts in VRAM to avoid swapping. With 8x H200s (141GB each), you have 1.1TB total, enough for FP8 weights plus cache. But expert parallelism requires all-to-all communication, which saturates NVLink. Real-world throughput on 8x H200s is ~50 tok/s for 671B MoE.

# VRAM calculation for 405B FP8
params = 405e9
bytes_per_param = 1  # FP8
weights_gb = params * bytes_per_param / 1e9  # 405 GB
kv_cache_gb = 2  # per 4K context, 80 layers, 8 KV heads
kv_cache_total = kv_cache_gb * 8  # 8 GPUs
vram_needed = weights_gb + kv_cache_total  # 421 GB
print(f"VRAM needed: {vram_needed:.0f} GB")

Token Throughput and Latency: Real Benchmarks

Throughput is the key metric for economics. For a 70B model on a single H100 with FP8, vLLM achieves 80 tok/s at batch size 1. With continuous batching (batch size 32), throughput jumps to 600 tok/s but latency per request increases to 2 seconds. On two RTX 4090s with tensor parallelism, llama.cpp achieves 30 tok/s at batch size 1, and 200 tok/s with batch size 16. The RTX 4090's memory bandwidth is 1.0 TB/s vs H100's 3.35 TB/s, which directly limits decode speed. For 405B on 8x H100s with NVLink, vLLM reports 150 tok/s at batch size 1, and 1,200 tok/s with batch size 64. Without NVLink, throughput drops to 90 tok/s at batch size 1. For 671B MoE on 8x H200s, TensorRT-LLM achieves 50 tok/s at batch size 1, and 400 tok/s with batch size 32. The MoE model's advantage is lower active parameter count, but the all-to-all communication overhead limits scaling. Key takeaway: for real-time applications (batch size 1), single-GPU throughput is 2-3x higher than multi-GPU due to interconnect latency. For batch inference, multi-GPU scales linearly up to 8 GPUs with NVLink.

Quantization: FP8, INT4, and Quality Trade-offs

Quantization is the lever that makes on-prem feasible. FP8 (E5M2 or E4M3) reduces VRAM by 2x with negligible quality loss for inference. Studies show perplexity increases by less than 0.1 for 70B models. INT4 (e.g., AWQ, GPTQ) reduces VRAM by 4x but can degrade quality, especially on reasoning tasks. For a 70B model, INT4 requires only 35GB for weights, fitting on a single RTX 4090. But perplexity increases by 0.5–1.0, and few-shot accuracy drops by 2–3%. For 405B, INT4 brings weights to 101GB, fitting on 2x H100s (160GB total). However, INT4 inference on H100s is slower than FP8 because H100's tensor cores are optimized for FP8, not INT4. NVIDIA's Transformer Engine accelerates FP8 matrix multiplications by 2x over INT4. For MoE models, quantization is trickier because different experts have different activation distributions. DeepSeek-V3 uses FP8 for both weights and activations, achieving good quality. AWQ can be applied per-expert, but calibration requires representative data. Recommendation: use FP8 for production inference on H100/H200/B200; use INT4 only for memory-constrained setups on consumer GPUs.

# Quantize Llama 3.3 70B to FP8 with TensorRT-LLM
trtllm-build --model_dir ./llama-3.3-70b-hf \
  --dtype float16 \
  --use_fp8 \
  --output_dir ./llama-3.3-70b-fp8 \
  --max_batch_size 64 \
  --max_input_len 4096 \
  --max_output_len 4096

Interconnect Bottlenecks: NVLink vs. PCIe vs. Ethernet

Once you scale beyond one GPU, the interconnect becomes the dominant cost and performance factor. For tensor parallelism, each layer's forward pass requires all-reduce of activations and gradients. With 8 GPUs, an all-reduce on 1MB of data takes 1 microsecond on NVLink (900 GB/s) vs 16 microseconds on PCIe Gen 5 (64 GB/s). That 16x latency difference translates to 40% lower throughput for PCIe-based systems. For pipeline parallelism, the bottleneck is the bandwidth between pipeline stages. With 8 GPUs in a pipeline, each stage sends activations of size batch_size * seq_len * hidden_dim. For 405B, hidden_dim is 16384, so a single activation tensor is 128MB at batch size 1. Sending that over PCIe takes 2ms, while NVLink takes 0.14ms. Over a 200Gbps Ethernet network (e.g., InfiniBand NDR200), it takes 5ms. This is why multi-node inference is 3-5x slower than single-node with NVLink. For MoE models, expert parallelism requires all-to-all communication, which is even more demanding. Each token's hidden state must be sent to the GPU hosting the expert. With 8 GPUs and 256 experts, each GPU sends 32 tokens per expert, leading to high communication volume. NVLink handles this well, but PCIe systems see severe degradation. Recommendation: for models larger than 70B, invest in NVLink-capable GPUs (H100, H200, B200) or use AMD MI300X with Infinity Fabric.

Power, Cooling, and Total Cost of Ownership (TCO)

Hardware is only half the cost. A 4-GPU H100 rig draws 2.5kW under load. At $0.12/kWh, running 24/7 costs $720/month. Add cooling (CRAC or liquid) at 30% overhead, total $936/month. Over 3 years, that's $33,696 just for power and cooling. A 8-GPU H100 rack draws 10kW, costing $3,744/month. Undervolting GPUs can reduce power by 15-20% with minimal performance loss. For example, setting an H100's power limit to 500W (from 700W) reduces throughput by only 5% but saves 29% power. Liquid cooling adds $5,000 upfront but reduces cooling overhead to 10%, saving $100/month. For consumer GPUs like RTX 4090, power draw is 450W per card. A 4-GPU 4090 rig draws 2.0kW, costing $576/month. But 4090s are air-cooled, so no liquid cooling cost. The TCO for a 4x4090 rig over 3 years: hardware $12,000 + power $20,736 = $32,736. For a 4xH100 rig: hardware $120,000 + power $33,696 = $153,696. The H100s are 4x more expensive but deliver 3x more throughput per watt. For high-volume workloads, H100s are more cost-effective per token.

Software Stack: vLLM, TensorRT-LLM, and llama.cpp

The choice of inference engine affects throughput by 2-4x. For production deployments on NVIDIA GPUs, TensorRT-LLM is the fastest, achieving 1.5x the throughput of vLLM on the same hardware. It uses FP8 tensor cores, FlashAttention-2, and PagedAttention. However, it requires model compilation, which takes 30 minutes for a 70B model. vLLM is easier to deploy, supports continuous batching out of the box, and has a Python API. For 70B on H100, vLLM achieves 80 tok/s vs TensorRT-LLM's 120 tok/s. For 405B on 8x H100s, vLLM does 150 tok/s, TensorRT-LLM does 220 tok/s. llama.cpp is best for consumer GPUs and CPU offloading. It supports GGUF quantization (Q4_K_M, Q5_K_M) and runs on a single RTX 4090 with 70B at 15 tok/s. ExLlamaV2 is another option for consumer GPUs, achieving 25 tok/s on 70B with INT4. For AMD GPUs, use MLC-LLM or ROCm-based vLLM. MI300X with ROCm 6.0 achieves 90% of H100 throughput for 70B. For Apple Silicon, MLX is the fastest, running 70B on M3 Ultra at 10 tok/s. Recommendation: use TensorRT-LLM for production NVIDIA setups, vLLM for flexibility, and llama.cpp for consumer hardware.

# Serve 405B FP8 with vLLM on 8x H100s
python -m vllm.entrypoints.openai.api_server \
  --model /path/to/405b-fp8 \
  --tensor-parallel-size 8 \
  --dtype float16 \
  --kv-cache-dtype fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.95

Break-Even Analysis: API vs. On-Prem

Let's compute the break-even point. Assume API cost for 70B is $0.002 per token (OpenAI GPT-4 class). On-prem cost per token includes hardware amortization over 3 years, power, and maintenance. For a 4x4090 rig ($12,000 hardware + $20,736 power over 3 years = $32,736 total), generating 30 tok/s, the cost per token is $32,736 / (30 * 3600 * 24 * 365 * 3) = $0.000012. That's 167x cheaper than API. But if you only use it 10% of the time, the effective cost is $0.00012 per token, still 17x cheaper. For 405B on 8xH100s ($250,000 hardware + $100,000 power over 3 years = $350,000 total), generating 150 tok/s, cost per token is $0.000008. API for 405B-class models is ~$0.005 per token (Anthropic Claude Opus). So on-prem is 625x cheaper at full utilization. But at 10% utilization, cost per token is $0.00008, still 62.5x cheaper. The break-even volume is surprisingly low: for 70B, 3M tokens/month; for 405B, 50M tokens/month. Most serious AI builders generate 10-100M tokens per month, making on-prem economically compelling. However, don't forget opportunity cost: managing hardware, dealing with failures, and scaling up. For teams with less than 5 engineers, API may be better despite higher per-token cost.

Real-World Case Study: Running DeepSeek-V3 (671B) On-Prem

DeepSeek-V3 is a 671B MoE model with 37B active parameters. In FP8, weights are 671GB. With 8x H200s (141GB each), total VRAM is 1.1TB, leaving ~400GB for KV cache. Using TensorRT-LLM with FP8 and expert parallelism, throughput is 50 tok/s at batch size 1, and 400 tok/s at batch size 32. The hardware cost is $300,000 (8x H200 at $37,500 each). Power draw is 10kW, costing $3,744/month. Over 3 years, total cost is $300,000 + $134,784 = $434,784. At 50 tok/s continuous, tokens generated over 3 years = 50 * 3600 * 24 * 365 * 3 = 4.7e12 tokens. Cost per token = $434,784 / 4.7e12 = $0.0000000925. API cost for similar quality (e.g., GPT-4 Turbo) is $0.01 per token. On-prem is 108,000x cheaper. But this assumes 100% utilization. At 10% utilization, cost per token is $0.000000925, still 10,800x cheaper. The real challenge is interconnect: without NVLink, throughput drops to 20 tok/s. Also, DeepSeek-V3 requires custom kernels for MoE routing. TensorRT-LLM supports it, but vLLM does not yet. For teams building on-prem, DeepSeek-V3 is the most cost-effective frontier model, but requires significant engineering effort to deploy.

The Hidden Costs: Maintenance, Cooling, and Scaling

Beyond hardware and power, on-prem has hidden costs. GPU failures: H100s have a 1-2% annual failure rate. Replacing a GPU costs $30,000 and takes 1-2 weeks. For a 8-GPU rig, expected failure cost is $4,800/year. Cooling maintenance: liquid cooling loops need annual fluid replacement ($500) and pump replacement every 5 years ($2,000). Networking: multi-node setups need InfiniBand switches ($20,000) and cables ($1,000). Software: you need a team to manage drivers, kernels, and libraries. NVIDIA's CUDA toolkit updates can break compatibility. ROCm on AMD GPUs is less stable, requiring more debugging. Scaling: adding more GPUs means upgrading power supplies, circuit breakers, and cooling. A 16-GPU H100 rack requires 20kW and a dedicated 30A 240V circuit. Most office buildings need electrical upgrades ($5,000-$10,000). For small teams, these hidden costs can add 20-30% to TCO. Recommendation: budget 30% overhead on top of hardware and power costs for maintenance and scaling.

Further reading

• FlashAttention: Fast and Memory-Efficient Exact Attention
• vLLM: PagedAttention for Efficient LLM Serving
• DeepSeek-V3 Technical Report
• TensorRT-LLM Documentation
• The Economics of Large Language Model Inference