Multi-GPU Inference With vLLM and llama.cpp

Tensor parallel vs pipeline parallel

There are two ways to split a model across multiple GPUs. Tensor parallelism splits each layer's matrices across cards, every card holds a slice of every layer, and they communicate per-token over PCIe or NVLink. This is the fast option but it demands high inter-GPU bandwidth. Pipeline parallelism puts entire layers on different cards, card 0 owns layers 0-39, card 1 owns 40-79, and a token flows through them sequentially. This needs almost no inter-GPU bandwidth but adds latency per token.

vLLM uses tensor parallel by default. llama.cpp historically used pipeline parallel (because it works fine over PCIe), but its recent versions also support tensor parallel for power users. The right choice depends on your hardware: NVLinked cards = tensor parallel always wins. PCIe-only cards = pipeline can be faster for small batch sizes.

Step 1: Confirm the kernel sees all GPUs

Before any framework setup, the OS must see every GPU. Run nvidia-smi and check for two-or-more lines in the device table. If only one card shows up, suspect PCIe slot configuration in BIOS, many consumer boards split the lanes when you populate the second slot, and some boards require manually setting 'x8/x8 mode'. The motherboard manual will tell you.

nvidia-smi --list-gpus
# Should print 2+ lines:
# GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-...)
# GPU 1: NVIDIA GeForce RTX 4090 (UUID: GPU-...)

# Detailed topology
nvidia-smi topo -m
# Look for PIX (PCIe switch) or NV# (NVLink) between GPUs.
# PHB (host bridge) is the slowest legit topology, works but slow.

Step 2: Multi-GPU with Ollama (easiest)

Ollama is the lowest-friction path. With no configuration at all, Ollama 0.5+ detects multiple GPUs and uses pipeline parallelism, layers spread automatically. For most home users this is the only multi-GPU setup they ever need. You only graduate to vLLM if you need batching (multiple simultaneous users) or higher throughput.

# With two 24 GB cards, Ollama will use 48 GB of VRAM total.
# This lets you run models that wouldn't fit on one card.
ollama pull llama3.3:70b
ollama run llama3.3:70b

# Verify both GPUs are in use:
nvidia-smi -l 1   # refresh every second, both GPUs should show load

Step 3: Multi-GPU with vLLM (highest throughput)

vLLM is the production-grade inference server. It supports continuous batching, paged attention, and tensor parallelism. Use it when you need throughput, say, serving 20 simultaneous chat sessions. Install via pip in a fresh venv (vLLM has many CUDA dependencies and will fight a polluted Python env).

# Install vLLM
python3.11 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip install --upgrade pip
pip install vllm

# Serve Llama 3 70B with tensor parallel across 2 GPUs
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192 \
  --quantization awq          # if you pulled an AWQ-quantized variant

# vLLM exposes an OpenAI-compatible API on http://localhost:8000

Step 4: Validate tensor parallel actually works

The gotcha with tensor parallel is that it silently downgrades to pipeline parallel if NCCL can't establish a fast peer connection between GPUs. Test peer-to-peer access, if it's not enabled, your tensor-parallel build will run 10x slower than expected.

# Check peer-to-peer access
import torch
print("CUDA devices:", torch.cuda.device_count())
for i in range(torch.cuda.device_count()):
    for j in range(torch.cuda.device_count()):
        if i != j:
            can = torch.cuda.can_device_access_peer(i, j)
            print(f"  GPU {i} -> GPU {j}: peer access = {can}")

# If any pair reports False on consumer hardware, you're stuck with PCIe.
# Nothing you can do about it, NVIDIA artificially disables P2P on RTX.

Step 5: When NVLink matters (and when it doesn't)

NVLink between two GPUs gives you 600 GB/s of inter-GPU bandwidth vs PCIe 4.0 x16's ~32 GB/s. For tensor-parallel inference of a 70B model, NVLink roughly doubles throughput vs PCIe. For Ollama-style pipeline parallel, it makes almost no difference. NVIDIA artificially removed NVLink from the RTX 40 and 50 series, so this matters mostly to people running A6000s or older 3090 SLI setups.

Step 6: Quad-GPU and beyond

Going from 2 to 4 GPUs needs a workstation board with enough PCIe lanes (Threadripper Pro or EPYC). Consumer boards split lanes the more cards you add, and below x4 per card tensor parallel suffers. Power: a quad-4090 box pulls 1800W transient under load, you need a 2000W PSU or two PSUs ganged. At that point you have an underpowered datacenter rack, and renting an H100 hour for $2 starts to look smart for occasional training jobs.