NVLink

NVLink is a direct GPU-to-GPU communication fabric developed by NVIDIA, providing significantly higher bandwidth than traditional PCIe (up to 900 GB/s bi-directional on NVLink 4.0 vs 128 GB/s on PCIe Gen5). It is essential for training and inference of large LLMs that exceed a single GPU's HBM capacity, enabling tensor parallelism, pipeline parallelism, and data parallelism across multiple GPUs. NVLink connects GPUs in a fully-connected topology (e.g., 8 GPUs in a DGX H100) or hybrid cube-mesh designs. This allows fast gradient synchronization during training and splitting model layers across GPUs for inference. The key advantage is reduced communication overhead: model-parallel shards can transfer KV cache or activation tensors between GPUs at near-local memory speeds. NVLink is typically paired with NVSwitch for larger clusters (e.g., 256 GPUs in DGX SuperPOD). For AI builders, if you run a 70B model on two A100s with NVLink, you can split the model across GPUs and achieve smooth inference. Without NVLink (e.g., using PCIe only), cross-GPU transfers become a bottleneck, limiting scalability. NVLink also allows GPU memory pooling (unified memory) for workloads exceeding individual capacities. Note: NVLink consumes power and requires compatible motherboard and PSU. For cloud instances (e.g., AWS p4d, Azure ND series), NVLink is standard. When designing inference servers, prefer NVLink-connected GPUs for any model requiring multi-GPU sharding.