Pipeline Parallelism

Pipeline parallelism is a model parallelism technique where the layers of a neural network are divided across multiple GPUs, with each GPU processing a contiguous subset of layers. For an LLM with L layers and P GPUs, each GPU will handle approximately L/P layers. During inference, a micro-batch of tokens passes through GPU 0 (layers 1..L/P), then the intermediate activations are sent to GPU 1 (layers L/P+1..2L/P), continuing sequentially until the output is produced. Unlike tensor parallelism, which splits each layer across GPUs, pipeline parallelism keeps each entire layer on one GPU. This reduces inter-GPU communication per layer to just a single activation tensor transfer, rather than multiple all-reduce calls. The main benefit is that it allows models with very large memory requirements to be distributed across many GPUs, even if those GPUs are not tightly interconnected (e.g., across nodes via standard network). For example, a 70B model with 80 layers can be split across 8 GPUs, each holding 10 layers, requiring about 35GB of memory per GPU for FP16 weights (plus KV cache). Pipeline parallelism is a key component in both training (e.g., GPipe, Megatron-LM) and inference frameworks like vLLM and Hugging Face TGI. However, it suffers from a problem called pipeline bubbles: at the start of a batch, GPUs near the tail are idle waiting for data from the front. Techniques like gradient accumulation and micro-batching reduce this inefficiency in training. In inference, batch size is typically smaller, so pipeline parallelism is often combined with tensor parallelism (3D parallelism) for large-scale deployments.