FP16

FP16, short for floating-point 16, is a binary number format that uses 16 bits to represent a wide dynamic range of values, with 1 sign bit, 5 exponent bits, and 10 mantissa bits. In the context of AI hardware and LLMs, FP16 became the default precision for many training and inference workloads because it offers a good balance between numerical accuracy and computational efficiency. Compared to FP32 (32-bit float), FP16 halves the memory footprint and doubles the theoretical throughput on modern GPUs and accelerators that support native FP16 operations, such as NVIDIA's Tensor Cores. During training, mixed-precision techniques like AMP (automatic mixed precision) use FP16 for most calculations while keeping a master copy of weights in FP32 to preserve accuracy. For inference, many models are converted to FP16 to fit within GPU memory limits, especially for larger models like 7B or 13B parameter LLMs. However, FP16 has limited dynamic range compared to BF16 (bfloat16), which is becoming more popular for training as it maintains the same exponent range as FP32. FP16 is still widely used in older hardware and for inference on consumer GPUs. A key limitation is that FP16 can suffer from underflow or overflow when dealing with very small or very large gradients during training, which is why it is often paired with loss scaling. In the LLM ecosystem, FP16 is a common format for model weights distributed on Hugging Face and other model hubs. It is also the input format for many quantization methods, as converting from FP16 to lower-bit representations is straightforward. On the hardware side, FP16 operations are accelerated by Tensor Cores on NVIDIA GPUs (e.g., A100, H100) and by matrix engines on AMD and Intel accelerators. When benchmarking inference speed, measuring tokens-per-second in FP16 serves as a baseline against quantized formats.