FP8

FP8 introduces 8-bit floating-point representations specifically tailored for deep learning, with two standard variants: E4M3 (4 exponent bits, 3 mantissa bits) for higher precision during training, and E5M2 (5 exponent bits, 2 mantissa bits) for wider dynamic range during inference. FP8 is natively supported on NVIDIA Hopper (H100, H200) and Blackwell GPUs, as well as some AMD Instinct accelerators. It offers a dramatic improvement in memory efficiency and compute throughput over FP16, reducing memory bandwidth requirements by half and doubling tensor core operations for matrix multiplications. In practice, FP8 is used primarily for inference of large LLMs, where the reduced precision can maintain model quality with minimal degradation, especially when combined with scaling factors per tensor or per row. Techniques like FP8 quantization involve converting FP16 weights and activations to FP8 using calibration data and scaling factors to minimize quantization error. The main challenge with FP8 is its limited dynamic range, which can cause information loss if not managed carefully. E4M3 is preferred for forward and backward passes during training because it provides better accuracy for small values, while E5M2 is more reliable for inference when activations have high variance. Hardware support for FP8 varies; on H100, FP8 matrix operations are 2x faster than FP16. However, adopting FP8 requires software stack support, including torchao, vLLM, and llama.cpp, which are increasingly adding FP8 kernels. For LLM inference, FP8 can achieve throughput close to INT8 quantized models while maintaining higher accuracy, making it a sweet spot for latency-sensitive applications. FP8 is not yet as widely deployed as INT8 or INT4 due to hardware and software maturity, but it is becoming a standard for next-generation AI accelerators.