CUDA

CUDA, which stands for Compute Unified Device Architecture, is a parallel computing platform and application programming interface (API) model created by NVIDIA. It allows software developers to use a CUDA-enabled graphics processing unit (GPU) for general purpose processing - an approach known as GPGPU (General-Purpose computing on GPUs). For AI builders working with LLMs, CUDA is the dominant runtime for running deep learning frameworks like PyTorch, TensorFlow, and JAX. Most LLM inference engines, including vLLM and llama.cpp, leverage CUDA kernels to accelerate matrix multiplications, attention mechanisms, and other tensor operations. CUDA enables fine-grained parallelism by exposing thousands of GPU cores to perform SIMT (Single Instruction, Multiple Thread) operations. For LLMs, this is critical because operations like batched matrix multiplies in transformer layers can be decomposed into many parallel thread blocks. CUDA also provides libraries such as cuBLAS for optimized linear algebra, cuDNN for deep neural network primitives, and NCCL for multi-GPU communication. The CUDA toolkit includes a compiler (nvcc), debugger, and profiler (Nsight). AI builders must ensure their GPU compute capability (e.g., 8.0 for Ampere, 9.0 for Hopper) is compatible with the CUDA version they use. CUDA is proprietary but has become the de facto standard for GPU computing in AI, with almost all open-source LLM tools targeting it first. It supports features like tensor cores for mixed-precision training and FP8 quantization on recent hardware. While alternatives like ROCm exist, CUDA's ecosystem maturity and performance optimization make it the default choice for production AI workloads.