AI Hardware & LLM Glossary

Attention is a mechanism that computes weighted sums of input tokens, allowing the model to focus on the most relevant parts of the sequence when generating each output.

AWQ (Activation-aware Weight Quantization) is a quantization method that uses activation statistics to identify and protect important weights, achieving superior accuracy at low bitrates.

The number of independent inputs processed simultaneously during training or inference, affecting computational efficiency and memory usage.

The context window defines the maximum number of input tokens a transformer model can process at once, determining the length of conversation history or document it can reason over.

CUDA is NVIDIA's parallel computing platform and API that allows developers to use GPU accelerators for general-purpose processing, including LLM training and inference.

An embedding is a dense vector representation of text that captures semantic meaning, enabling similarity search, clustering, and retrieval for tasks like RAG and semantic search.

ExLlama v2 is a GPU-optimized inference engine and quantization format for Llama-family models, enabling high-throughput generation with low-bit weights and dynamic KV-cache management.

FlashAttention is an I/O-aware algorithm that computes exact attention without materializing the full attention matrix, reducing memory usage from O(n^2) to O(n) and improving speed 2-10x on GPUs.

FP16 is a 16-bit floating-point format used in AI inference and training to reduce memory use and accelerate computation while maintaining acceptable precision.

FP8 is an 8-bit floating-point format with two variants (E4M3 and E5M2) designed for efficient AI inference and training on modern hardware like NVIDIA H100.

GGUF is a binary format for storing quantized LLM models, designed for efficient CPU and GPU inference, superseding GGML.

GPTQ is a post-training quantization method that compresses LLM weights to 2-4 bits using an optimal brain quantization approach.

A high-performance DRAM technology stacked vertically with GPU dies, providing massive memory bandwidth for LLM inference and training.

INT4 is a 4-bit integer quantization format that compresses model weights by 8x vs FP32, enabling large LLMs to run on consumer hardware with modest quality loss.

INT8 is an 8-bit integer quantization format that significantly reduces model memory and speeds up inference by mapping floating-point values to 256 discrete levels.

The KV cache stores intermediate key and value tensors from previous tokens during autoregressive generation, enabling subquadratic computation by avoiding redundant re-encoding of the full context.

KV cache quantization compresses the key-value cache memory used in autoregressive LLM inference, reducing memory usage and enabling longer context windows.

llama.cpp is an open-source C/C++ implementation of LLM inference optimized for CPU and GPU, known for running large models locally on modest hardware with quantized formats.

LoRA is a parameter-efficient fine-tuning method that freezes original weights and injects trainable low-rank matrices, reducing memory and storage needs while achieving full-weight quality on specific tasks.

Metal is Apple's low-level GPU programming API and framework used to accelerate AI workloads on macOS, iOS, and iPadOS devices.

Mixture of Experts is a sparse architecture that divides the network into 'expert' subnetworks, activating only a subset per token to scale model capacity without proportional compute cost.

MLX is Apple's machine learning framework optimized for Apple Silicon (M-series chips), enabling efficient LLM inference and training on Mac hardware.

NVIDIA's high-speed interconnect technology that enables GPU-to-GPU data transfers with up to 900 GB/s bandwidth for scaling large AI models.

Ollama is a user-friendly tool for running and managing local LLMs, wrapping llama.cpp and providing simple CLI to pull, run, and customize models.

The number of parallel data channels on a PCI Express bus connecting GPUs and CPUs, determining bandwidth for inter-device communication.

Pipeline parallelism splits LLM layers across multiple GPUs, with each GPU handling a subset of consecutive layers, enabling larger models and better memory balancing.

Q4_K_M is a 4-bit quantization method in llama.cpp that combines 4-bit quantization for most weights with higher precision for important weights using a block size of 32.

Q5_K_M is a 5-bit mixed-precision quantization method in llama.cpp that offers higher accuracy than 4-bit methods with moderate memory increase.

Q8_0 is an 8-bit integer quantization format in llama.cpp that balances high accuracy with good compression, offering near-lossless quality for many LLMs.

QLoRA combines 4-bit model quantization with LoRA adapters to fine-tune massive LLMs on a single consumer GPU by compressing the base model while preserving adapter precision.

RAG augments LLM prompts with relevant documents retrieved from an external knowledge base, grounding responses in factual data to reduce hallucination and enable dynamic knowledge.

ROCm is AMD's open-source GPU computing platform designed as a CUDA alternative for running AI workloads on AMD GPUs.

An inference technique that uses a smaller draft model to guess multiple future tokens, verified in parallel by the large model to reduce latency.

Tensor parallelism splits model weight tensors across multiple GPUs to accelerate matrix operations, reducing memory per GPU and improving computation speed.

A key tradeoff in LLM serving: throughput measures total requests processed per time, while latency measures response time per individual request.

A preprocessing component that splits text into smaller units (tokens) and maps them to integer IDs for model input, and reverses the process for output.

The rate at which a language model generates or processes tokens, commonly used to measure inference throughput.

A neural network architecture using self-attention mechanisms to process sequential data in parallel, forming the foundation of modern LLMs.

A specialized database that stores and indexes high-dimensional vector embeddings for efficient similarity search in AI and LLM retrieval-augmented generation (RAG) pipelines.

vLLM is an open-source high-performance LLM inference engine using PagedAttention to efficiently manage KV cache memory and achieve high throughput.