Tokenizer

A tokenizer converts raw human language into numerical tokens that a transformer model can process. This includes two main operations: (1) segmentation-breaking text into subwords, characters, or bytes, and (2) encoding-mapping each segment to an integer ID from a fixed vocabulary. Subword tokenizers (e.g., Byte-Pair Encoding, WordPiece, SentencePiece-used by GPT, BERT, Llama) balance vocabulary size and token length by splitting rare words into smaller pieces (e.g., 'tokenizer' -> ['token', 'izer']). Vocabulary size typically ranges from 32k to 128k. Tokenization dramatically impacts model behavior: a given word may be represented as one token in one language or multiple in another, affecting memory usage and generation speed. For example, many LLMs use byte-level tokenization (like GPT-4's tiktoken) to handle any Unicode character. Pitfalls: leading spaces, punctuation, and digits can produce unexpected token counts. Always use the exact tokenizer matching your model-mismatches degrade performance. Token-to-word ratio varies by language (English ~1.3, Chinese ~2.5). Tokenizer choice affects context window utilization; a longer tokenization per word reduces effective context. For AI builders, profiling token counts on your dataset is essential before fine-tuning or RAG to avoid truncation. Tokenizers also influence inference cost since compute scales with token count both in attention (quadratic) and KV cache (linear).