Ollama
Ollama is a user-friendly tool for running and managing local LLMs, wrapping llama.cpp and providing simple CLI to pull, run, and customize models.
Ollama is a free and open-source application that simplifies running large language models locally on macOS, Linux, and Windows. It acts as a wrapper around llama.cpp (and also supports vLLM for some backends), providing a streamlined command-line interface and a local REST API that mimics OpenAI's API format. The primary value for AI builders is the ease of use: a single command, `ollama pull llama3.2`, downloads the model, sets up the inference engine with optimal parameters, and makes it ready to use. Ollama supports a growing library of pre-built models, including Llama 3.2, Mistral, Gemma 2, Phi-3, Qwen 2.5, and many fine-tuned variants. It also allows you to create custom models using Modelfiles, which let you specify prompt templates, temperature, system prompts, and quantization levels. Under the hood, Ollama uses GGUF format models and automatically selects the best GPU backend (CUDA, Metal, or Vulkan) based on your hardware. It also supports concurrent requests, batch processing, and custom stop tokens. For AI builders who want to prototype quickly or build desktop applications that call local LLMs, Ollama's API is straightforward: `POST /api/generate` with a JSON body. It also integrates with LangChain, LlamaIndex, and other frameworks. Ollama's key advantage is removing the friction of compiling llama.cpp, managing model files, and configuring GPU backends. However, it abstracts away many performance-tuning knobs, which might be a downside for advanced users. Ollama is not designed for high-throughput production serving; for that, vLLM or TGI are better choices.
Ollama is not a standalone inference engine; it relies on llama.cpp (or other backends) under the hood to perform actual model execution.
Related terms
Concepts you'll usually encounter alongside this one.
llama.cpp is an open-source C/C++ implementation of LLM inference optimized for CPU and GPU, known for running large models locally on modest hardware with quantized formats.
GGUF is a binary format for storing quantized LLM models, designed for efficient CPU and GPU inference, superseding GGML.
vLLM is an open-source high-performance LLM inference engine using PagedAttention to efficiently manage KV cache memory and achieve high throughput.
Mentioned in tutorials
Hands-on guides that put this concept to work.
Stand up a fully functional local LLM stack on Ubuntu 22.04, NVIDIA drivers, CUDA, Ollama, and your first model, in under 30 minutes.
Run local LLMs on Windows 11 the right way, WSL2, the WSL CUDA toolkit, and Ollama with full GPU acceleration. Native installer covered too.
Apple Silicon is shockingly good at LLM inference. Get Ollama running with unified memory acceleration on any M-series Mac in 15 minutes.
DeepSeek's reasoning model rivals GPT-4 on math and code. Run the distilled 7B/14B/32B variants on consumer hardware in 30 minutes.
See how this term affects your memory budget.
Real numbers, real chips, every workload.
Picks for builders, by use case.
Stay Ahead of the AI Curve
Get weekly AI hardware news, benchmark updates, and deals in your inbox. Founding-subscriber list, be one of the first.