Install Ollama on Ubuntu 22.04 with NVIDIA GPU

Why Ubuntu 22.04 for local AI

Ubuntu 22.04 LTS is the path of least resistance for running large language models on NVIDIA hardware. Almost every PyTorch wheel, every CUDA build, every llama.cpp binary, and every Hugging Face inference example is tested first against Ubuntu LTS. If you fight the OS, you spend your weekend reading dmesg instead of running models.

We'll keep the install boring and reproducible. By the end you will have NVIDIA drivers, the proprietary CUDA runtime that Ollama actually needs, the Ollama daemon running as a systemd service, and a 7B-class model answering questions on the command line. From there, plugging in Open WebUI or wiring it into LangChain is a five-minute job.

Step 1: Update the base system

Before installing anything else, get Ubuntu fully patched. This avoids a class of weird DKMS failures where a half-patched kernel module refuses to rebuild against a freshly-installed driver. Reboot if a new kernel comes down, do not skip that reboot, even though it looks optional.

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential dkms curl wget git
sudo reboot

Step 2: Install the NVIDIA proprietary driver

Ubuntu's ubuntu-drivers tool will pick the recommended driver for your card. For the RTX 30/40/50 series this will be 550 or newer. Do not install the open-source nvidia-open variant unless you know your card supports it, many Turing/Ampere cards don't. After installation you must reboot so the kernel loads the new module.

sudo ubuntu-drivers devices                  # show recommended driver
sudo ubuntu-drivers autoinstall              # install it
sudo reboot

# After reboot, confirm:
nvidia-smi

Step 3: Install Ollama with the official one-liner

Ollama ships an install script that drops a static binary into /usr/local/bin, registers a systemd service called ollama.service, and creates an ollama system user. It will autodetect your NVIDIA driver and pull the matching CUDA runtime. There is nothing to configure, if your GPU shows up in nvidia-smi, Ollama will use it.

curl -fsSL https://ollama.com/install.sh | sh

# Confirm service is up
systemctl status ollama

# Confirm CUDA is detected
ollama --version
journalctl -u ollama --no-pager | grep -i 'cuda\|gpu'

Step 4: Pull and run your first model

Start with llama3.1:8b, it's the smallest model that still feels like a real assistant, runs comfortably on 8 GB of VRAM with Q4 quantization, and downloads in a few minutes on residential broadband. The first ollama run command pulls the model, loads it into VRAM, and drops you into an interactive REPL.

If this works, your install is complete. Everything else in this site, agents, RAG, Open WebUI, fine-tuning, assumes you have a working ollama serve and at least one model pulled.

ollama pull llama3.1:8b
ollama run llama3.1:8b

# In the REPL:
>>> Write a haiku about a GPU running too hot.
>>> /bye   # exit

Step 5: Enable remote access (optional)

By default Ollama binds to 127.0.0.1:11434, local-only. If you want to call it from another machine on your LAN (say, a desktop hitting a beefier server), edit the systemd unit and set OLLAMA_HOST=0.0.0.0:11434. This is fine on a trusted home network. Do not expose port 11434 to the public internet without authentication in front of it.

sudo systemctl edit ollama.service

# In the editor paste:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_ORIGINS=*"

# Save, then reload and restart:
sudo systemctl daemon-reload
sudo systemctl restart ollama

# Verify from another machine:
curl http://<server-ip>:11434/api/tags

Step 6: Verify everything is using your GPU

The number-one source of confusion for new local-AI users is silent CPU fallback. The model still works, it's just twenty times slower. Watch nvidia-smi in a second terminal while you send a prompt. You should see your GPU's power draw spike to 90%+ of its limit and VRAM usage jump by the model's size.

If GPU utilization stays at 0%, the most common cause is that the model is larger than your VRAM and Ollama is offloading layers to CPU. Pull a smaller model or a more aggressive quantization to confirm the GPU path works.

# Terminal 1
watch -n 0.5 nvidia-smi

# Terminal 2
ollama run llama3.1:8b "Explain transformer attention in 5 sentences."

Where to go next

You have a working local LLM. The natural next steps are: (1) install Open WebUI so you have a ChatGPT-like web interface, (2) try a reasoning model like DeepSeek-R1 to see what the new generation can do, and (3) wire the Ollama API into your own scripts. All three are linked at the bottom of this tutorial.