Before you start
- ✓Ollama installed and reachable on http://localhost:11434
- ✓At least one model pulled (e.g., llama3.1:8b)
- ✓Comfort with HTTP and JSON
Required hardware
- Amazon
Any modern GPU running Ollama
Two APIs in one server
Ollama exposes two HTTP APIs simultaneously on port 11434. The native API at /api/generate, /api/chat, /api/embeddings is Ollama-specific and is the cleanest for streaming. The OpenAI-compatible API at /v1/chat/completions, /v1/embeddings drops in as a replacement for OpenAI's API, meaning any library or app that talks to OpenAI can be pointed at Ollama with one config change. The OpenAI-compatible endpoint is the right choice when integrating with existing tools (LangChain, LlamaIndex, the openai SDK); the native API is the right choice when writing new code from scratch.
Step 1: Plain curl
The 30-second smoke test. POST to /api/generate with a model and prompt, get streamed JSON lines back. This is the simplest possible Ollama integration and proves the daemon is reachable.
curl http://localhost:11434/api/generate -d '{
"model": "llama3.1:8b",
"prompt": "Why is the sky blue? Answer in one sentence.",
"stream": false
}'
# Streaming:
curl -N http://localhost:11434/api/generate -d '{
"model": "llama3.1:8b",
"prompt": "Count from 1 to 10 in words.",
"stream": true
}'Step 2: Python with the native API
Use the official `ollama` Python package. It wraps the HTTP API in a clean sync/async client and handles streaming. Install with pip. For non-streaming responses use .chat() or .generate(); for streamed token output use the stream=True kwarg and iterate.
# pip install ollama
import ollama
# One-shot
resp = ollama.chat(model="llama3.1:8b", messages=[
{"role": "user", "content": "In one sentence, what is a transformer?"}
])
print(resp.message.content)
# Streaming
for chunk in ollama.chat(
model="llama3.1:8b",
messages=[{"role": "user", "content": "Tell me a joke."}],
stream=True,
):
print(chunk.message.content, end="", flush=True)
print()Step 3: Python with the OpenAI-compatible endpoint
If you already use the openai SDK in your code, point it at Ollama by overriding base_url. API key is required by the SDK but ignored by Ollama, use any non-empty string. This is the cleanest path for porting existing OpenAI code to local.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1",
api_key="ollama", # ignored, but required by the SDK
)
resp = client.chat.completions.create(
model="llama3.1:8b",
messages=[{"role": "user", "content": "Explain RAG in 3 bullet points."}],
stream=False,
)
print(resp.choices[0].message.content)Step 4: JavaScript / TypeScript
Both the native API and the OpenAI-compatible API work in browsers and Node. There's an official ollama-js package. For browser code, remember Ollama must have OLLAMA_ORIGINS=* set (or the domain serving your site) for CORS to allow the request, otherwise you'll see CORS errors in DevTools.
// npm install ollama
import ollama from "ollama";
const response = await ollama.chat({
model: "llama3.1:8b",
messages: [{ role: "user", content: "List three uses for a local LLM." }],
});
console.log(response.message.content);
// Streaming
const stream = await ollama.chat({
model: "llama3.1:8b",
messages: [{ role: "user", content: "Tell me a story." }],
stream: true,
});
for await (const part of stream) {
process.stdout.write(part.message.content);
}Step 5: Structured output (JSON mode)
For programmatic consumers you want JSON, not prose. Ollama's `format` parameter lets you request strict JSON output. As of Ollama 0.5+, you can pass a JSON Schema and the model is constrained to produce conforming JSON, which is enormously useful for extracting structured data.
import ollama, json
schema = {
"type": "object",
"properties": {
"product": {"type": "string"},
"price_usd": {"type": "number"},
"in_stock": {"type": "boolean"},
},
"required": ["product", "price_usd", "in_stock"],
}
resp = ollama.chat(
model="llama3.1:8b",
messages=[
{"role": "user", "content": "Extract: 'I bought a NVIDIA RTX 4090 for $1599 from Newegg, currently out of stock.'"}
],
format=schema,
)
data = json.loads(resp.message.content)
print(data) # {'product': 'NVIDIA RTX 4090', 'price_usd': 1599, 'in_stock': False}JSON Schema enforcement makes Ollama work brilliantly for extraction tasks, the model literally cannot produce malformed JSON anymore.
Step 6: Embeddings for RAG
For retrieval-augmented generation you need embeddings, vector representations of text. Ollama exposes /api/embeddings. Pull a small embedding-specific model (nomic-embed-text is the default), then call it like any other Ollama endpoint. Store the resulting vectors in your vector DB of choice, Qdrant, Chroma, pgvector all work fine with Ollama-generated embeddings.
import ollama
ollama.pull("nomic-embed-text") # one-time
vec = ollama.embeddings(
model="nomic-embed-text",
prompt="The NVIDIA RTX 4090 has 24 GB of VRAM.",
).embedding
print(len(vec)) # 768 dimensions
# Bulk-embed N docs in parallel using asyncio for throughput.Step 7: Keep-alive and concurrency
Two performance gotchas. (1) By default Ollama unloads a model from VRAM 5 minutes after its last request. If your app uses the model intermittently you'll pay the 10-second reload tax on every cold start. Set keep_alive in the request (or OLLAMA_KEEP_ALIVE env var) to keep the model resident. (2) Ollama processes requests serially per model by default. To handle concurrent users, set OLLAMA_NUM_PARALLEL to the number of simultaneous requests you want, but each parallel slot reserves its own KV cache, so don't go higher than your VRAM allows.
# Keep the model warm forever
ollama.chat(model="llama3.1:8b", messages=[...], keep_alive="24h")
# Or set the daemon-wide default
sudo systemctl edit ollama.service
# Environment="OLLAMA_KEEP_ALIVE=24h"
# Environment="OLLAMA_NUM_PARALLEL=4"Step 8: Error handling that actually works
Local LLMs fail in ways cloud LLMs don't. The model isn't pulled (404). The GPU OOMed mid-request (500). The daemon crashed (connection refused). Wrap your calls with retries, but with exponential backoff capped at 30s, because if it's truly down you want to fail fast not spin forever. Log the response status, a request that returns 200 but with `{"error": "model not found"}` is the most common silent failure mode.
Tags
Stuck? Share your build?
Hundreds of homelabbers are working through these same tutorials in our community. Drop your config, ask the hard question, or show off what you built.
Join the discussion