Build Local AI Agents With Ollama, LangChain, and Custom Tools

What 'agent' means here

An agent is an LLM that can take actions, not just produce text. Concretely: the model produces a structured request like 'call search_web("latest AI hardware news")' instead of producing the answer directly. Your code intercepts that, runs the tool, feeds the result back to the model, and the model uses it to produce the final answer. Loop until the model says it's done.

The magic isn't the LLM, it's the loop. Local LLMs got 'good enough' at tool calling around the Llama 3.1 / Qwen 2.5 generation. We'll use Qwen 2.5 14B specifically because it has the best tool-calling reliability in its weight class. The framework is LangChain because it's the most documented; LangGraph or smolagents work just as well.

Step 1: Set up the Python environment

Use uv (the fast Python package manager) or pip. We need langchain-ollama (the Ollama integration), the LangChain core, and a couple of tool libraries: duckduckgo-search for web, and python's built-in shutil for files. Keep it minimal, every dep is a potential break point in a year.

# Using uv (fast)
uv venv
source .venv/bin/activate
uv pip install langchain langchain-ollama langchain-community duckduckgo-search

# Or pip
python -m venv venv && source venv/bin/activate
pip install langchain langchain-ollama langchain-community duckduckgo-search

Step 2: Pull a tool-calling-capable model

Not every Ollama model handles tool calls well. As of writing the reliable choices are: qwen2.5:14b (best), qwen2.5:32b (better but slower), llama3.1:8b (good), llama3.3:70b (excellent if you have the VRAM). Smaller models hallucinate tool calls or get the JSON schema wrong. If your hardware can't run 14B, go to llama3.1:8b, anything smaller is not worth the time.

ollama pull qwen2.5:14b

# Quick tool-call smoke test
ollama run qwen2.5:14b
>>> Pretend you have a tool 'get_weather(city)'. The user asks: "What's the weather in Tokyo?". Respond in the form {"tool": "get_weather", "args": {"city": "Tokyo"}}.

Step 3: A minimal agent in 30 lines

Here's a complete working agent. It can search the web and read local files. The LLM is told what tools exist via the bind_tools call, and LangChain handles the JSON-schema dance behind the scenes.

from langchain_ollama import ChatOllama
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_core.tools import tool
from langchain_core.messages import HumanMessage

llm = ChatOllama(model="qwen2.5:14b", temperature=0.1)

@tool
def read_file(path: str) -> str:
    """Read a text file from disk and return its contents."""
    with open(path, encoding="utf-8") as f:
        return f.read()[:4000]

search = DuckDuckGoSearchRun()
tools = [search, read_file]
agent = llm.bind_tools(tools)

# Simple loop
messages = [HumanMessage(content="What's the cheapest GPU under $500 with 16+ GB VRAM in 2026?")]

for _ in range(5):
    response = agent.invoke(messages)
    messages.append(response)
    if not response.tool_calls:
        print(response.content)
        break
    for call in response.tool_calls:
        result = [t for t in tools if t.name == call["name"]][0].invoke(call["args"])
        messages.append({"role": "tool", "content": str(result), "tool_call_id": call["id"]})

Step 4: Add a real custom tool

Web search is fun, but the killer feature of local agents is tools that touch your private data, your filesystem, your home automation, your local databases. Here's a tool that lets the agent query your home's Prometheus metrics. The pattern generalizes to any HTTP/SQL/CLI target you control.

import requests

@tool
def query_metric(metric_name: str) -> str:
    """Query a Prometheus metric by exact name. Returns latest value.
    
    Available metrics include: node_cpu_seconds_total, node_memory_MemFree_bytes,
    nvidia_gpu_temperature_celsius, ollama_request_duration_seconds.
    """
    r = requests.get(
        "http://prometheus.local:9090/api/v1/query",
        params={"query": metric_name},
        timeout=5,
    )
    return str(r.json()["data"]["result"][:3])

# Now ask:
#   "What's the GPU temperature on the AI server right now?"
# The agent will figure out which Prometheus metric to query.

Step 5: Memory across turns

The above agent has no memory between invocations. For a real assistant you want it to remember the conversation. The cleanest pattern is to keep the full message history in a list and pass it on every invocation, Ollama handles the context window. For longer-term memory across sessions, persist to SQLite. LangGraph adds proper checkpointing if you want to get fancy, but a 10-line SQLite store is usually enough.

import json, sqlite3

db = sqlite3.connect("agent_memory.db")
db.execute("CREATE TABLE IF NOT EXISTS msgs (session TEXT, role TEXT, content TEXT, ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP)")

def remember(session: str, role: str, content: str):
    db.execute("INSERT INTO msgs VALUES (?, ?, ?, datetime('now'))", (session, role, content))
    db.commit()

def recall(session: str, limit: int = 20):
    cur = db.execute("SELECT role, content FROM msgs WHERE session = ? ORDER BY ts DESC LIMIT ?", (session, limit))
    return list(reversed(cur.fetchall()))

Step 6: Guardrails for local agents

A local agent with filesystem and shell access is dangerous, a hallucinated rm -rf is forever. Two principles. (1) Whitelist, don't blacklist: tools should accept paths under /home/user/agent-sandbox, not 'any path that isn't /etc'. (2) Confirm destructive actions: any tool that writes or deletes should print what it's about to do and require explicit confirmation. For agents running 24/7 unattended, never give them write access at all, read-only tools plus a 'queue a Pull Request' tool is the safe pattern.

Step 7: When to graduate to a framework

The above is 50 lines of Python and gets you 80% of the way. Once you want streaming output, parallel tool calls, reliable retry logic, or a UI, look at: LangGraph (LangChain's stateful graph framework, great for complex flows), smolagents (Hugging Face's lightweight alternative), or AutoGen (Microsoft's multi-agent framework, overkill for most cases). For most home use, the bare LangChain + Ollama loop above is the right level of complexity.