AI Homelab: The 2026 Build & Setup Guide
A homelab is a private cloud you run in your house. An AI homelab adds always-on local intelligence, chat, coding agents, RAG, automation , under your control, behind your firewall, on hardware you own. This is the full 2026 build, network, software, and operations guide. Three tiers, opinionated picks, real wiring diagrams.
Why an AI homelab?
The cloud is convenient until it isn't. API rate limits, model deprecation schedules, opaque content moderation, your data leaving your network, the monthly bill, every reason that homelabs took off for media servers, backups, and home automation now applies to AI. A modest homelab in 2026 can host the equivalent of ChatGPT for a household of four, plus a coding agent on call, plus RAG-over-your-documents, on hardware that pays for itself in 18 months of saved API spend.
The trade is operational. You become the SRE. You patch containers, you monitor GPUs, you handle UPS battery replacements every three years, you read changelogs. For some people that is a tax, for others a hobby, both are valid. This guide assumes you are in the hobby camp.
Hardware awards
Single-Node Homelab
One used RTX 3090, mini-ITX 7700 build, 64 GB RAM, 4 TB NVMe. Docker stack: Ollama + Open WebUI + Caddy + Tailscale. Plenty for a household.
Two-Node Homelab + NAS
Inference node (RTX 4090) + general-compute Proxmox host + 48 TB TrueNAS Scale. Tagged VLANs, separate IoT subnet, full backups.
Rack Homelab
12U StarTech rack, 25 GbE switch, two Proxmox nodes, one inference node with RTX 5090, 96 TB NAS, UPS-backed. Goal: a small office in your basement.
Apple-First Homelab
Mac Mini M4 Pro for AI + Synology DS1825+ for storage + UniFi Dream Router. Quiet, low-power, easy. The 'family-friendly' homelab.
Three homelab tiers
Pick the tier that matches both your budget and your patience for cable management. You can grow from Tier 1 to Tier 2 incrementally; jumping straight to Tier 3 is a weekend disappeared.
The first homelab
$1,500–$2,200
A single mini-tower box with a 12 GB GPU running Docker Compose. Hosts Ollama + Open WebUI, accessible via Tailscale from your phone. The first reasonable homelab for an individual.
Compact base
Cheapest 12 GB CUDA
Models + datasets
Headroom
Mini-ITX case
Brown-out protection
Two-node + NAS
$5,500–$7,500
Real separation of duties. An inference node (RTX 4090), a Proxmox host for VMs and home automation, a dedicated TrueNAS Scale NAS for datasets and backups. Tagged VLANs, dedicated IoT SSID. The classic family-grade homelab.
Main AI box
VMs, agents
24 TB raw, 16 TB usable RAID-Z2
VLANs, IDS, gateway
10 GbE between AI ↔ NAS
5-minute graceful shutdown
The rack homelab
$14,000–$18,000
12U StarTech rack, dual Proxmox hosts for HA, RTX 5090 inference node, 100 TB NAS, 25 GbE backbone, IPMI-managed. The 'small office in the basement' tier where homelab becomes lifestyle.
Single-GPU monster
Quorum cluster
96 TB raw
100 GbE backbone
Network core
Real rack
Online double-conversion
The AI homelab software stack
Eight battle-tested pieces of software. Each is open-source and runs in a container; the whole thing is one docker compose up -d away.
Ollama
Model serving daemon with OpenAI-compatible API. The base layer of every modern homelab AI stack.
Open WebUI
Chat UI with multi-user accounts, RAG against your docs, web search, and model switching.
LiteLLM
Multi-model gateway. Auto-fallback to a paid API when local can't handle a request. Cost tracking.
Tailscale
Mesh VPN. Access your homelab from anywhere without opening ports on your router.
Caddy / NPM
Reverse proxy with automatic HTTPS via Let's Encrypt. Front of all your services.
Grafana + Prometheus
Dashboards for GPU temp, VRAM, request throughput. nvidia-smi exporter for AI-specific metrics.
n8n / Continue.dev
Agentic workflows + IDE integration. Wire your local LLM into automations and your editor.
Watchtower / Renovate
Automated container updates on a schedule. Critical for a homelab you actually want to maintain.
GPU passthrough on Proxmox
The half-day setup that unlocks one host running everything: AI VMs get the GPU, the network VM gets the Mellanox NIC, the media VM gets the iGPU for transcoding.
1. Enable IOMMU
# BIOS: enable VT-d (Intel) or AMD-Vi
# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet \
intel_iommu=on iommu=pt"
sudo update-grub && sudo reboot2. Bind to vfio-pci
# Find PCI IDs
lspci -nn | grep -i nvidia
# 01:00.0 VGA: NVIDIA RTX 4090 [10de:2684]
# /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2684,10de:22ba
# RTX 4090 has both GPU + audio function3. Blacklist nvidia on host
# /etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
sudo update-initramfs -u && sudo reboot4. Attach to VM
# Proxmox UI: VM → Hardware → Add → PCI Device
# Pick the GPU. PCI Express ON, ROM-Bar ON.
# Or via CLI:
qm set 100 -hostpci0 01:00,pcie=1,x-vga=1
# Restart VM. nvidia-smi inside VM.Networking, power, cooling
Networking
UniFi for the gateway and AP layer, boring, stable, web UI. MikroTik for the cheap-but-real 10/25/100 GbE switching. Tagged VLANs: 10 mgmt / 20 trusted / 30 IoT / 99 storage. A separate SSID per VLAN keeps Apple TVs and lightbulbs off your AI stack.
Storage
A dedicated TrueNAS Scale box for datasets, model archives, RAG corpora, and Docker volume backups. RAID-Z2 with 4–8 drives. Sync snapshots to a cheap cloud bucket nightly. Models on local NVMe; cold archives on the NAS.
Power & UPS
UPS sized for graceful shutdown, not battery-backed runtime, a 1500 VA APC SMT holds a homelab for 5–15 minutes, long enough for a clean shutdown via NUT. Track kWh via a Kill-a-Watt or smart plug; an always-on Tier 2 lab costs roughly $15–$30/month at US rates.
Cooling
The room is part of the thermal solution. Inference nodes dump 300–600 W of heat; a small bedroom warms 3 °C in 30 minutes. Plan extraction, an inline duct fan vented out a window, or a dedicated mini-split for the lab room. Don't forget the noise side: a 60 dBA rack downstairs is not livable upstairs.
Frequently asked questions
What is an AI homelab?
A self-hosted environment combining compute (CPU + GPU), storage (NAS), networking (managed switch, VLANs, VPN), and software (Ollama, Open WebUI, Tailscale, monitoring), designed for privacy-respecting, always-on AI services for your household.
How much does an AI homelab cost?
Single-node entry-level: ~$1,500–$2,000. Two-node + NAS: ~$5,000–$7,000. Rack-mounted with redundancy: ~$12,000+. Power and noise costs are real and ongoing, budget $15–$50/month in electricity per kW of always-on draw.
Proxmox or bare metal for AI homelab?
Proxmox if you want VLAN-isolated VMs for AI, networking, media, and home automation on the same hardware. Bare metal Ubuntu Server if you only run AI workloads, GPU passthrough overhead is real (5–8%) and bare metal is one less moving part.
Do I need a NAS for an AI homelab?
Strongly recommended. Datasets, model weights, RAG corpora, embeddings indexes, and Docker volumes all want a separate storage tier. A 4-bay Synology DS923+ or a TrueNAS Scale mini-PC with 4× 8 TB drives works for years.
How do I do GPU passthrough on Proxmox?
Enable IOMMU in BIOS (Intel: VT-d, AMD: AMD-Vi), pass the PCIe IDs to vfio-pci in /etc/modprobe.d/vfio.conf, blacklist the host nvidia driver, then attach the device to the VM as a PCI device. Plan on a half-day of setup the first time.
Best LLM stack for an AI homelab?
Ollama (model serving) + Open WebUI (chat UI) + LiteLLM (gateway / multi-model fallback) + Caddy or Nginx Proxy Manager (TLS) + Tailscale (remote access). Optional: SearXNG for web search, n8n for agentic flows, Continue.dev for IDE integration.
How much power does an AI homelab draw?
A single 24 GB GPU + dev workstation idles around 80–120 W and peaks at 450 W. Two-node + NAS commonly runs 200 W idle, 800 W peak. Plan UPS sizing around the peak.
Related guides
Local AI Server Builds, Dev Box to Rack 2026
Four production-grade local AI server builds. Single-GPU dev box, dual-3090 home server, EPYC + 4× GPU workstation, rackmount inference cluster. Power, networking, IPMI, headless Linux.
Read guide WorkstationBest AI Workstation Builds in 2026 (Every Budget)
Four hand-tuned AI workstation builds from $1,500 to $20,000+. Component-level recommendations for training, fine-tuning, and large-model inference. Cooling, PSU, motherboard, multi-GPU.
Read guide GPUBest GPU for Local LLMs (2026)
Hands-on tested ranking of the best GPUs for running Llama 3, DeepSeek, Mixtral, and other large language models on your own hardware. VRAM, tokens/sec, $/perf, updated May 2026.
Read guide SoftwareOllama Hardware Requirements, 2026 Guide
Minimum and recommended hardware for every Ollama model size from 3B to 70B+. RAM, VRAM, CPU, storage. NVIDIA vs Apple Silicon vs AMD ROCm vs Intel. Real measurements, not specs.
Read guideStay Ahead of the AI Curve
Get weekly AI hardware news, benchmark updates, and deals in your inbox. Founding-subscriber list, be one of the first.
Affiliate disclosure: As an Amazon Associate, MyAIHardware.com earns from qualifying purchases. The homelab builds in this guide are derived from machines we actually run.