Server BuildsUpdated 2026-05-104 tiers · $1.8k–$32k

Local AI Server Builds: From Dev Box to Rackmount

Building a server is not the same as building a workstation. A server runs headless, lives in a closet or rack, must be remotely manageable, and is expected to be up 24/7. These four tiers cover the realistic spectrum, from a closet dev box to a basement rack , with hardware, networking, and OS choices opinionated for production-style operation.

Daniel Park · Workstation Reviewer Updated 2026-05-10 24 min read Built in our Brooklyn rack

Awards at a glance

best budget

Single-GPU Dev Box ($1,800)

Mini-tower with Ryzen 7700, 64 GB DDR5, RTX 4070 Super 12 GB. Headless Ubuntu Server LTS, Tailscale for remote access. The right first AI server.

$1,800
best value

Dual-3090 Home Server ($4,200)

X570 board, Ryzen 9 5950X, 128 GB ECC RAM, 2× RTX 3090 with NVLink, 8 TB Gen 4 NVMe + 24 TB rust NAS. Hosts a household for years.

$4,200
best overall

EPYC + 4× GPU Workstation ($12,500)

Tyan/Asus EPYC 9354 platform, 256 GB ECC, 4× RTX 4090 24 GB, dual 25 GbE. A real inference cluster in one chassis.

$12,500
premium

Rackmount Inference Cluster ($30k+)

2U Supermicro 4124GS, dual EPYC 9554, 1 TB ECC, 4× RTX 6000 Ada or 4× L40S, redundant 2000 W PSU, IPMI. Production hosting at home.

$32,000
TIER 1

Single-GPU Dev Box

$1,800 · the AI server you bring home Friday

The first AI server most people should build. Plenty for one developer plus a household, Ollama serving 8B–13B chat, Open WebUI front-end, a coding agent in the background. Sits in a closet and is forgotten.

Power~150 W idle / 420 W peak
Noise30 dBA idle / 40 dBA load
Form factorMini-tower, headless Ubuntu Server LTS 24.04
ServesOllama · Open WebUI · Tailscale · 1–2 concurrent users
$1,800total

Wins

  • Quiet enough for a bedroom closet
  • Easy to maintain
  • Cheap to power

Drawbacks

  • 12 GB caps you at 13B models
  • No remote BMC
  • Single point of failure

CPU

AMD Ryzen 7 7700 (8C/16T)

65 W, plenty for 1-GPU server.

$279

GPU

NVIDIA RTX 4070 Super 12 GB

Single-slot height makes future upgrades easy.

$599Amazon

RAM

64 GB DDR5-5600 ECC (2×32)

ECC even on consumer AM5, free reliability win.

$229

Storage

Samsung 990 Pro 4 TB NVMe + 8 TB Seagate IronWolf

NVMe for hot models, rust for archives.

$369Amazon

Motherboard

Gigabyte B650M Aorus Elite AX

M-ATX, IPMI not required at this tier.

$149

PSU

Corsair RM850e (ATX 3.1)

Room for a future 2nd GPU.

$139

Case

Fractal Design Define 7 Compact

Sound-dampened panels, fanless capable.

$119
TIER 2

Dual-3090 Home Server

$4,200 · the family-AI workhorse

Two used RTX 3090s with NVLink, X570 platform, 128 GB ECC RAM. Runs Llama 3 70B Q4 across both GPUs at 22 tok/s. Hosts a household of 4 with multiple agents in flight. The sweet-spot home AI server in 2026.

Power~250 W idle / 1100 W peak
Noise35 dBA idle / 50 dBA load
Form factorMid-tower, headless Ubuntu LTS, Docker Compose stack
ServesOllama 70B · vLLM small models · 4–8 concurrent users
$4,200total

Wins

  • Real 70B host
  • Plenty of bulk storage
  • Reasonable to maintain

Drawbacks

  • Loud under sustained load
  • 350 W idle is not trivial
  • Used GPU risk

CPU

AMD Ryzen 9 5950X (16C/32T)

Used market, incredible value, X570 board reuse.

$449

GPU ×2

2× NVIDIA RTX 3090 24 GB + NVLink bridge

Used RTX 3090 ~$800 ea + $200 NVLink bridge.

$1,899

RAM

128 GB DDR4-3600 ECC RDIMM (4×32)

ECC required for 24/7 uptime.

$549

Storage

Crucial T705 2 TB + 4× 8 TB WD Red Plus RAID-Z2

Fast NVMe + 24 TB usable bulk storage.

$749

Motherboard

ASUS ProArt X570-Creator WiFi

Two PCIe 4.0 x16 slots @ x8/x8 with PLX.

$369

PSU

Seasonic PRIME PX-1300 ATX 3.1

1300 W headroom for dual 3090.

$269

Case

Fractal Define 7 XL

Full tower, hot-swap drive bays.

$209

Network

MikroTik CRS305-1G-4S+IN 10 GbE switch

10 GbE bonded NIC for NAS reads.

$159
TIER 3

EPYC + 4× GPU Workstation

$12,500 · the serious self-hosted lab

Single-socket EPYC 9354, 256 GB ECC, four RTX 4090 24 GB. 96 GB combined VRAM hosts Llama 3 70B at Q6 or Mixtral 8x22B at Q4. Real PCIe topology, no bandwidth-starved x4 slot. Bridge to actually-professional territory.

Power~400 W idle / 2200 W peak
Noise45 dBA idle / 60 dBA load
Form factorFull tower or 4U, dedicated AC line, ideally a separate room
ServesLlama 70B Q6 · Mixtral 8x22B · 16+ concurrent users
$12,500total

Wins

  • 96 GB total VRAM unlocks 70B Q6 + multiple models
  • IPMI for remote control
  • Real datacenter parts

Drawbacks

  • Loud, needs a closed room or rack
  • Heavy power draw
  • Not living-room friendly

CPU

AMD EPYC 9354 (32C/64T)

128 PCIe 5.0 lanes, 12-channel DDR5.

$3,299

GPU ×4

4× NVIDIA RTX 4090 24 GB

Used $1,400 ea. Blower-style fans recommended.

$5,600Amazon

RAM

256 GB DDR5-4800 ECC RDIMM (8×32)

Eight-channel, ECC mandatory.

$1,899

Storage

Solidigm D7-PS1010 3.84 TB U.2 ×2 (mirror)

Datacenter NVMe, mirrored for safety.

$1,499

Motherboard

Asus K14PA-U24 EPYC server board

6× PCIe 5.0 x16, IPMI, dual 10 GbE.

$999

PSU

FSP CUP-T2 2000 W Platinum

2000 W to handle 4× GPU bursts.

$419

Case

SilverStone RM43-320-RS 4U rackmount

Hot-swap drive bays, IPMI cable routing.

$459

Network

Mellanox ConnectX-4 25 GbE

25 GbE NIC; SFP28 to MikroTik CRS504.

$199
TIER 4

Rackmount Inference Cluster

$32,000+ · production-grade hosting at home

Supermicro 4124GS-TNR, dual EPYC 9554, 1 TB ECC RAM, four RTX 6000 Ada (192 GB total VRAM), redundant 2200 W PSUs, IPMI. The smallest viable production AI server you can build at home, and the largest that fits in a normal house.

Power~600 W idle / 3000 W peak
Noise55 dBA idle / 75 dBA load (real rack noise)
Form factor2U server, dedicated 30 A circuit, basement / garage rack
Serves70B FP16 · 405B Q3 · multi-tenant inference
$32,000total

Wins

  • Datacenter-grade reliability
  • Production multi-tenant capable
  • Redundant PSUs

Drawbacks

  • Sounds like a small jet engine
  • 30 A circuit required
  • Eye-watering price

CPU ×2

2× AMD EPYC 9554 (64C/128T each)

Dual-socket for max PCIe + memory channels.

$14,598

GPU ×4

4× NVIDIA RTX 6000 Ada 48 GB

Or 4× L40S, datacenter-grade.

$25,200

RAM

1 TB DDR5-4800 ECC RDIMM (16×64)

1 TB lets you host the cold weights of MoE models.

$8,499

Storage

4× Solidigm D7-PS1010 7.68 TB (RAID-Z1)

~23 TB usable, datacenter NVMe.

$5,796

Server chassis

Supermicro AS-4124GS-TNR

2U, 8× PCIe slots, dual-CPU, IPMI.

$4,999

PSU

Built-in 2× 2200 W Platinum (redundant)

Included with chassis.

$0

Networking

2× ConnectX-6 100 GbE + UniFi PRO 24 PoE

100 GbE for NVMe-over-fabric across nodes.

$1,299

Rack & UPS

StarTech 25U + APC SRT3000RMXLA 3 kVA UPS

Real rack, real UPS.

$2,199

Headless Linux setup, the AI server stack

We standardize on Ubuntu Server LTS 24.04 + Docker Compose + Tailscale + Caddy. The boring choices are the right ones for an always-on server.

Base OS

# Ubuntu Server 24.04 LTS
# Pick: minimal install, OpenSSH, no GUI
sudo apt update && sudo apt full-upgrade -y
sudo apt install -y curl tmux htop nvtop \
                    docker.io docker-compose-v2 \
                    fail2ban ufw

NVIDIA stack

# Driver + Container Toolkit
sudo ubuntu-drivers autoinstall
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure \
     --runtime=docker
sudo systemctl restart docker

Tailscale (remote access)

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --ssh \
     --advertise-tags=tag:server \
     --accept-routes
# Now reachable at 100.x.x.x from anywhere

Docker Compose, Ollama + WebUI

services:
  ollama:
    image: ollama/ollama:latest
    runtime: nvidia
    volumes: ["./models:/root/.ollama"]
    restart: unless-stopped
  webui:
    image: ghcr.io/open-webui/open-webui:main
    ports: ["3000:8080"]
    depends_on: [ollama]

Power, networking, IPMI, the un-fun parts

Power planning

A standard 15 A residential circuit safely sustains ~1,440 W (15 A × 120 V × 0.8 derating). Anything past that wants its own 20 A or 30 A circuit. Plan UPS sizing around the peak, not idle, a 1500 VA UPS covers 4× RTX 4090 during a brief brownout but cannot sustain it.

Networking

1 GbE works for serving chat. NAS-backed training datasets want 10 GbE minimum; multi-node inference (vLLM with TP>1) wants 25/100 GbE. MikroTik CRS305 (4× SFP+) and CRS504 (4× QSFP28) are the cheap-but-real switches we use in our lab.

IPMI / BMC

Worth it on EPYC / Xeon W boards. Remote power, virtual KVM, sensor logging, virtual media for OS install. Never expose IPMI to the internet, even with a strong password. Put it on a tagged VLAN or behind Tailscale.

Cooling & noise

Multi-GPU servers run hot. Blower-style cards (RTX 6000 Ada, RTX A6000) exhaust heat out the back rather than dumping it on the next card. For consumer cards (4090, 5090), space them with at least one slot of breathing room, three-slot cards in adjacent slots will thermal-throttle.

Frequently asked questions

Can I host a local AI server in my closet?

Yes, but plan for heat and noise. A single-GPU dev box draws 350–500 W and is closet-tolerable with passive ventilation. A 4-GPU workstation draws 2,000+ W and needs active extraction or a dedicated room.

Should I run Linux or Windows on an AI server?

Linux. Driver maturity, CUDA stack stability, Docker support, and remote management (SSH, tmux, htop) are all dramatically better. Ubuntu Server LTS 24.04 or Debian 12 are the default picks.

Do I need IPMI for a home AI server?

Not strictly, Tailscale + ssh + a Pi-KVM gets you 90% there for free. IPMI is worth it on EPYC / Xeon W server boards because remote console + power control + sensor monitoring + virtual-media are bundled.

How loud is a 4-GPU AI server?

Idle, 35–45 dBA with mid-range Noctua fans. Loaded with blower-style GPUs at 80% fan, 55–65 dBA, comparable to a vacuum cleaner. Move it out of your bedroom.

Is dual EPYC overkill for AI inference?

Single EPYC is enough for almost all inference workloads, 4 GPUs hang off 64 PCIe lanes comfortably. Dual EPYC only makes sense for very high-batch inference servers or when CPU-side preprocessing is the bottleneck.

Can I expose my AI server to the internet?

Don't, at least not directly. Put it behind Tailscale, WireGuard, or a Cloudflare Tunnel. Add basic-auth to Open WebUI. A bare-internet Ollama port is asking to be turned into someone else's free GPT.

Related guides

Workstation

Best AI Workstation Builds in 2026 (Every Budget)

Four hand-tuned AI workstation builds from $1,500 to $20,000+. Component-level recommendations for training, fine-tuning, and large-model inference. Cooling, PSU, motherboard, multi-GPU.

Read guide Homelab

AI Homelab Setup: The 2026 Build Guide

Build a complete AI homelab. Networking, NAS for datasets, GPU passthrough on Proxmox, Ollama + Open WebUI, local agents, monitoring, cooling, power budgeting. Three budget tiers.

Read guide GPU

Best GPU for Local LLMs (2026)

Hands-on tested ranking of the best GPUs for running Llama 3, DeepSeek, Mixtral, and other large language models on your own hardware. VRAM, tokens/sec, $/perf, updated May 2026.

Read guide Software

Run DeepSeek Locally, R1, V3 & Coder Guide

Hardware sweet spots for DeepSeek-R1, DeepSeek V3, and DeepSeek-Coder. Quantization, VRAM math, real tokens/sec measurements on consumer GPUs, workstations, and Apple Silicon.

Read guide
AI PC Builder GPU database Deals

Stay Ahead of the AI Curve

Get weekly AI hardware news, benchmark updates, and deals in your inbox. Founding-subscriber list, be one of the first.

✓ No spam✓ Weekly digest✓ Unsubscribe anytime

Affiliate disclosure: As an Amazon Associate, MyAIHardware.com earns from qualifying purchases. Server builds are spec'd from neutral evaluation.