Run Llama 3 70B on a 24GB Consumer GPU With Q4 Quantization

The premise: 70B on consumer hardware

Llama 3 70B has 70 billion parameters. At FP16 (16 bits per parameter) the weights alone are 140 GB. At FP8 they're 70 GB. At Q4 they're around 40 GB, still too much for a single 24 GB GPU. But the Q4_K_M variant in GGUF format compresses the embeddings and certain layers more aggressively, dropping the size to ~38-40 GB. Even that doesn't quite fit in 24 GB of VRAM.

The trick is intelligent layer offload: keep most of the weights on the GPU, push the rest to system RAM, and let llama.cpp shuttle them in and out per token. With 24 GB of VRAM and a reasonable amount of system RAM, you can run 70B at 8-12 tokens/sec, slow by API standards but fully usable for chat.

Step 1: Pick the right quant

Ollama's default `llama3.3:70b` tag points to Q4_K_M, which is the sweet spot for quality vs size. Lower quants (Q3, Q2) lose noticeable quality. Higher quants (Q5, Q6) need more VRAM for diminishing returns. Stick with Q4_K_M unless you have specific reasons to deviate.

# Default tag (Q4_K_M)
ollama pull llama3.3:70b

# Or specify the quant explicitly
ollama pull llama3.3:70b-instruct-q4_K_M

# Smaller, lower quality
ollama pull llama3.3:70b-instruct-q3_K_M  # ~32 GB, fits in 24 GB with spillover

# Larger, higher quality (needs 2x 24 GB cards)
ollama pull llama3.3:70b-instruct-q5_K_M  # ~50 GB

Step 2: Configure layer offload

Ollama autodetects how many layers fit in VRAM. For Llama 3 70B Q4_K_M on a 24 GB card it usually offloads 60-70 of the 80 layers. The remaining layers run on CPU. You can override this with the num_gpu parameter, set it to a higher number to push more to GPU (risks OOM), or lower to keep CPU fallback safe.

# Create a Modelfile that pins layer offload
cat > Modelfile-70b <<EOF
FROM llama3.3:70b
PARAMETER num_gpu 70
PARAMETER num_ctx 8192
PARAMETER num_batch 512
EOF

ollama create llama3-70b-tuned -f Modelfile-70b
ollama run llama3-70b-tuned

Step 3: Tune the KV cache

Beyond model weights, the biggest VRAM hog is the KV cache, the model's internal memory of every token in the current conversation. For a 70B model at 8K context the FP16 KV cache is around 2.5 GB. At 32K context it's 10 GB. If you don't need long contexts, set num_ctx to 4096, you'll save 5+ GB of VRAM. If you need long contexts but are tight on VRAM, quantize the KV cache to 8 bits.

# In your Modelfile
PARAMETER num_ctx 4096       # smaller context = less KV cache

# Or quantize the KV cache via env var
launchctl setenv OLLAMA_KV_CACHE_TYPE q8_0  # macOS
sudo systemctl edit ollama.service           # Linux, set in [Service]
# Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Step 4: Benchmark and validate quality

Run the model and measure tokens/sec with `--verbose`. On a 4090 + DDR5 + Q4_K_M expect 9-12 tok/s on simple prompts and 6-8 tok/s as context grows. Then validate quality: ask it the same question you'd ask the cloud model. Q4_K_M of 70B is genuinely close to FP16 70B for most tasks, close enough that you won't notice the difference in chat.

ollama run llama3.3:70b --verbose "Write a 300-word explanation of how the Transformer's attention mechanism scales with context length."

# Look at the final stats. eval rate (tokens/s) should be 6-12 on a 4090.
# prompt eval rate should be 200+ tokens/s.

Step 5: When 70B isn't worth it

Be honest about whether you actually need 70B. For chat, RAG, coding assistants, and most agentic workflows, Llama 3.1 8B or Qwen 2.5 14B are good enough, and they run at 40+ tokens/sec, not 8. 70B pays off when you're doing complex reasoning, creative writing where style matters, or specialty domains where the smaller models hallucinate. Pick the right tool, bigger isn't always better.

Step 6: Multi-GPU is the cure for speed

If you have two 24 GB cards, you don't need to quantize down or offload to CPU. Ollama distributes layers across both cards automatically. A dual-3090 setup runs 70B Q5_K_M at 18-22 tokens/sec, fully on GPU. The cost-effectiveness of two used 3090s vs one new 4090 is the open secret of the local-AI community, see our multi-GPU tutorial for the wiring details.