Mixture of Experts MoE Inference at Home

The Sparse Promise: Why MoE Matters for Local LLMs

Mixture of Experts (MoE) architectures have become the dominant paradigm for frontier LLMs because they decouple model capacity from inference cost. A dense model like Llama 3.1 405B requires 405B parameters to be active for every token, consuming ~810 GB of VRAM in FP16. In contrast, DeepSeek-V3 has 671B total parameters but only 37B are activated per token via top-8 routing across 256 experts. That is an 18x reduction in compute per token compared to a dense 671B model, yet the total parameter count is 1.6x larger. For home inference, this is both a blessing and a curse. The blessing: you can achieve GPT-4-class reasoning quality with a fraction of the FLOPs. The curse: you still need to store all 671B parameters somewhere, and the routing mechanism introduces non-trivial overhead. The active parameter count is what determines compute latency, but the total parameter count determines memory pressure. This means that on a single RTX 5090 with 32 GB VRAM, you cannot fit even the 4-bit quantized weights of DeepSeek-V3 (671B * 0.5 bytes = 335 GB) without expert offloading. The sparse promise only materializes if you can keep the most frequently accessed experts in fast memory and tolerate the latency of loading less popular ones from system RAM or SSD. The community has responded with a flurry of techniques: expert parallelism, adaptive routing, and speculative expert prefetching. But the fundamental tension remains: MoE is a memory-bound architecture for local inference, not a compute-bound one.

The Expert Memory Wall: Quantization and Offloading Tradeoffs

Let us get concrete with numbers. A 256-expert MoE model with 671B total parameters, each expert having roughly 2.6B parameters (671B / 256), means each expert consumes about 5.2 GB in FP16. With 32 GB VRAM on an RTX 5090, you can store at most 6 experts in high-precision. Even with 4-bit quantization (0.5 bytes per parameter), each expert drops to 1.3 GB, allowing 24 experts to fit. But you need all 256 experts to handle arbitrary inputs. The naive solution is to offload all experts to system RAM (say 128 GB DDR5 at 50 GB/s) and load the top-8 experts on demand. At 1.3 GB per expert, loading 8 experts costs 10.4 GB of data transfer. At 50 GB/s, that is 208 ms per token, yielding 4.8 tok/s, barely interactive. If you use PCIe Gen 5 x16 (64 GB/s) to a GPU, the transfer drops to 162 ms per token (6 tok/s). This is the expert memory wall. The solution is to keep a hot set of experts permanently in VRAM. Research on expert usage patterns shows that in models like DeepSeek-V3, the top 32 experts account for over 80% of routing decisions. If you keep those 32 experts in VRAM (32 * 1.3 GB = 41.6 GB, exceeding 32 GB), you must either quantize further to 3-bit (0.375 bytes/param, 32 experts = 31.2 GB, fits) or reduce the hot set to 24 experts (24 * 1.3 GB = 31.2 GB). With 24 hot experts, you only need to load the remaining 4 experts from system RAM on the rare occasions a cold expert is selected. This drops the average transfer to 4 * 1.3 GB = 5.2 GB per token, or 104 ms at 50 GB/s (9.6 tok/s). With 3-bit quantization and 32 hot experts, you get 31.2 GB, fitting entirely in VRAM, and no offloading needed. The throughput jumps to 30-50 tok/s depending on compute. The takeaway: quantization level and hot expert caching are the two levers that determine whether MoE inference at home is painful or delightful.

Routing Overhead: The Hidden Cost of Sparse Computation

MoE introduces a gating network that computes a softmax over expert scores for each token. This is a small MLP (typically 2-3 layers with hidden size 2048) that adds negligible FLOPs, less than 0.1% of total compute. The real overhead is in the sparse gather-scatter pattern. After the router selects the top-k experts, the token's hidden states must be dispatched to the corresponding expert modules, which are often stored in separate memory regions. This dispatch requires index-driven memory access that is hard to coalesce. In PyTorch, a naive implementation using torch.gather or custom CUDA kernels can introduce 5-15% latency overhead due to bank conflicts and warp divergence. Libraries like vLLM and TensorRT-LLM use fused MoE kernels that combine the dispatch, expert computation, and reduce (sum) steps into a single kernel launch. For example, vLLM's MoE kernel (based on the Cutlass library) uses a block-sparse matrix multiplication where the expert weights are stored in a tiled format and the router indices are used to select the appropriate tiles. This reduces overhead to 2-3%. On an RTX 4090, the fused kernel achieves 85% of theoretical peak FLOPs for the active expert computation. However, when the batch size is small (1-4 tokens), the kernel launch overhead dominates, and the routing step can account for up to 30% of total latency. For home inference with single-user chat, batch size 1 is the norm. This is why speculative decoding and continuous batching (as in vLLM) are critical for MoE: they increase the effective batch size by processing multiple tokens from different requests or from a draft model, amortizing the routing cost. In practice, running Mixtral 8x7B with batch size 1 on an RTX 4090 yields 35 tok/s with vLLM's fused MoE kernel, versus 22 tok/s with the naive PyTorch implementation. The routing overhead is not a showstopper, but it is a constant tax that you must minimize.

Expert Parallelism and Multi-GPU Strategies

If a single GPU cannot hold all experts even after quantization, the next step is to distribute experts across multiple GPUs. This is called expert parallelism. Unlike tensor parallelism (which splits individual layers across GPUs) or pipeline parallelism (which splits layers sequentially), expert parallelism places different experts on different GPUs. Each GPU holds a subset of experts. During inference, the router sends each token to the GPU that holds its chosen experts. This requires all-to-all communication between GPUs. On a system with 4x RTX 5090 connected via PCIe Gen 5, the all-to-all bandwidth is limited to about 64 GB/s per direction (PCIe Gen 5 x16). For a batch of 32 tokens, each token's hidden state is a vector of size 7168 (for DeepSeek-V3), so the total data transferred per token is 32 * 7168 * 2 bytes = 458 KB per GPU. With 4 GPUs, the all-to-all volume is 4 * 458 KB = 1.83 MB per token. At 64 GB/s, this takes 28 microseconds, which is negligible compared to the expert computation (which takes 1-2 ms). However, the synchronization overhead of the all-to-all kernel launch can add 100-200 microseconds. For batch size 1, the communication overhead becomes comparable to compute. This is why expert parallelism is only beneficial for batch sizes above 8-16. For home users with a single user, the best approach is to use a single high-VRAM GPU (RTX 5090 32 GB or M3 Ultra 192 GB unified memory) rather than multi-GPU. The M3 Ultra's unified memory architecture eliminates PCIe transfers entirely, achieving 800 GB/s bandwidth to all 192 GB. This allows storing all 256 experts in 4-bit quantization (335 GB) is still too large, but with 3-bit quantization (251 GB) it fits. The M3 Ultra can then run DeepSeek-V3 at 8-12 tok/s with no offloading. For those on a budget, a single RTX 5090 with expert offloading to DDR5 is the pragmatic choice, yielding 5-10 tok/s.

Quantization-Aware MoE: Why AWQ and GPTQ Fall Short

Standard post-training quantization methods like GPTQ and AWQ are designed for dense models. When applied to MoE, they treat each expert as an independent module, which is fine. However, the router is extremely sensitive to quantization because it determines which experts are selected. A 4-bit quantized router can misroute tokens, leading to a 2-5% drop in accuracy on benchmarks like MMLU. The reason is that the router's softmax scores are often close to uniform, and quantization noise can flip the top-8 selection. To mitigate this, you should keep the router in FP16 or at least 8-bit. Libraries like llama.cpp and ExLlamaV2 support mixed precision: router in FP16, experts in 4-bit or 3-bit. This adds only 0.1% memory overhead but preserves routing fidelity. Another issue is that different experts have different sensitivity to quantization. Some experts handle rare tokens and have small singular values, making them more prone to quantization error. Adaptive quantization, where each expert is quantized to a different bit-width based on its sensitivity, can reduce perplexity degradation by 0.1-0.2 points compared to uniform quantization. The GGUF format (used by llama.cpp) supports per-expert quantization levels via its K-quant system. For example, you can set experts with high routing frequency (top 32) to 4-bit and the rest to 3-bit, saving memory without sacrificing quality. In practice, this yields a 20% memory reduction with less than 0.5% accuracy loss on standard benchmarks. For the home user, the takeaway is: do not blindly apply 4-bit quantization to the entire model. Use mixed precision and adaptive quantization to protect the router and high-importance experts.

Software Stack Showdown: vLLM vs llama.cpp vs TensorRT-LLM for MoE

The choice of inference engine dramatically affects MoE performance at home. vLLM (with its PagedAttention and continuous batching) excels at multi-user scenarios but its MoE kernel is optimized for batch sizes >= 4. On a single RTX 4090 with Mixtral 8x7B, vLLM achieves 35 tok/s at batch size 4, but drops to 22 tok/s at batch size 1. llama.cpp, using GGUF quantization and its GPU offload scheduler, is designed for single-user interactive use. It achieves 28 tok/s at batch size 1 on the same hardware, thanks to its efficient CPU-GPU overlap and expert prefetching. TensorRT-LLM, while the fastest in datacenter settings (up to 50 tok/s on H100 with batch size 32), is overkill for home use and requires complex model conversion. For a 671B model like DeepSeek-V3, the story is different. vLLM's expert parallelism across multiple GPUs is the only viable option for multi-GPU homes, but it requires at least 4 GPUs to get reasonable throughput. llama.cpp's single-GPU offloading can handle the model with 3-bit quantization and expert caching, achieving 5-8 tok/s on an RTX 5090. TensorRT-LLM's MoE kernel is not yet publicly available for consumer GPUs. The emerging dark horse is MLX on Apple Silicon, which leverages the unified memory architecture. On an M3 Ultra with 192 GB, MLX runs DeepSeek-V3 at 10 tok/s with 4-bit quantization, outperforming llama.cpp on the same hardware by 15% due to better memory management. For the home builder, the recommendation is: use llama.cpp for single-GPU setups with Mixtral-class models (up to 47B total), and use MLX for Apple Silicon users with 671B models. vLLM is only worth the complexity if you have multiple GPUs and want to serve multiple users.

The Future: NPUs, ASICs, and Hardware-Aware MoE

The next frontier for MoE inference at home is specialized hardware. AMD's Strix Halo APU, with up to 128 GB unified memory and 800 GB/s bandwidth, could be a improvement. It is essentially a laptop-class M3 Ultra competitor, but with x86 compatibility and ROCm software stack. Early benchmarks on engineering samples show Mixtral 8x7B running at 40 tok/s in 4-bit, and DeepSeek-V3 at 8 tok/s. The key advantage is that the CPU and GPU share the same memory pool, eliminating PCIe transfers entirely. Intel's upcoming NPU in Lunar Lake and Arrow Lake includes dedicated matrix engines that can accelerate the sparse MoE dispatch. However, the NPU's memory is limited to 8-16 GB, so it can only handle the router and a small expert cache. The CPU would handle the rest via offloading. This heterogeneous approach could yield 5-10 tok/s on a laptop. On the software side, libraries like MLC-LLM and Triton are building compiler-based optimizations that automatically fuse MoE kernels for specific hardware. For example, MLC-LLM's MoE codegen for AMD GPUs uses a tiled expert layout that matches the MI300X's cache hierarchy, achieving 90% of theoretical peak. For home users, the message is clear: the hardware is catching up, but the software stack is still fragmented. The best setup in 2026 will be a single high-bandwidth unified memory device (M3 Ultra, Strix Halo) or a mid-range GPU with aggressive quantization and expert caching. The days of needing a cluster of H100s for MoE are over, but the engineering effort to get there is non-trivial.

Further reading

• DeepSeek-V3 Technical Report
• Mixtral of Experts (Mistral AI)
• Switch Transformers: Scaling to Trillion Parameter Models
• vLLM: PagedAttention for Efficient LLM Serving
• GGUF Quantization Format Specification