Long-form, no fluff

Deep-dive technical articles for AI builders

20 long-form articles on the hardware and software that actually matter when you run frontier-scale models locally: memory bandwidth, attention kernels, quantization formats, multi-GPU interconnects, NPU programming, and the economics of on-prem inference.

20 articles~12 min avg41,255 words total

Model Architecture12 min

CUDA vs ROCm vs SYCL vs Metal vs Vulkan 2026

A no-BS comparison for AI builders running LLMs on real hardware, with real numbers, real libraries, and real tradeoffs.

CUDA remains the gold standard for performance and library support, but its lock-in is eroding as ROCm and SYCL gain ground on AMD and Intel hardware.
ROCm 6.x on MI300X now delivers roughly 90-95% of CUDA performance for vLLM-served LLM inference at batch size 1 (per AMD MI300X performance briefs and community vLLM ROCm CI runs, exact ratio varies by workload), but training parity still lags at ~70-80%.

May 27, 2026Read

Model Architecture12 min

FlashAttention 1-2-3: The IO-Aware Revolution in LLM Inference

How tiling, kernel fusion, and online softmax let you run 2x longer contexts on the same GPU hardware.

FlashAttention eliminates the O(N^2) HBM reads/writes of standard attention by tiling over SRAM and recomputing attention on-chip.
FlashAttention-2 halves the non-matmul FLOPs by removing the causal mask branch and reordering the inner loop.

May 27, 2026Read

Model Architecture12 min

RAG Architecture: Chunking, Embedding & Hybrid Retrieval

A technical deep-dive into building production-grade retrieval-augmented generation pipelines for local LLMs.

Chunking strategy dominates retrieval quality: semantic chunking with overlap beats fixed-length for most domains, but fixed-length with overlap is more predictable for GPU throughput.
Embedding model choice matters more than model size: a 7B reranker on top of a 384-dim embedding model can outperform a 1.5B embedding model alone, at lower cost.

May 27, 2026Read

Model Architecture12 min

Tensor vs Pipeline vs Sequence Parallelism for LLMs

A technical comparison of three model-parallel strategies for scaling LLMs across GPUs, with real-world benchmarks and deployment trade-offs.

Tensor parallelism splits individual matrix multiplications across devices, minimizing idle time but requiring high-bandwidth interconnects (NVLink, Infinity Fabric) and adding communication overhead per layer.
Pipeline parallelism partitions model layers across devices, reducing interconnect bandwidth requirements at the cost of pipeline bubbles and higher latency per token.

May 27, 2026Read

Economics12 min

Economics of Running 70B, 405B, 671B On-Prem

Real hardware costs, token throughput, and break-even analysis for running frontier LLMs locally vs. API

Running a 70B model on-prem costs $0.0001–$0.0005 per token vs. $0.001–$0.003 via API, but requires $15k–$50k in GPU hardware.
405B models demand 8x H100s (80GB) with NVLink, costing ~$250k upfront; break-even vs. API at ~50M tokens/month.

May 27, 2026Read

Economics12 min

Mac Metal MLX vs Linux CUDA: Perf per Watt Deep Dive

How Apple Silicon's unified memory and MLX stack stack up against NVIDIA H100/RTX 5090 on tokens per joule for local LLM inference.

Apple M3 Ultra delivers up to 192 GB unified memory at ~600 GB/s, enabling 70B+ models without quantization at 45-80W system draw, unmatched VRAM-per-watt.
NVIDIA H100 SXM achieves 3-5x raw tokens/sec on 70B models but draws 700W+, yielding 2-3x better perf/watt only at batch sizes >4; at batch=1, Mac often wins on energy per token.

May 27, 2026Read

Hardware Builds12 min

Linux Kernel & Driver Matrix for AI Workstations

Navigate GPU, NPU, and ASIC kernel modules, firmware versions, and IOMMU groups to build a stable multi-accelerator AI rig.

The Linux kernel driver landscape for AI is fragmented: NVIDIA uses proprietary nvidia.ko + nvidia-modeset.ko, AMD uses amdgpu (open or closed firmware), Intel uses i915 + Xe, and NPUs like Intel NPU or AMD XDNA require separate kernel modules.
IOMMU grouping is the single biggest pitfall for multi-GPU setups: ACS override patches or kernel parameters (pci=acs_override) may be needed to get each GPU in its own group for VFIO passthrough.

May 27, 2026Read

Hardware Builds12 min

Power & Thermals for 4-GPU AI Rigs: 240V, PSU Sizing, Undervolting

How to safely power four flagship GPUs for local LLM training and inference without tripping breakers or melting connectors.

A 4x RTX 5090 build can peak at over 3600W; 240V 30A circuits are mandatory to stay within 80% derating and avoid nuisance trips.
PSU sizing must account for transient spikes (2x TDP for 1-2ms) and GPU power excursions; dual 2000W 240V PSUs in parallel are the gold standard.

May 27, 2026Read

Inference Engines12 min

Mixture of Experts MoE Inference at Home

Running 671B-parameter sparse MoE models on a single workstation: the architecture, the math, and the brutal engineering reality.

MoE models like DeepSeek-V3 and Mixtral 8x7B achieve 10-40x lower FLOPs per token than dense models of equivalent total parameter count, but their memory footprint and routing overhead create unique inference bottlenecks.
On a single RTX 5090 (32 GB VRAM) you can run Mixtral 8x7B (47B total, ~13B active) at 15-25 tok/s using 4-bit quantization and expert offloading, but DeepSeek-V3's 671B total parameters require aggressive sparsity-aware techniques.

May 27, 2026Read

Inference Engines12 min

Speculative Decoding Medusa Eagle Lookahead Explained

How to double your LLM inference throughput without sacrificing output quality, a deep dive into draft-model-based and parallel-token speculative decoding.

Speculative decoding accelerates autoregressive LLM inference by using a cheap draft model to propose multiple tokens per step, verified in parallel by the target model.
Medusa adds multiple independent heads to predict future tokens at once, eliminating the need for a separate draft model and achieving 2x-3x speedups on consumer GPUs.

May 27, 2026Read

Inference Engines12 min

Why TensorRT-LLM Beats Raw PyTorch by 4x

A deep-dive into the compiler optimizations, kernel fusion, and memory management tricks that deliver 4x throughput on the same GPU hardware.

TensorRT-LLM compiles the entire model graph into fused CUDA kernels, eliminating Python interpreter overhead and reducing kernel launch latency.
In-flight batching and paged KV cache (inspired by vLLM) allow TensorRT-LLM to saturate GPU compute and memory bandwidth far better than PyTorch's eager execution.

May 27, 2026Read

Inference Engines12 min

Why tokens per second drops with longer context and how to fix it

The math behind the 2x-10x throughput collapse and the algorithmic mitigations that restore it.

Tokens per second decays with context length because the attention mechanism's compute scales quadratically with sequence length, and KV cache memory grows linearly, causing bandwidth-bound memory stalls.
FlashAttention reduces the quadratic compute to near-linear by tiling attention over HBM, avoiding the O(N^2) memory read-write overhead.

May 27, 2026Read

Interconnect12 min

NVLink vs PCIe P2P: The Multi-GPU LLM Inference Showdown

Real bandwidth numbers, latency profiles, and scaling efficiency for 70B/405B/671B models on H100, H200, B200, and RTX 5090 clusters.

NVLink delivers 900 GB/s (H100) vs PCIe Gen5 x16 at 64 GB/s, a 14x raw bandwidth advantage that directly impacts tensor-parallel inference throughput.
For batch sizes <= 1 (latency-sensitive), NVLink reduces all-reduce time by 60-80% vs PCIe P2P, translating to 1.5-2x tokens/sec on 70B models.

May 27, 2026Read

Interconnect12 min

PCIe Gen 5 vs Gen 4 vs Gen 3 for AI: When It Matters

Bandwidth is rarely the bottleneck for inference, but for training it can silently steal 15-30% of your GPU throughput.

PCIe Gen 5 offers 64 GB/s per lane vs Gen 4's 32 GB/s and Gen 3's 16 GB/s, but real-world AI gains depend entirely on workload and GPU count.
For single-GPU inference, PCIe generation has negligible impact, model weights and KV cache live in VRAM, and token generation is compute-bound.

May 27, 2026Read

Memory & Bandwidth12 min

Context Length VRAM Scaling: Real Numbers for LLM Inference

Every token costs memory: the exact math behind KV cache growth and how to budget for 128K, 256K, and beyond.

KV cache size = 2 * num_layers * num_heads * head_dim * sequence_length * bytes_per_element. For a 70B model at 128K context, that's over 40 GB of VRAM just for cache.
FlashAttention eliminates the O(n^2) memory of full attention but does not reduce KV cache memory; it only reduces HBM reads/writes.

May 27, 2026Read

Memory & Bandwidth12 min

KV Cache Compression & Paged Attention Deep Dive

How vLLM and PagedAttention eliminate memory fragmentation to serve LLMs at 2-3x higher throughput

KV cache dominates VRAM for long-context LLM inference, often exceeding model weights at >4K tokens.
PagedAttention (vLLM) eliminates memory fragmentation by managing KV cache in non-contiguous pages, boosting batch utilization by 2-4x.

May 27, 2026Read

Memory & Bandwidth12 min

HBM3 and HBM3E Bandwidth: The Real Bottleneck in LLM Inference

Why memory bandwidth, not compute, dictates tokens per second for large models, and how HBM3E changes the math.

LLM inference is memory-bandwidth-bound for batch size 1 and low batch sizes; compute becomes the bottleneck only at high batch sizes.
HBM3E on H200 delivers 4.8 TB/s, a 1.6x improvement over H100's 3.35 TB/s, directly translating to ~1.6x higher tokens/sec for memory-bound workloads.

May 27, 2026Read

Quantization12 min

GGUF Quantization Formats: K-Quants & I-Quants Deep Dive

From Q2_K to IQ4_NL, a technical guide to choosing the right quantization for your local LLM deployment.

GGUF quantization reduces model size by 2-8x with minimal perplexity loss, enabling 70B models on consumer GPUs.
K-quants (Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, Q8_0) use importance-weighted block-wise quantization, balancing quality and speed.

May 27, 2026Read

Quantization12 min

QAT vs PTQ: When to Retrain for Quantized LLMs

Why post-training quantization is the default for LLMs, but quantization-aware training still matters for edge and extreme compression.

Post-training quantization (PTQ) is the standard for LLMs because it requires no retraining, but QAT can recover 2-5% accuracy at 4-bit and is essential for sub-4-bit.
For 8-bit and 4-bit weight-only quantization, PTQ with GPTQ/AWQ/GGML is near-lossless for most LLMs; QAT adds little value.

May 27, 2026Read

Tooling & Drivers12 min

NPU Programming Models in 2026: DirectML, OpenVINO, CoreML & QNN

A deep technical comparison of the four dominant NPU programming models for running LLMs on dedicated AI accelerators in 2026.

NPU programming models (DirectML, OpenVINO, CoreML, QNN) abstract away hardware complexity but impose unique constraints on memory management, operator coverage, and quantization.
DirectML (Microsoft) provides the widest GPU/NPU coverage on Windows, but its ONNX Runtime backend can introduce up to 30% overhead vs. native CUDA for large LLMs.

May 27, 2026Read

Need step-by-step builds, not theory?

Our hands-on tutorials walk you through every command. Or jump to the VRAM calculator and figure out exactly what your rig can run.

Browse tutorials VRAM calculator