LLM Inference Benchmarking, Methodology v1.0

Bandwidth-bound workloads are the ones where the GPU spends most of its cycles waiting for the next chunk of weights to arrive from VRAM. A 70B model at Q4_K_M is roughly 40 GB of weights. To produce one token, the runtime reads most of those weights once. A card with 1 TB/s of memory bandwidth therefore caps near 25 tokens per second on a 40 GB read, ignoring KV cache overhead. The H100 at 3.35 TB/s caps near 80 tokens per second by the same back-of-envelope.

Compute-bound workloads, like prefill on a long prompt or SDXL denoising, do the opposite. The runtime has loaded the weights once and is now doing a lot of math with them in parallel. Tensor-core utilization climbs, and FP16 TFLOPS starts to matter. The 4090's 82.6 TFLOPS rating becomes a real ceiling.

Memory-capacity-bound workloads are the ones where the model simply does not fit. DeepSeek V3 at Q4 is 380 GB of weights. No single consumer card runs it. The question becomes, can you partition across cards, across CPU and GPU, or across machines, and at what tokens-per-second penalty. We benchmark the realistic multi-device configurations, not the impossible single-card ones.

Two workloads in the list deserve their own paragraph because they look unusual at first glance. The prefill-32k workload measures how fast a model processes a 32,768-token input prompt before producing a single output token. This is the bottleneck for retrieval-augmented chatbots, long-document summarization, and codebase Q&A. On a 4090, prompt prefill for Llama 3 8B at 32K runs around 4,000 to 5,500 tokens per second, two orders of magnitude faster than the same card's decode rate, because prefill is one large batched matmul rather than 32K sequential reads. We report prefill separately because conflating it with decode would give an over-rosy picture of long-context latency.

The batched-decode-8 workload measures throughput when eight independent users are decoding simultaneously on the same card. Continuous batching amortizes the weight read across all eight streams, so total tokens per second climbs to roughly 4 to 6x the single-stream rate, even though per-user latency degrades by 10 to 25 percent. This is the relevant number for anyone planning to host a small inference service for their team. Buyers in Lagos and Istanbul running shared internal chatbots on a single workstation card care about this number far more than the single-stream rate.

Quantization is a runtime weight-compression scheme. A 16-bit weight gets stored in 4, 5, or 8 bits, with a small per-block scale factor. The point is to shrink the VRAM footprint and reduce the bandwidth read per token, paying for it in a small loss of output quality. The four tiers we test cover the practical bands buyers actually run.

• Q4_K_M, roughly 4.85 bits per weight on average. The hobbyist default. Llama 3 70B drops to about 40 GB. Quality loss on standard benchmarks (MMLU, HumanEval) is typically 1 to 3 points versus FP16. This is the tier we publish first for every model.
• Q5_K_M, roughly 5.69 bits per weight. The quality-conscious tier. The same 70B drops to about 48 GB instead of 40. Quality loss versus FP16 is usually under 1 point. We publish Q5 for models where the Q4 drop is measurable.
• Q8_0, 8 bits per weight, near-lossless versus FP16 on every benchmark we have tested. Doubles VRAM versus Q4. We publish Q8 numbers for hardware that has the VRAM headroom (24 GB+ for 8B models, 80 GB+ for 70B models). Useful as a quality ceiling reference.
• FP16, 16 bits per weight, the uncompressed reference. Two full bytes per parameter, so 70B is 140 GB. Only workstation-class hardware (H100 80GB pairs, MI300X 192GB, multi-card consumer setups) can hold this. We publish FP16 numbers as the upper bound, never the expected case.

We do not currently publish Q2 or Q3 numbers. Below Q4, quality loss becomes model-dependent in ways that need per-model evaluation harnesses, and the tokens-per- second gain over Q4 is small because bandwidth savings flatten. We will add Q3_K_S when we have community-submitted runs that include MMLU and HumanEval scores at the same quant, so the speed gain can be priced against the quality loss.

Every benchmark record in our database carries a verification tier. The tier describes how the number was measured and how much weight we give it when there is a conflict between sources.

When two tiers conflict for the same (device, workload, quant, context) tuple, Tier 1 wins unconditionally. Otherwise we publish the lower-tier number and append a dissenting-source note linking to the conflicting measurement. We do not silently pick a winner.

Our cost-per-MTokens figure is a three-year-amortized total cost of ownership divided by the tokens that hardware will plausibly generate in that period at the published utilization. The formula is below, and every input is configurable on the cost calculator page so a reader can substitute their own assumptions.

cost_per_MTokens =
  (
    hardware_capex / amortization_years
    + power_draw_watts * 8760 * utilization * electricity_rate_per_kWh
  )
  / (tokens_per_second * 86400 * 365 * utilization / 1_000_000)

The v1.0 default assumptions are:

• Amortization, 3 years. Typical refresh cycle for prosumer AI hardware. The 4090 launched October 2022, the 5090 January 2025, a 28-month consumer cadence.
• Utilization, 30 percent. Median for a self-hosted local-LLM operator who runs the rig for inference during the day, training experiments at night, and idle otherwise. Cloud-style 24/7 utilization is a separate scenario.
• Electricity rate, 0.15 USD per kWh. Rough US residential average. We expose this control because rates in Lagos (off-grid generator backup) and Istanbul (TEDAS tariffs) are materially different, and a Berlin reader paying 0.40 EUR per kWh needs a different answer.
• Power draw, the published TDP. A 4090 at 450W, an MI300X at 750W, an M4 Max at 140W system-level. We acknowledge actual draw is workload-dependent and frequently below TDP for memory-bound inference; the calculator will accept a measured-watts override in v1.1.

We do not include cooling, networking, or storage in the v1.0 figure. A reasonable upper-bound for cooling overhead on a single-card consumer setup is 5 to 10 percent of card power, and we will add this in v1.1 alongside a multi-card data-center profile.

One worked example, because the formula above is dense. A new RTX 4090 at 1,800 USD capex, three-year amortization, 30 percent utilization, 0.15 USD per kWh, 450W TDP, running Llama 3 70B Q4_K_M at a measured 13 tokens per second. The annualized capex share is 600 USD. The annualized power share is 450W times 8,760 hours times 0.30 utilization times 0.15 USD per kWh, which is roughly 177 USD. Total annual cost, about 777 USD. Annual tokens generated, 13 times 86,400 times 365 times 0.30, roughly 123 million. Cost per million tokens, 6.30 USD. Compare this to a typical hosted API at 4 to 8 USD per million output tokens, and the picture for a heavy user becomes interesting. For an Istanbul operator paying TEDAS Tarife 1 (closer to 0.07 USD per kWh after subsidies), the same rig drops to about 4.55 USD per million tokens. For a Lagos operator on diesel-backup power averaging closer to 0.22 USD per kWh, the same rig rises to about 7.40 USD per million. The cost calculator on this site computes this for any input you supply.

We deliberately exclude the price of operator time. Local inference is cheap on machine cost and expensive on attention, and pricing attention is too personal to bake into a public formula. A hobbyist who enjoys tuning their rig gets near-zero effective hourly cost. A founder spending eight hours debugging a vLLM deployment instead of shipping product is paying a real cost the calculator cannot see. We flag this trade-off in the buyer's-guide articles rather than the formula.

A benchmark you cannot reproduce is a rumor. Our reproducibility commitments for v1.0 are listed below. Each is a thing a reader can check against our published data.

• llama.cpp commit pin. Every Tier 1 and Tier 2 record cites the llama.cpp commit hash that produced the number. We refuse to ingest a record without it. The current reference pin for v1.0 is llama.cpp@b3447 (built 2026-05-15). We will bump quarterly.
• Prompt-suite hash. Our reference prompt set is a 50-prompt JSON file at fixed seed, SHA-256 7b9a...e3. The full file lives in the public API at /api/v1/prompt-suite-v1.json. Anyone running this suite against the cited llama.cpp pin should land within 5 percent of our reported tokens-per-second.
• Hardware spec format. Each record specifies GPU model, VRAM in GB, system RAM in GB, CPU model, OS and kernel, PCIe lane width, and power limit. Missing fields are flagged as "under-specified" on the record card.
• Run count and dispersion. Median of five runs is our default. We publish both the median and the interquartile range. A record with IQR above 15 percent of the median is flagged for re-measurement.
• Submission via myai-bench CLI. Our open- source CLI parses llama-bench output, runs the canonical 5-run median, and emits schema-matching JSON. Reproducibility is therefore one command away. See the Tools for builders section on the main methodology page.

A methodology is more useful when it lists its own blind spots. Our v1.0 has the following gaps. We will close them in subsequent versions.

• No first-party lab yet. We aggregate published numbers and verify contributor submissions. We do not yet rerun every number on our own hardware. Q3 2026 target for the first reference rig.
• No multi-tenant or production-batch numbers. Our published tokens-per-second is single-stream or batch-of-8 at most. Real production deployments at batch 64 or 128 with continuous batching (vLLM, TensorRT-LLM) show very different per-card economics. We will add a production batched track in v1.1.
• No fine-tuning or training numbers. Methodology v1.0 is inference-only. Training and fine-tuning have different bottleneck profiles (gradient sync, optimizer state, activation checkpointing) and deserve their own methodology document. Planned v2.0.
• No multimodal or vision-language tokens-per-second. We benchmark text generation. We do not yet publish a comparable figure for VLMs (LLaVA, Qwen-VL) because the image-prefill step distorts the tokens-per-second number and the right measure is still under community debate.
• No quality-versus-speed Pareto frontier. We report quantization speed gains, but not the per-quant quality loss in MMLU, HumanEval, or GSM8K terms. We will add a quality column once we have a defensible reference harness.
• Limited ARM and edge coverage. Our Apple Silicon and Snapdragon X coverage is thinner than NVIDIA and AMD. We are prioritizing community submissions on these platforms in 2026.

If you reference our data in a paper, blog post, or comparison article, please cite the methodology version your numbers were drawn against. BibTeX entry below.

@techreport{myaihardware_methodology_v1_2026,
  author      = {Ohwofa, Fredoline},
  title       = {LLM Inference Benchmarking, Methodology v1.0},
  institution = {MyAIHardware},
  year        = {2026},
  month       = {June},
  type        = {Whitepaper},
  url         = {https://www.myaihardware.com/methodology/llm-inference-benchmarking-v1},
  note        = {CC BY 4.0}
}

Plain-text reference for non-academic use:

Ohwofa, F. (2026). LLM Inference Benchmarking, Methodology v1.0. MyAIHardware Whitepaper, June 2026. Available at https://www.myaihardware.com/methodology/llm-inference-benchmarking-v1.

Reuse is permitted under the Creative Commons Attribution 4.0 International license. Translation into other languages is welcome; please open a pull request or email [email protected] so we can link your translation from this page.