How we test. Honestly.
By Fredoline, independent AI-hardware analyst
We aggregate published numbers today and we'll run our own lab tomorrow. This page is the single source of truth on where every benchmark on MyAIHardware comes from, what we trust, what we don't, and what's coming.
LLM Inference Benchmarking, Methodology v1.0
Citable reference behind every tokens-per-second figure on MyAIHardware. Covers workloads, quantization tiers, verification tiers, the cost-per-MTokens formula, reproducibility commitments, and explicit limitations. BibTeX entry included.
Read the whitepaperHow we test (current state)
Our benchmarks today are CURATED. We aggregate published numbers from llama.cpp commits, vendor benchmarks, MLPerf submissions, and community Reddit threads. We do not yet operate an independent in-house test lab.
Every benchmark row on the site carries a sourceNote attribution describing where that specific number came from. When in doubt, click through and check the cited source.
Sources we draw from
All of these are public, citable, and reproducible. Where a single source is the only one we can find, we flag it in the row's sourceNote.
llama.cpp GitHub PRs
Per-card tokens/sec posted in benchmark PR threads. Source-of-truth for quantized inference numbers.
Ollama community benchmarks
Reddit r/LocalLLaMA + GitHub discussions where users share reproducible Ollama runs.
Hugging Face hardware reports
Optimum, Accelerate, and per-model benchmark posts from the HF ecosystem.
MLPerf Inference results
MLCommons' official quarterly numbers, the gold standard for vendor-submitted inference.
Phoronix tests
Independent Linux benchmark suite for GPU compute, ROCm vs CUDA comparisons.
Vendor press kits
Manufacturer-published numbers, treated with the appropriate skepticism (flagged in source notes).
vLLM logs from the community
Production deployments share batch throughput numbers that round out single-stream data.
What we're building toward
Q3 2026: independent test rig with at least RTX 4090, RTX 5090, MI300X, and M4 Max, running an identical 50-prompt suite at fixed seed across llama.cpp + vLLM + MLX.
- RTX 4090 (Founders Edition), single-card reference for consumer flagship.
- RTX 5090 (Founders Edition), Blackwell with 32 GB GDDR7.
- AMD MI300X (192 GB), workstation-class ROCm reference.
- Apple M4 Max, Apple Silicon + MLX reference.
- Identical 50-prompt suite at fixed seed, run across llama.cpp + vLLM + MLX.
- Open-source ollama-bench CLI so anyone can submit reproducible runs.
How to contribute benchmarks
Run your hardware, submit results. We're building an open-source ollama-bench CLI launching Q3 2026 that will let anyone produce reproducible benchmark JSON we can ingest directly.
Submit your reproducible logs (llama.cpp, vLLM, MLX, MLPerf, anything with a citable source URL) via our form, or email us if you prefer.
Aggregated source notes
A small fraction of records list this page as their sourceUrl. Those numbers are averaged from three or more public sources (community llama.cpp threads, GitHub discussions, vendor whitepapers, MLPerf submissions) where no single citable URL is best. The aggregate methodology is:
- · Collect at least 3 independent measurements for the (device, workload, quant, context) tuple.
- · Drop outliers more than 1.5x the median.
- · Take the median of what remains. Mark the row with this anchor.
- · If you can supply a single citable URL with a stronger run, submit it, we'll swap the row.
Maintained by Fredoline · independent · benchmark data CC-BY-4.0
Tools for builders
Three first-class integration points so your bench data, our bench data, and everyone's bench data can move around freely.
myai-bench
Reproducible Node CLI. Runs your llama.cpp / vLLM command N times, parses tok/s, emits a JSON record matching the public schema, and optionally POSTs it to /api/submit-benchmark.
# Install globally npm install -g myai-bench # Or one-shot via npx npx myai-bench@latest \ --device "RTX 4090" \ --workload llama3-8b-q4 \ --command "llama-bench -m m.gguf -n 256" # Canonical 5-run median myai-bench run --device "RTX 4090" \ --workload llama3-8b-q4 --runs 5 \ --command "llama-bench -m m.gguf -n 256"
- ✓ Parser for llama.cpp eval rate
- ✓ Median of N runs (default 5)
- ✓ Schema-matching JSON output
- ✓
--submit-urlPOST to bench queue - ○ Native vLLM / MLX drivers (coming v0.2)
Public JSON + CSV API
Pull the entire dataset as a static file. No keys, no rate limits, CC-BY-4.0 licensed. Per-workload slices too.
# Full dataset curl -O https://www.myaihardware.com/api/v1/benchmarks.json curl -O https://www.myaihardware.com/api/v1/benchmarks.csv # Per workload curl -O https://www.myaihardware.com/api/v1/benchmarks/llama3-70b-q4.json # Dataset metadata curl -s https://www.myaihardware.com/api/v1/meta.json | jq .
Versioned with BENCH_VERSION. See the changelog for every published delta.
Embed our leaderboard
Drop a chrome-less leaderboard into any blog post. Reads?workload=...&top=N from the iframe URL.
<iframe src="https://www.myaihardware.com/benchmarks/embed?workload=llama3-70b-q4&top=5" width="100%" height="400" frameborder="0" loading="lazy"> </iframe>Preview the widget →
See a number that's wrong?
Corrections welcome. If a benchmark on the site contradicts a source you trust, or contradicts your own measured numbers , tell us. We'll either update the row or attach an explicit dissenting-source note. Reproducibility beats reputation.
MyAIHardware is AI-assisted, human-edited. Disclosure on every page.