llama.cpp benchmarks and evidence
llama.cpp performance depends on the model artifact, quantization, context, backend and batch size. The earlier fixed comparison table did not link individual run artifacts and has been withdrawn. We do not claim to have tested every GPU, Apple Silicon machine or mini PC under one protocol.
Llama 3 8B Q4: comparable records
Only records with the same workload, quantization, context, batch-one setting and stated llama.cpp version qualify. Missing settings remain visible in the broader catalog but are excluded from this speed ranking. Matching metadata does not independently verify a report.
No multi-device llama.cpp comparison has sufficient runtime metadata in the current catalog. A winner or speed ratio cannot be established from these records.
Inspect all Llama 3 8B Q4 source records and their settingsCollect a result that another reader can check
Use the upstream llama-bench utility and record its exact commit and command. Publish the model artifact filename, revision and complete hash, prompt-processing and generation settings, context, batch, backend and offloaded layers. Record warmups separately and retain all raw outputs, including failed runs.
Prompt processing and token generation measure different work. A large prompt-processing number does not predict chat response speed. Likewise, concurrent throughput is not the speed experienced by one user.
Upstream llama-bench documentationChoose hardware from memory and software requirements
Check the exact artifact size before comparing speed. A 70B Q4_K_M artifact is roughly 40GB before runtime and KV cache; a 32GB GPU cannot hold it fully. CPU offload or model splitting changes the configuration and must be stated in the result.
CUDA, HIP/ROCm, Metal, Vulkan and CPU backends support different devices and operations. Consult the current upstream compatibility documentation rather than assuming a fixed speed ratio across vendors.