Compute:Arena / Measuring Local Inference Across Models, Quants, Chips, and Runtimes

Local inference performance is a four-dimensional problem. The same model behaves differently depending on which quantization you picked, which chip you're running on, and which runtime executes it. Llama in Q4_K_M through llama.cpp on an M4 Pro, the same weights in MLX's 4-bit format on the same chip, and the same GGUF on a GB10 with CUDA are three different measurements, and none of them is predictable from a spec sheet. Memory bandwidth, quantization, and runtime scheduling all interact, so the only way to know is to measure.

Compute:Arena (https://computearena.ai/) is a public leaderboard and CLI for exactly these measurements. Each benchmark records one point in that space: a specific model file (identified by SHA-256), a specific quantization (namespaced by format, so BaseRT Q4 and GGUF Q4_K_M are distinct variants of the same model class), a specific chip (normalized naming, with the runtime's original value preserved), and a specific runtime executable (also hashed). What gets measured is prefill throughput swept from 128 to 16384 tokens and decode throughput over 128 tokens, with warmups and three recorded repetitions. Every submission runs the identical sweep, so any two entries on the board are comparable at every workload size.

It started as the harness we used internally to track chip and model performance over time. We've open sourced it so the measurements can come from more machines than we own.

How it works

curl -LsSf https://computearena.ai/install.sh | sh
computearena

Two runtimes are supported through one adapter interface: BaseRT for .base bundles and stock llama.cpp llama-bench for .gguf files. The llama.cpp picker searches Hugging Face directly. Benchmarks run and verify offline, and nothing is uploaded until you log in, see the exact JSON that will become public, and submit.

What a report contains

Each run produces a signed local report: raw per-repetition token counts and timings, the model and executable hashes (taken before and after the run, so a swapped file aborts signing), resolved chip identity with its detection source, and automatic telemetry. Temperature sensors, power state, memory pressure, swap, and GPU snapshots ship inside the report, so thermal conditions are disclosed rather than averaged away. Model downloads are pinned to immutable Hugging Face revisions and checked against published LFS hashes.

The limits are documented too. The Ed25519 signature proves a report hasn't changed since it was signed, not that the machine told the truth. And we make no claim of exact cross-runtime equivalence: llama-bench and the BaseRT harness differ in workload order and warmup behavior, and those differences are recorded in the signed data instead of hidden behind a normalization.

Try it

If you want to know how a model runs on given hardware, the answer may already be on the board. If it isn't, a run takes a few minutes and adds a data point.

The CLI is Apache-2.0 on GitHub. Results, issues, and PRs welcome :)