← back

openbench

Provider agnostic eval harness

Evals is a messy space, where many metrics are released without any insight into how the eval is ran. This information is necessary for high fidelity of eval results and comparisons. At Groq, we built openbench, a provider agnostic eval harness, that aimed at standardizing the eval process by making it completely open. Openbench allows you to benchmark any model, from any provider, with 1 to 1 equivalence to the original eval implementation. Openbench covers thousands of evals across long horizon SWE, cybersecurity, multimodal, science, etc.

How it works

Openbench is a CLI tool built on top of Inspect AI. Each eval is implemented straight from its original paper: the same dataset, solver agent and scoring method. We provide a shared interface for those components across benchmarks so that common things like math grading are written once and reused everywhere. To run an eval, you just pick a benchmark and a model:

bench eval mmlu --model groq/openai/gpt-oss-120b

The model string is what makes it provider agnostic. Swap groq/ for openai/, anthropic/, ollama/. Openbench supports 30+ providers and the eval runs exactly the same way. Coding and agentic evals run in Docker sandboxes, results are logged locally and can be explored with bench view or pushed to Hugging Face, and you can point openbench at your own eval file to run private benchmarks with the same setup.