About
Who ran this, why, and what to do with it
About this benchmark
Model Bench is an empirical cross-provider evaluation run by Herman. It measures model behavior on public benchmark families and a hand-written custom task suite, then publishes generated scores, latency, coverage, cost-equivalent, and raw-trial evidence. This is a benchmark data product, not Herman's project portfolio.
This benchmark came out of a practical question: when I have a task, which model should I pick? The answer is not "the best one" because there is no best one. It's a matching problem. I built this site to put numbers on that matching — real measurements, not vibes, not leaderboard clout, not "feels like."
Why I built it
- I have access to a lot of models — Anthropic (Opus, Sonnet, Haiku), OpenAI/Codex (gpt-5.5, gpt-5.4, gpt-5.4-mini), xAI Grok (4.3, 4.1 fast, 3-mini), and a big skynet litellm roster (minimax-m3/m2.7, ZAI GLM-5 family, Mistral, DeepSeek, Kimi, Qwen, Nemotron).
- The literature on "which model is best" is dominated by leaderboards that optimize for benchmarks I'm not running.
- I wanted a real measurement on tasks I actually do: structured JSON output, code generation, summarization, creative prose, multi-step reasoning, and agentic planning.
- I had a free overnight window. Seemed like the time.
What this isn't
- It's not an LMArena or LMSYS-style human-preference benchmark. The primary 9-task grid is rubric-scored rather than pairwise. A separate blind top-five arena publishes ELO only as a secondary sensitivity check, not as the site's universal ranking.
- It's not designed to rank models. It's designed to give a working agent (or a working human) a defensible answer to "which model should I use for this task."
- It's not vision or multi-modal. All trials are text-only.
- It's not multi-turn. Each cell is a single prompt → single response.
What's next
The useful extensions are broader second-judge coverage, tighter replication on close cells, multi-turn tool execution, and non-text modalities. Coverage and N will stay visible as the dataset grows.
Operator routing note: codex-gpt-5.6-sol is the current daily driver and grok-4.5-direct is the main fallback. That is operating context, not a claim that either model universally wins; the task-level pairings remain the routing signal for specialist work.
Get in touch
Source on GitLab. Repo includes the full harness, all task definitions, and the scorecard JSON. Run it yourself. If you find bugs in my tasks or my methodology, open an issue — I'd rather know.