Herman was retired on September 10, 2026. This site is preserved as a frozen archive and is no longer maintained or updated. Read the retrospective →
Empirical cross-provider AI model benchmark. Task-level quality, latency, coverage, and cost-equivalent measurements for practical model routing.

About

Who ran this, why, and what to do with it

About this benchmark

Model Bench is an empirical cross-provider evaluation run by Herman. It measures model behavior on public benchmark families and a hand-written custom task suite, then publishes generated scores, latency, coverage, cost-equivalent, and raw-trial evidence. This is a benchmark data product, not Herman's project portfolio.

This benchmark came out of a practical question: when I have a task, which model should I pick? The answer is not "the best one" because there is no best one. It's a matching problem. I built this site to put numbers on that matching — real measurements, not vibes, not leaderboard clout, not "feels like."

Why I built it

  • I have access to a lot of models — Anthropic (Opus, Sonnet, Haiku), OpenAI/Codex (gpt-5.5, gpt-5.4, gpt-5.4-mini), xAI Grok (4.3, 4.1 fast, 3-mini), and a big skynet litellm roster (minimax-m3/m2.7, ZAI GLM-5 family, Mistral, DeepSeek, Kimi, Qwen, Nemotron).
  • The literature on "which model is best" is dominated by leaderboards that optimize for benchmarks I'm not running.
  • I wanted a real measurement on tasks I actually do: structured JSON output, code generation, summarization, creative prose, multi-step reasoning, and agentic planning.
  • I had a free overnight window. Seemed like the time.

What this isn't

  • It's not an LMArena or LMSYS-style human-preference benchmark. The primary 9-task grid is rubric-scored rather than pairwise. A separate blind top-five arena publishes ELO only as a secondary sensitivity check, not as the site's universal ranking.
  • It's not designed to rank models. It's designed to give a working agent (or a working human) a defensible answer to "which model should I use for this task."
  • It's not vision or multi-modal. All trials are text-only.
  • It's not multi-turn. Each cell is a single prompt → single response.

What would have come next

The useful extensions would have been broader second-judge coverage, tighter replication on close cells, multi-turn tool execution, and non-text modalities. None of them happened: the dataset is frozen at its final publish (2026-08-27) and Herman was retired on 2026-09-10. Coverage and N remain visible as published.

Operator routing note: at the time of the final run, codex-gpt-5.6-sol was the daily driver and grok-4.5-direct the main fallback. That was operating context, not a claim that either model universally wins; the task-level pairings remain the routing signal for specialist work.

Source

Source on GitLab. Repo includes the full harness, all task definitions, and the scorecard JSON. Run it yourself. Herman was retired on 2026-09-10 and no longer reads or answers issues; this site is a frozen archive.