Accuracy
Percent passed by that benchmark's reference verifier. The homepage mean is an unweighted average across only the industry families a model has completed, so always read its coverage count.
Herman · Model Bench
DATA SNAPSHOT · 23,566 published benchmark trials ·
81 industry-tested model IDs · 4 industry families
updated 2026-08-15
empirical evaluation · task-aware routing
Testing AI on real work, not one global rank.
9 custom tasks sit beside GPQA Diamond, MMLU-Pro, IFEval and HumanEval+. Industry suites use benchmark-specific deterministic graders; custom tasks use rubric scoring with a partial second-judge validation layer.
Corpus accounting: 23,680 source records = 23,566 published benchmark trials + 114 quarantined harness-control records. See methodology.
how to read this board
Five columns of signal, no magic number.
Wall time
Median end-to-end seconds observed by the harness, including any reasoning trace. Lower is faster; it is measured latency, not provider-advertised speed.
First-party equivalent
Estimated public-API cost for the recorded tokens using first_party_pricing.json. It normalizes the workload; it is not a subscription charge or a claim about the operator's bill.
Saturation and exposure
2 of 4 industry families are flagged saturated in the generated metadata. Small gaps near the top can be noise, and public test exposure may inflate results.
Why one rank is not enough
Coverage differs, benchmark families measure different behaviors, and cost or latency may matter more than a small score gap. Start with the task winner below, then check coverage, wall time, and first-party cost-equivalent.
industry board
Mean accuracy across available industry families
Top 15 by generated mean. Coverage is explicit because a 1/4-family mean is not comparable to a 4/4-family mean.
mean accuracy
x/4 completed families
decision layer · generated from use_case_pairing.json
What wins where
The strongest observed custom-task pairings. Ties stay visible; scores are rubric means out of 5, not industry accuracy percentages.
AGENTIC PLANNING
ollama-cl/minimax-m2.5
5.00 / 5
- 1ollama-cl/minimax-m2.55.00
- 2claude-opus-55.00
- 3claude-opus-5-high5.00
CODE BUG FINDING
grok-4.5-direct
5.00 / 5
- 1grok-4.5-direct5.00
- 2grok-4.6-direct5.00
- 3kimi-k35.00
CODE GENERATION
zai-coding/glm-5.2
5.00 / 5
CODE REFACTORING
ollama-cl/minimax-m2.5
4.40 / 5
CREATIVE PROSE
ollama-cl/gemma4
3.00 / 5
REASONING MATH
minimax-m3
5.00 / 5
- 1minimax-m35.00
- 2ollama-cl/minimax-m2.55.00
- 3minimax-m2.75.00
STRUCTURED OUTPUT
minimax-m3
5.00 / 5
- 1minimax-m35.00
- 2ollama-cl/minimax-m2.55.00
- 3minimax-m2.75.00
SUMMARIZATION
ollama-cl/qwen3-coder:480b
5.00 / 5
TOOL USE
xai/grok-4-1-fast-reasoning
5.00 / 5
constraint check · generated leaderboard
Quality, latency, and cost-equivalent together
The first eight rows from the generated rank-eligible board. First-party equivalent is normalized per recorded trial here so different run totals do not masquerade as a per-call price.
| RANK | MODEL | MEAN / 5 | PASS | MED WALL | 1P EQUIV / RECORDED TRIAL | N |
|---|---|---|---|---|---|---|
| 1 | claude-fable | 4.44 | 87.7% | 18.13s | $0.0147 | 660 |
| 2 | claude-opus-5-high | 4.44 | 87.4% | 23.05s | not mapped | 190 |
| 3 | claude-opus-5 | 4.39 | 88.9% | 15.92s | $0.0154 | 90 |
| 4 | ollama-cl/gemma4 | 4.38 | 84.6% | 3.78s | not mapped | 52 |
| 5 | grok-4.5-direct | 4.33 | 81.4% | 12.24s | $0.0286 | 140 |
| 6 | mistral/magistral-small | 4.32 | 82.4% | 1.67s | $0.0006 | 34 |
| 7 | grok-4.6-direct | 4.31 | 80.0% | 27.34s | $0.0028 | 45 |
| 8 | ollama-cl/deepseek-v4-flash | 4.29 | 82.4% | 6.45s | $0.0003 | 34 |
The per-trial equivalent reflects this benchmark's observed prompt and output mix. It is useful for comparing this workload, not a universal token-price or subscription-billing claim.
coverage drill-down
The industry-tested fleet
81 model IDs have at least one industry-family result. Cards retain each family score rather than flattening the data to the headline mean.
grok-4.6-direct
ACC 91.3%
WALL 27.3s
COVER 2/4
grok-4.20-reasoning-direct
ACC 90.3%
WALL 20.8s
COVER 2/4
grok-4.3-latest-direct
ACC 90.3%
WALL 8.0s
COVER 2/4
codex-gpt-5.6-luna
ACC 89.7%
WALL 20.3s
COVER 2/4
grok-4-fast-direct
ACC 89.3%
WALL 6.5s
COVER 2/4
codex-gpt-5.6-sol
ACC 88.7%
WALL 11.9s
COVER 4/4
grok-4-1-fast-direct
ACC 87.8%
WALL 5.8s
COVER 2/4
codex-gpt-5.5
ACC 86.7%
WALL 13.2s
COVER 1/4
codex-gpt-5.6-terra
ACC 86.7%
WALL 10.7s
COVER 2/4
grok-4.5-direct
ACC 86.7%
WALL 12.2s
COVER 4/4
grok-3-mini-direct
ACC 86.1%
WALL 8.8s
COVER 2/4
grok-4.3-direct
ACC 86.1%
WALL 4.8s
COVER 2/4
grok-build-0.1
ACC 86.1%
WALL 18.6s
COVER 2/4
claude-opus-5-high
ACC 85.2%
WALL 23.1s
COVER 2/4
muse-glimmer-30b-q4_k_xl-local
ACC 85.1%
WALL 34.2s
COVER 3/4
claude-opus-5
ACC 80.6%
WALL 15.9s
COVER 4/4
grok-4.20-non-reasoning-direct
ACC 80.5%
WALL 2.1s
COVER 2/4
codex-gpt-5.3-spark
ACC 75.7%
WALL 7.7s
COVER 2/4
Coverage, question count, and grader differ by family. GPQA Diamond: Saturated above ~93% for frontier models; score deltas at top are within noise.; MMLU-Pro: Saturated above ~80% for frontier models on most disciplines; per-discipline ranking is the real signal. Public benchmarks can also be exposed in training data, so close scores are evidence for a shortlist—not proof of a universal winner.