Methodology
How the benchmark was designed, run, and scored
Current evaluation surface
This snapshot contains 23,680 source records: 23,566 published benchmark trials, plus 114 quarantined harness-control or unclassified records. The published set contains 8,033 custom-suite trials across 69 custom-suite tested model IDs and 15,533 industry-suite trials across 81 industry-tested model IDs, spanning 9 custom tasks and 4 industry benchmark families. The two sets overlap, so neither is the catalogue total. These totals come from one generated snapshot contract rather than page copy.
The dataset has accumulated over multiple waves, so repetitions are not uniform. Every ranking should be read with its task coverage and trial count. The generated board separates 63 rank-eligible rows from 5 incomplete-coverage rows.
Two scoring paths
Custom task suite
Custom trials run at temperature 0 where the provider accepts it. A task-specific rubric defines the expected behavior. The primary minimax-m3 judge receives the task, expected criteria, and the first 4,000 characters of the response, then returns a 0–5 score, a short reason, and a pass/fail flag.
A second-family Fable overlay checks a subset of trials for judge sensitivity; it is validation coverage, not a claim that every custom result has two judges. The generated overlay currently contains 1168 records across 166 cells, with 85% pass agreement.
Industry benchmark suite
Industry trials do not use the custom LLM judge. Each family keeps its own prompt and deterministic grader, as recorded in generated benchmark metadata:
| BENCHMARK | PROMPT | GRADER | TRIALS |
|---|---|---|---|
| GPQA Diamond | 5-shot CoT (vals.ai compatible) | regex on `answer is (X)` | 4,418 |
| MMLU-Pro | Question + (A) ... (J) options → Answer: | regex on `Answer: (X)` letter | 4,546 |
| IFEval | verbatim user prompt + verifiable constraints | strict + loose verifier (constraint-by-constraint) | 4,611 |
| HumanEval+ | Function signature + docstring → complete the body | execute candidate code against hidden tests (sandbox via tools/python) | 1,958 |
Accuracy is the percent passed by that family's verifier. The homepage industry mean is the unweighted mean of a model's available family percentages; it is always paired with an x/4 coverage count because incomplete means are not apples-to-apples.
Cost and latency
Wall time is end-to-end elapsed time captured by the harness, including reasoning. The site usually shows the median so a few long calls do not dominate the comparison.
Wire cost is the provider-side signal captured for that route. A zero can mean subscription routing, a flat-rate route, or no metered signal; it must not be read as a universal “free” price.
First-party cost-equivalent applies the public-API input and output rates in first_party_pricing.json to the recorded token counts. It estimates what the observed workload would cost at those rates. It is not the operator's subscription bill, and models without a mapped public rate remain unknown rather than being treated as free.
Custom task coverage
The nine generated task IDs cover structured output, code, summarization, prose, reasoning, planning, refactoring, and tool orchestration:
| TASK | CAPABILITY | RUBRIC FOCUS |
|---|---|---|
code_debug | Find and repair a concrete bug | Correct line, cause, and viable fix |
code_gen_long | Generate a complete Python component | Required API, types, locking, and runnable demo |
json_strict | Follow a strict output schema | Valid JSON, exact keys, types, and values |
summarize | Compress a technical source | Exact bullet and word limits without drift |
creative_write | Satisfy prose and structure constraints | Length, voice, required motifs, and ending |
reasoning_multistep | Carry a chained calculation | All intermediate steps and final answer |
agentic_prompt | Plan and begin an executable task | Valid plan, risks, and concrete first action |
refactor_existing_code | Change code without breaking behavior | Preserved semantics and requested structure |
agentic_tool_use | Orchestrate a multi-step tool sequence | Correct order, tools, arguments, verification, and risks |
Known limitations
- Coverage and N vary. More trials tighten a result; a small score difference with low N is not a durable lead.
- The second judge is partial. The Fable overlay measures judge sensitivity on selected cells, not the entire corpus. Creative-writing scores are especially judge-dependent in the generated cross-judge data.
- Public benchmarks can saturate or leak into training.
GPQA Diamond: Saturated above ~93% for frontier models; score deltas at top are within noise.;MMLU-Pro: Saturated above ~80% for frontier models on most disciplines; per-discipline ranking is the real signal. - A zero is operational as well as semantic. No response, a parse failure, or a model that spends its output budget on reasoning can all produce a failing trial.
- The custom suite is single-turn and text-only. It does not measure long-running tool execution, memory across turns, vision, audio, or video.
Reproducing the generated site data
The companion harness owns the append-only trial logs and deterministic aggregation. The bundled benchmark/ directory is a source snapshot; its production paths assume the host's /var/lib/hermes/model-benchmark layout.
- Run a scoped trial wave with
.venv/bin/python3 harness.py --models <subset> --tasks <subset> --reps N --parallel 4 --out <jsonl>. - Regenerate aggregates with
.venv/bin/python3 scorecard.py. - Sync the generated JSON and bounded raw-trial export with
.venv/bin/python3 sync_to_site.py --trial-txt-limit 200. - Build the site with
hugo --minify.