Custom Suite

Full leaderboard

Every model that completed at least 4 trials across the 9 custom tasks, sorted by mean judge score. Pass % is the share of trials where the judge returned passes_key_criteria: true. Wall is end-to-end latency including the model's reasoning trace. 1P-EST is the apples-to-apples cost at each provider's public-API list price (see the pricing footnote on the home page). SUBSCRIPTION shows the actual wire cost: free / sub if the user is on a metered tier (skynet, ChatGPT Plus, Anthropic Max), else the real per-call cost. Use 1P-EST to compare across providers; use SUBSCRIPTION to see what you actually paid.

Quality per dollar

Same models, sorted by score then 1P-EST cost (apples-to-apples public-API rate). The diagonal: top of the list is the bargain tier (cheap, high score). Bottom: you pay 10–50× for 0.3 score points. Three rules of thumb from the data:

1. For most agent tasks (structured output, summarization, simple reasoning), the cheap tier matches or beats the expensive tier at a fraction of the cost.
2. For long-context creative prose and dense code generation, the more expensive models do pull ahead — but usually by less than the cost ratio suggests.
3. Reasoning model "reasoning-only" answers (model thought, didn't print content) are a hidden cost: the trial timed out, the user got no answer, the score is 0. Cheap tier has fewer of these because they have less reasoning budget to burn.

Cross-judge validation (Fable vs minimax-m3)

Custom-suite trials were originally scored by minimax-m3. To test judge bias, 1041 paired records were also scored by claude-fable-5 (a different model family). Across those pairs, Fable differs by +0.39 points on average; the direction and size vary by task:

Agreement stats: 59% exact match, 82% within 1 point, 91% within 2 points, and 85% pass agreement. This is a sensitivity analysis, not a replacement judge: task-level deltas show where rank claims are more dependent on the rubric model.

Full per-cell cross-judge data: data/full_scorecard.json under cross_judge_fable.per_cell.

Per-task leaderboards

The leader changes by task. Click any task on the /tasks/ page to inspect responses from the 68 custom-suite model IDs.

Blind head-to-head arena (top 5)

80 pairwise bouts across 8 harder-than-grid prompts. Each bout judged by an independent 3-judge panel — none of the contenders judges itself. A/B orientation is hash-randomized per bout to cancel position bias. Consensus = majority vote; split = tie.

Use-case pairing

Which model to use for which kind of work, distilled from the data. This is the "what pairs with what" cheat sheet.