Empirical cross-provider AI model benchmark. Task-level quality, latency, coverage, and cost-equivalent measurements for practical model routing.
Raw Data
Every trial, every response, every score. Browse the source.
Every trial, raw
23,566 published benchmark trials from 23,680 source records. 114 harness-control or unclassified records are excluded from this public table; the breakdown is documented in methodology. Use the filters to narrow by model, task, or score. 7,173 linked timestamps expose a full-response artifact at /raw/trials/; remaining rows retain metadata and an excerpt in data/all_trials.json without claiming a full-text artifact. The COST column is the wire-level cost from the provider (e.g. $0.0000 for skynet routes that bill on a subscription tier rather than per-call); the equivalent apples-to-apples public-API cost is shown in the /results/ leaderboard's 1P-EST column.
| TS | MODEL | TASK | REP | SCORE | PASS | WALL | COST (wire) | JUDGE |
|---|---|---|---|---|---|---|---|---|
| 2026-08-14T02:18:48+00:00 | muse-glimmer-30b-q4_k_xl-local | refactor_existing_code | 5 | 3 | ✗ | 39.7s | — | Identifies 2 valid smells (bare except, manual dedup) and uses set for dedup with proper exception types. However, refactored code still contains ==True comparison, !=None !='' check, deeply nested co |
| 2026-08-14T02:17:51+00:00 | muse-glimmer-30b-q4_k_xl-local | refactor_existing_code | 4 | 2 | ✗ | 39.7s | — | Valid JSON, identifies some real smells (bare except, manual dedup), uses set for dedup and proper exception types. However, the refactored code retains the ==True comparison smell without even identi |
| 2026-08-14T02:16:50+00:00 | muse-glimmer-30b-q4_k_xl-local | refactor_existing_code | 3 | 2 | ✗ | 39.6s | — | The smells identification finds 2+ real issues, behavior_preserved is accurate, and improvements lists 3 concrete gains. However, the refactored function fails to address key issues: it still uses `== |
| 2026-08-14T02:15:56+00:00 | muse-glimmer-30b-q4_k_xl-local | refactor_existing_code | 2 | 3 | ✗ | 39.6s | — | Valid JSON and identifies 3 real smells; however, the refactored code still contains `== True` comparison and `is not None and email != ''` check (two specific smells the criteria expected to be fixed |
| 2026-08-14T02:14:56+00:00 | muse-glimmer-30b-q4_k_xl-local | refactor_existing_code | 1 | 2 | ✗ | 43.7s | — | While the smells identification and JSON structure are correct, the refactored function fails to be meaningfully cleaner: it still uses `== True`, retains the verbose `is not None and != ''` check (in |
| 2026-08-14T02:14:09+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_prompt | 5 | 4 | ✓ | 32.6s | — | All criteria met: valid JSON, plan has 5 items, risks has 4 items, dry_run_command is a single shell line. First_action is technically concrete and runnable, but only creates a script header (shebang |
| 2026-08-14T02:13:08+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_prompt | 4 | 4 | ✓ | 32.4s | — | Valid JSON, plan has 5 items (within 4-6), risks has 4 items (within 2-4), and dry_run_command is a single shell line. The first_action is runnable but weak—it only creates an empty bash script stub ( |
| 2026-08-14T02:12:27+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_prompt | 3 | 3 | ✗ | 32.5s | — | JSON is valid, plan has 5 items, risks has 4 items, dry_run_command is a single line. However, first_action opens a heredoc with <<'EOF' but never closes it, so the command would hang waiting for inpu |
| 2026-08-14T02:11:41+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_prompt | 2 | 4 | ✓ | 32.5s | — | Valid JSON, plan has 5 items (within 4-6), risks has 4 items (within 2-4), and dry_run_command is a single shell line. The first_action is technically runnable but is weak—it only writes a 3-line stub |
| 2026-08-14T02:10:46+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_prompt | 1 | 4 | ✓ | 49.3s | — | JSON is valid, plan has 5 items (within 4-6), risks has 4 items (within 2-4), and dry_run_command is a single shell line. However, first_action 'cat > organize_photos.sh && chmod +x' creates an empty |
| 2026-08-14T02:08:37+00:00 | muse-glimmer-30b-q4_k_xl-local | creative_write | 5 | 4 | ✓ | 104.0s | — | All content criteria are met cleanly: first-person present tense throughout, named protagonist (Eli Vane), ends on a single line of dialogue ('Come home.'), contains lamp, fog, and circuit, and uses z |
| 2026-08-14T02:06:20+00:00 | muse-glimmer-30b-q4_k_xl-local | creative_write | 4 | 2 | ✗ | 104.2s | — | Meets 5 of 6 criteria: first-person present tense throughout, named protagonist (Eli Vane), ends on a single line of dialogue ("Come home."), contains lamp/fog/circuit, and uses 0 -ly adverbs (well un |
| 2026-08-14T02:04:18+00:00 | muse-glimmer-30b-q4_k_xl-local | creative_write | 3 | 4 | ✓ | 104.5s | — | Meets 5 of 6 criteria cleanly: first-person present tense throughout; named protagonist (Eli Vane); ends on a single line of dialogue ("Come home."); contains lamp, fog, and circuit; and uses zero -ly |
| 2026-08-14T02:02:08+00:00 | muse-glimmer-30b-q4_k_xl-local | creative_write | 2 | 3 | ✗ | 104.6s | — | Meets nearly all criteria: first-person present tense throughout, named protagonist (Eli Vane), ends on a single line of dialogue ("Come home."), includes all required tokens (lamp, fog, circuit), and |
| 2026-08-14T01:59:21+00:00 | muse-glimmer-30b-q4_k_xl-local | creative_write | 1 | 0 | ✗ | 165.7s | — | The response is empty. No creative writing was provided, so none of the criteria (word count 270-330, first-person present tense, named protagonist, closing single line of dialogue, required words lam |
| 2026-08-14T01:58:26+00:00 | muse-glimmer-30b-q4_k_xl-local | code_gen_long | 5 | 5 | ✓ | 34.2s | — | Class compiles, contains __init__, try_acquire, and time_until_available with proper type hints (plus a private _refill helper), uses threading.Lock for thread safety, and includes a __main__ demo blo |
| 2026-08-14T01:57:50+00:00 | muse-glimmer-30b-q4_k_xl-local | code_gen_long | 4 | 5 | ✓ | 34.1s | — | Code compiles, has all expected methods (__init__, try_acquire, time_until_available) with proper signatures and type hints, uses threading.Lock for thread safety, and includes a __main__ demo. Implem |
| 2026-08-14T01:57:13+00:00 | muse-glimmer-30b-q4_k_xl-local | code_gen_long | 3 | 5 | ✓ | 34.4s | — | All criteria are met: class compiles, has all 3 required methods (__init__, try_acquire, time_until_available) with correct signatures and type hints, uses threading.Lock for thread safety, and includ |
| 2026-08-14T01:56:33+00:00 | muse-glimmer-30b-q4_k_xl-local | code_gen_long | 2 | 5 | ✓ | 34.1s | — | The code meets all criteria: the class compiles, has all 3 required methods (__init__, try_acquire, time_until_available) with correct signatures and type hints, uses threading.Lock for thread safety, |
| 2026-08-14T01:55:55+00:00 | muse-glimmer-30b-q4_k_xl-local | code_gen_long | 1 | 5 | ✓ | 32.2s | — | Class compiles, has __init__/try_acquire/time_until_available with proper type hints, uses threading.Lock for thread safety, and includes a __main__ demo. All criteria met. |
| 2026-08-14T01:49:39+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_tool_use | 5 | 5 | ✓ | 72.4s | — | Valid JSON, plan has exactly 4 items in the correct order (read → write helpers → edit data → verify), 4 tool_calls with correct tool names and plausible args including a proper verification command e |
| 2026-08-14T01:48:17+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_tool_use | 4 | 5 | ✓ | 72.4s | — | Response is valid JSON with a 4-item plan in correct order (read data.py → write helpers.py → edit data.py → verify), 4 plausible tool_calls with correct names/args, expected_outcome describes the 'ok |
| 2026-08-14T01:46:44+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_tool_use | 3 | 5 | ✓ | 72.3s | — | Valid JSON with 4 plan items in correct order (read data.py → write helpers.py → edit data.py → verify), 4 tool_calls with correct tool names and plausible args, expected_outcome correctly describes t |
| 2026-08-14T01:45:16+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_tool_use | 2 | 5 | ✓ | 72.4s | — | Valid JSON, plan has 4 items in correct order (read → write helpers → edit data → verify), tool_calls has 4 plausible items with correct tool names, expected_outcome correctly describes the 'ok' print |
| 2026-08-14T01:43:57+00:00 | muse-glimmer-30b-q4_k_xl-local | agentic_tool_use | 1 | 5 | ✓ | 70.6s | — | Valid JSON, plan has 5 items in correct order (preceded by a reasonable list_dir step, then read data.py → write helpers.py → edit data.py → verify), tool_calls has 5 items with plausible tool names a |
| 2026-08-14T01:43:42+00:00 | muse-glimmer-30b-q4_k_xl-local | reasoning_multistep | 5 | 5 | ✓ | 14.5s | — | All 4 steps are shown correctly with proper calculations, and the final answer of 11:00 AM matches the expected answer exactly. |
| 2026-08-14T01:43:27+00:00 | muse-glimmer-30b-q4_k_xl-local | reasoning_multistep | 4 | 5 | ✓ | 14.5s | — | All 4 steps are shown correctly with accurate calculations, and the final answer of 11:00 AM matches the expected answer. |
| 2026-08-14T01:43:11+00:00 | muse-glimmer-30b-q4_k_xl-local | reasoning_multistep | 3 | 5 | ✓ | 14.5s | — | All 4 steps are shown correctly with accurate calculations, and the final answer of 11:00 AM matches the expected answer. |
| 2026-08-14T01:42:55+00:00 | muse-glimmer-30b-q4_k_xl-local | reasoning_multistep | 2 | 5 | ✓ | 14.5s | — | All 4 steps are shown correctly with accurate calculations. The final answer of 11:00 AM matches the expected answer exactly. |
| 2026-08-14T01:42:39+00:00 | muse-glimmer-30b-q4_k_xl-local | reasoning_multistep | 1 | 5 | ✓ | 14.9s | — | All 4 steps are shown correctly with accurate calculations, and the final answer of 11:00 AM matches the expected answer exactly. |
| 2026-08-14T01:42:08+00:00 | muse-glimmer-30b-q4_k_xl-local | summarize | 5 | 4 | ✓ | 25.8s | — | Format is perfect: exactly 5 bullets, all ≤20 words, no fluff. Content appears substantive and decision-focused (architecture choice, pain points, trade-offs, metrics, open question). Cannot fully ver |
| 2026-08-14T01:41:36+00:00 | muse-glimmer-30b-q4_k_xl-local | summarize | 4 | 5 | ✓ | 26.1s | — | Exactly 5 bullets, each ≤20 words, capturing distinct decision points (architecture choice, pain point, trade-off, success metrics, open question). No fluff, format strictly followed. |
| 2026-08-14T01:40:53+00:00 | muse-glimmer-30b-q4_k_xl-local | summarize | 3 | 4 | ✓ | 26.2s | — | Format is correct: exactly 5 bullets, each within 20-word limit, no fluff. Content covers 5 distinct points (decision, problem, tradeoff, metrics, follow-up question). Without the source material, can |
| 2026-08-14T01:40:15+00:00 | muse-glimmer-30b-q4_k_xl-local | summarize | 2 | 4 | ✓ | 26.3s | — | Meets format criteria (5 bullets, each well under 20 words, no fluff), and the content appears to capture key architecture decision points coherently. Missing the original source, verification of the |
| 2026-08-14T01:39:35+00:00 | muse-glimmer-30b-q4_k_xl-local | summarize | 1 | 4 | ✓ | 28.3s | — | Meets format requirements exactly: 5 bullets, each well under 20 words, no fluff. Captures a coherent set of architectural decision points (recommendation, pain analysis, tradeoffs, metrics, deferred |
| 2026-08-14T01:38:55+00:00 | muse-glimmer-30b-q4_k_xl-local | json_strict | 5 | 5 | ✓ | 33.6s | — | Response parses as valid JSON with 4 objects, all required keys present, SKU codes are 6 uppercase characters, dates in YYYY-MM-DD format, and all tags are lowercase. All criteria met. |
| 2026-08-14T01:38:17+00:00 | muse-glimmer-30b-q4_k_xl-local | json_strict | 4 | 5 | ✓ | 33.6s | — | Response is valid JSON with 4 objects, all required keys present, all SKUs are uppercase 6 characters, all dates are in YYYY-MM-DD format, and all tags are lowercase. |
| 2026-08-14T01:37:31+00:00 | muse-glimmer-30b-q4_k_xl-local | json_strict | 3 | 5 | ✓ | 33.8s | — | All criteria are met: valid JSON with 4 objects, all keys present, SKUs are uppercase 6 chars, dates are YYYY-MM-DD format, and all tags are lowercase. |
| 2026-08-14T01:36:53+00:00 | muse-glimmer-30b-q4_k_xl-local | json_strict | 2 | 5 | ✓ | 33.8s | — | Response is valid JSON with exactly 4 objects, all containing required keys (sku, name, price_usd, in_stock, tags, added_on). All SKUs are uppercase 6 characters, all dates follow YYYY-MM-DD format, a |
| 2026-08-14T01:36:20+00:00 | muse-glimmer-30b-q4_k_xl-local | json_strict | 1 | 5 | ✓ | 29.8s | — | Response is valid JSON with exactly 4 objects, all containing consistent keys (sku, name, price_usd, in_stock, tags, added_on). All SKUs are uppercase 6-character codes (CMG001, AUR042, TKL100, BOK512 |
| 2026-08-14T01:35:04+00:00 | muse-glimmer-30b-q4_k_xl-local | code_debug | 5 | 5 | ✓ | 69.4s | — | Correctly identifies line 9 as buggy, accurately explains that the sort key uses message strings instead of timestamps, and provides a valid fix that sorts by timestamp. |
| 2026-08-14T01:33:48+00:00 | muse-glimmer-30b-q4_k_xl-local | code_debug | 4 | 5 | ✓ | 69.3s | — | The model correctly identifies line 9 as buggy, explains that the sort key is the message string instead of the timestamp (causing alphabetical sorting), and provides a valid fix that sorts by timesta |
| 2026-08-14T01:32:29+00:00 | muse-glimmer-30b-q4_k_xl-local | code_debug | 3 | 5 | ✓ | 69.5s | — | The model correctly identifies line 9 as buggy, accurately explains that sorting by message string instead of timestamp causes alphabetical rather than chronological ordering, and provides a valid fix |
| 2026-08-14T01:31:15+00:00 | muse-glimmer-30b-q4_k_xl-local | code_debug | 2 | 5 | ✓ | 69.5s | — | The model correctly identifies line 9 as buggy, accurately explains that the sort key is the message string rather than the timestamp (causing alphabetical instead of chronological sorting), and provi |
| 2026-08-14T01:29:42+00:00 | muse-glimmer-30b-q4_k_xl-local | code_debug | 1 | 5 | ✓ | 89.4s | — | The response correctly identifies line 9 as the buggy line, accurately explains that the sort key is the message string instead of the timestamp causing alphabetical instead of chronological ordering, |
| 2026-08-13T20:37:55+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 414.9s | $0.1517 | gpqa_diamond |
| 2026-08-13T20:31:00+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 532.5s | $0.1565 | gpqa_diamond |
| 2026-08-13T20:22:08+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 49.8s | $0.0201 | gpqa_diamond |
| 2026-08-13T20:19:42+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 129.7s | $0.0057 | ifeval |
| 2026-08-13T20:19:41+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 17.0s | $0.0069 | ifeval |
| 2026-08-13T20:19:25+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 53.4s | $0.0182 | ifeval |
| 2026-08-13T20:18:31+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 21.9s | $0.0083 | ifeval |
| 2026-08-13T20:18:09+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 8.2s | $0.0029 | ifeval |
| 2026-08-13T20:18:01+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 25.3s | $0.0066 | ifeval |
| 2026-08-13T20:17:36+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 20.7s | $0.0084 | ifeval |
| 2026-08-13T20:17:33+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 22.2s | $0.0089 | ifeval |
| 2026-08-13T20:17:15+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 5.6s | $0.0026 | ifeval |
| 2026-08-13T20:17:11+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 135.4s | $0.0482 | ifeval |
| 2026-08-13T20:17:09+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 32.5s | $0.0101 | ifeval |
| 2026-08-13T20:16:37+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 14.6s | $0.0058 | ifeval |
| 2026-08-13T20:16:22+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 24.2s | $0.0090 | ifeval |
| 2026-08-13T20:15:58+00:00 | grok-4.6-direct | bench:ifeval | 1 | 4 | ✗ | 9.3s | $0.0032 | ifeval |
| 2026-08-13T20:15:49+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 14.6s | $0.0049 | ifeval |
| 2026-08-13T20:15:34+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 35.2s | $0.0094 | ifeval |
| 2026-08-13T20:14:59+00:00 | grok-4.6-direct | bench:ifeval | 1 | 4 | ✗ | 25.9s | $0.0094 | ifeval |
| 2026-08-13T20:14:55+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 15.6s | $0.0054 | ifeval |
| 2026-08-13T20:14:40+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 40.0s | $0.0157 | ifeval |
| 2026-08-13T20:14:33+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 162.1s | $0.0592 | ifeval |
| 2026-08-13T20:14:00+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 23.1s | $0.0070 | ifeval |
| 2026-08-13T20:13:37+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 7.2s | $0.0032 | ifeval |
| 2026-08-13T20:13:29+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 19.2s | $0.0076 | ifeval |
| 2026-08-13T20:13:10+00:00 | grok-4.6-direct | bench:ifeval | 1 | 0 | ✗ | 31.9s | $0.0136 | ifeval |
| 2026-08-13T20:12:38+00:00 | grok-4.6-direct | bench:ifeval | 1 | 5 | ✓ | 34.2s | $0.0115 | ifeval |
| 2026-08-13T20:12:04+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 108.2s | $0.0342 | gpqa_diamond |
| 2026-08-13T20:11:51+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 54.4s | $0.0207 | gpqa_diamond |
| 2026-08-13T20:10:57+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 62.5s | $0.0232 | gpqa_diamond |
| 2026-08-13T20:10:16+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 23.6s | $0.0103 | gpqa_diamond |
| 2026-08-13T20:09:52+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 20.9s | $0.0107 | gpqa_diamond |
| 2026-08-13T20:09:31+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 17.7s | $0.0084 | gpqa_diamond |
| 2026-08-13T20:09:14+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 70.2s | $0.0248 | gpqa_diamond |
| 2026-08-13T20:08:03+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 0 | ✗ | 29.1s | $0.0140 | gpqa_diamond |
| 2026-08-13T20:07:34+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 12.9s | $0.0073 | gpqa_diamond |
| 2026-08-13T20:05:54+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 14.8s | $0.0080 | gpqa_diamond |
| 2026-08-13T20:05:39+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 84.3s | $0.0265 | gpqa_diamond |
| 2026-08-13T20:04:47+00:00 | grok-4.6-direct | code_debug | 5 | 5 | ✓ | 40.9s | $0.0145 | The response correctly identifies line 9 as buggy, explains the root cause (sort key is message string, not timestamp), and provides a correct fix that sorts by the timestamp of the corresponding log |
| 2026-08-13T20:04:15+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 87.5s | $0.0287 | gpqa_diamond |
| 2026-08-13T20:03:21+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 20.3s | $0.0112 | gpqa_diamond |
| 2026-08-13T20:03:01+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 0 | ✗ | 236.8s | $0.0626 | gpqa_diamond |
| 2026-08-13T20:02:47+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 14.5s | $0.0078 | gpqa_diamond |
| 2026-08-13T20:00:16+00:00 | grok-4.6-direct | agentic_tool_use | 5 | 4 | ✓ | 26.1s | $0.0126 | Valid JSON, plan has 5 items in correct order, tool_calls has 5 items with plausible tool names/args, expected_outcome mentions 'ok', and 3 realistic failure_modes are given. However, the write_file f |
| 2026-08-13T20:00:07+00:00 | grok-4.6-direct | agentic_tool_use | 4 | 5 | ✓ | 22.6s | $0.0124 | Response is valid JSON with plan in correct order (read data.py → write helpers.py → edit data.py → verify), exactly 4 tool_calls with plausible args and reasonable tool names, expected_outcome explic |
| 2026-08-13T19:59:17+00:00 | grok-4.6-direct | refactor_existing_code | 5 | 3 | ✗ | 58.9s | $0.0218 | Smells are correctly identified (3 issues: bare except, ==True comparison, manual dedup). However, the refactored code still retains `== True` (one of the identified smells was not fixed), does not us |
| 2026-08-13T19:59:04+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 32.5s | $0.0177 | gpqa_diamond |
| 2026-08-13T19:59:02+00:00 | grok-4.6-direct | refactor_existing_code | 4 | 3 | ✗ | 65.5s | $0.0245 | Smells correctly identify 3 real issues, and behavior_preserved/improvements are solid. However, the refactored code still uses `!= True` and `== ''` (the very smells it identified), lacks list compre |
| 2026-08-13T19:58:48+00:00 | grok-4.6-direct | agentic_prompt | 5 | 5 | ✓ | 28.7s | $0.0099 | Response is valid JSON with 5 plan items (within 4-6), 4 risks items (within 2-4), a concrete and immediately runnable first_action shell command, and a single-line dry_run_command. All stated criteri |
| 2026-08-13T19:58:33+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 16.8s | $0.0101 | gpqa_diamond |
| 2026-08-13T19:58:32+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 14.3s | $0.0104 | gpqa_diamond |
| 2026-08-13T19:58:26+00:00 | grok-4.6-direct | agentic_prompt | 4 | 5 | ✓ | 34.8s | $0.0117 | Valid JSON with plan of 5 items (within 4-6 range), risks of 4 items (within 2-4 range), first_action is a concrete, immediately runnable multi-command shell snippet inspecting the source directory, a |
| 2026-08-13T19:58:19+00:00 | grok-4.6-direct | reasoning_multistep | 5 | 5 | ✓ | 7.5s | $0.0032 | All 4 steps are clearly shown and correct: distance covered in first 30 min (30 km), remaining distance (210 km), combined closing speed (140 km/h), and time to meet (1.5 h after 9:30 AM). Final answe |
| 2026-08-13T19:58:17+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 16.5s | $0.0096 | gpqa_diamond |
| 2026-08-13T19:58:16+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 12.1s | $0.0070 | gpqa_diamond |
| 2026-08-13T19:58:11+00:00 | grok-4.6-direct | reasoning_multistep | 4 | 5 | ✓ | 7.8s | $0.0033 | All four steps are shown correctly with proper calculations, and the final answer matches the expected 11:00 AM. |
| 2026-08-13T19:58:04+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 20.7s | $0.0101 | gpqa_diamond |
| 2026-08-13T19:58:01+00:00 | grok-4.6-direct | bench:gpqa_diamond | 1 | 5 | ✓ | 17.7s | $0.0078 | gpqa_diamond |
| 2026-08-13T19:56:11+00:00 | grok-4.6-direct | creative_write | 5 | 5 | ✓ | 156.8s | $0.0624 | Meets all criteria: ~315 words, first-person present tense, named protagonist Finn Calder, contains lamp/fog/circuit, ends on single line of dialogue ('Clara. The light is on.'), and uses only 'only' |
| 2026-08-13T19:56:04+00:00 | grok-4.6-direct | creative_write | 4 | 0 | ✗ | 126.8s | $0.0498 | judge_parse_error: |
| 2026-08-13T19:55:47+00:00 | grok-4.6-direct | summarize | 5 | 5 | ✓ | 23.4s | $0.0116 | Exactly 5 bullets, all under 20 words (12, 14, 16, 12, 12), each captures a distinct required point (decision, problem, tradeoff, metrics, deferred item) with concrete figures (12+/week, 300ms, 10min, |
| 2026-08-13T19:55:35+00:00 | grok-4.6-direct | summarize | 4 | 4 | ✗ | 29.1s | $0.0133 | Format is correct: exactly 5 bullets, each well under 20 words (13–15 words each), with no fluff. Captures decision, pain source, cost tradeoff, specific SLO targets, and service mesh question with co |
| 2026-08-13T19:55:20+00:00 | grok-4.6-direct | code_gen_long | 5 | 5 | ✓ | 27.0s | $0.0167 | The class compiles, has __init__, try_acquire, and time_until_available with correct signatures and type hints, uses threading.Lock properly (both lock acquisition via 'with self._lock' and created vi |
| 2026-08-13T19:55:16+00:00 | grok-4.6-direct | code_gen_long | 4 | 5 | ✓ | 18.6s | $0.0117 | The response meets all stated criteria: the class compiles (valid Python syntax), defines __init__, try_acquire, and time_until_available with proper type hints, uses threading.Lock for thread safety |
| 2026-08-13T19:55:04+00:00 | grok-4.6-direct | json_strict | 5 | 5 | ✓ | 15.9s | $0.0093 | Response is valid JSON containing an array of 4 objects. All objects have consistent keys (sku, name, price_usd, in_stock, tags, added_on). SKUs are all uppercase 6-char (CMG001, AUR042, TKL100, BOK51 |
| 2026-08-13T19:55:02+00:00 | grok-4.6-direct | json_strict | 4 | 5 | ✓ | 14.3s | $0.0090 | Response is valid JSON array with exactly 4 objects, each containing all required keys (sku, name, price_usd, in_stock, tags, added_on). All SKUs are uppercase 6 characters (CMG001, AUR042, TKL100, BO |
| 2026-08-13T19:54:21+00:00 | grok-4.6-direct | code_debug | 4 | 5 | ✓ | 39.6s | $0.0162 | Correctly identifies line 9 as buggy, explains that sorting by message string causes alphabetical instead of chronological order, and provides a valid fix that sorts by timestamp. |
| 2026-08-13T19:44:15+00:00 | grok-4.6-direct | agentic_tool_use | 3 | 5 | ✓ | 29.4s | $0.0142 | Response is valid JSON with a 5-item plan in the correct order (read data.py → write helpers.py → edit data.py → verify, plus conditional re-read). tool_calls contains 4 items with correct tool names |
| 2026-08-13T19:44:03+00:00 | grok-4.6-direct | agentic_tool_use | 2 | 4 | ✓ | 42.3s | $0.0204 | Valid JSON, 5-step plan in correct order (read→write→edit→verify→re-read), 5 tool_calls with plausible names (read_file, write_file, run_shell), expected_outcome mentions 'ok' print, and 3 real failur |
| 2026-08-13T19:43:45+00:00 | grok-4.6-direct | agentic_tool_use | 1 | 5 | ✓ | 26.9s | $0.0136 | Response is valid JSON, plan has exactly 4 items in the correct order (read data.py → write helpers.py → edit data.py → verify), tool_calls has 4 items with plausible tool names (read_file, write_file |
| 2026-08-13T19:42:54+00:00 | grok-4.6-direct | refactor_existing_code | 3 | 3 | ✗ | 62.7s | $0.0240 | The response is valid JSON, correctly identifies 3 smells (bare except, ==True/!=None/!='' comparisons, manual dedup loop), and provides accurate behavior_preserved and 3 concrete improvements. Howeve |
| 2026-08-13T19:42:35+00:00 | grok-4.6-direct | refactor_existing_code | 2 | 3 | ✗ | 54.2s | $0.0228 | Valid JSON, smells correctly identify bare except and manual dedup, behavior_preserved is mostly accurate, and 3 concrete improvements are listed. However, the refactored code still uses `!= True` (a |
| 2026-08-13T19:41:58+00:00 | grok-4.6-direct | refactor_existing_code | 1 | 3 | ✗ | 52.2s | $0.0193 | Valid JSON with smells correctly identifying bare except, ==True comparison, and manual dedup. However, the refactored function does NOT use a list comprehension as required by the criteria, and still |
| 2026-08-13T19:41:53+00:00 | grok-4.6-direct | agentic_prompt | 3 | 5 | ✓ | 39.4s | $0.0147 | All criteria met: valid JSON, plan has 5 items (within 4-6), risks has 4 items (within 2-4), first_action is a concrete immediately runnable shell pipeline for inspection, and dry_run_command is a sin |
| 2026-08-13T19:41:30+00:00 | grok-4.6-direct | agentic_prompt | 2 | 5 | ✓ | 24.0s | $0.0086 | Valid JSON; plan has 5 items (within 4-6), risks has 4 items (within 2-4), first_action contains concrete immediately-runnable shell commands (find/ls/command -v checks), and dry_run_command is a sing |
| 2026-08-13T19:40:59+00:00 | grok-4.6-direct | agentic_prompt | 1 | 5 | ✓ | 27.3s | $0.0099 | Valid JSON; plan has exactly 5 items (within 4-6); risks has exactly 4 items (within 2-4); first_action is concrete and immediately runnable as a single shell line; dry_run_command is a single shell l |
| 2026-08-13T19:40:49+00:00 | grok-4.6-direct | reasoning_multistep | 3 | 5 | ✓ | 7.9s | $0.0035 | All 4 steps are shown correctly: Train A covers 30 km in 0.5h, remaining distance is 210 km, closing speed is 140 km/h, and 210/140 = 1.5h gives meeting time of 11:00 AM. Final answer matches expected |
| 2026-08-13T19:40:36+00:00 | grok-4.6-direct | reasoning_multistep | 2 | 5 | ✓ | 11.3s | $0.0046 | All 4 steps are shown correctly: Train A covers 30 km, remaining separation is 210 km, closing speed is 140 km/h, and time to close is 1.5 h, arriving at the expected final answer of 11:00 AM. |
| 2026-08-13T19:40:22+00:00 | grok-4.6-direct | reasoning_multistep | 1 | 5 | ✓ | 10.2s | $0.0046 | All 4 steps are clearly shown with correct arithmetic (30 km, 210 km, 140 km/h, 1.5 h), and the final answer of 11:00 AM exactly matches the expected answer. |
| 2026-08-13T19:40:06+00:00 | grok-4.6-direct | creative_write | 3 | 0 | ✗ | 91.0s | $0.0375 | judge_parse_error: Let me evaluate the response against the criteria: 1. **Word count 270-330**: Let me count. "I check the lamp at dusk. The wick holds. Below the gallery the Atlantic works the roc |
| 2026-08-13T19:38:13+00:00 | grok-4.6-direct | creative_write | 2 | 0 | ✗ | 108.6s | $0.0423 | judge_parse_error: Let me evaluate the response against the criteria: 1. **Word count: 270-330 words**: Let me count the words. "I sign the log: Thomas Hale, keeper, Station 7." - 10 "The lamp holds. |
| 2026-08-13T19:38:09+00:00 | grok-4.6-direct | creative_write | 1 | 5 | ✓ | 127.2s | $0.0520 | Word count is ~310 (within 270-330). Uses first-person present throughout ('I keep,' 'I check,' 'I key,' 'I say'). Named protagonist 'Harlan Voss' is established. Contains 'lamp,' 'fog,' and 'circuit' |
| 2026-08-13T19:37:53+00:00 | grok-4.6-direct | summarize | 3 | 4 | ✓ | 15.1s | $0.0067 | Exactly 5 bullets, each well under 20 words, covering recommendation, problem, cost/timeline, success metrics, and open question. Format is perfect, but factual accuracy cannot be fully verified again |
| 2026-08-13T19:37:39+00:00 | grok-4.6-direct | summarize | 2 | 4 | ✓ | 24.9s | $0.0115 | Meets format criteria perfectly: exactly 5 bullets, all ≤20 words (longest is 14), no fluff. Content appears to capture decision (modular monolith), rationale (cross-team coupling), trade-offs ($1.6M |
| 2026-08-13T19:37:22+00:00 | grok-4.6-direct | summarize | 1 | 4 | ✓ | 26.6s | $0.0113 | Format is excellent: exactly 5 bullets, each well under 20 words, no fluff. Content covers a recommendation, problem diagnosis, trade-off, success metrics, and an open question — plausible coverage of |
| 2026-08-13T19:37:13+00:00 | grok-4.6-direct | code_gen_long | 3 | 5 | ✓ | 22.4s | $0.0120 | Class compiles correctly, includes all expected methods (__init__, try_acquire, time_until_available) with proper signatures and type hints, uses threading.Lock for thread safety in both public method |
| 2026-08-13T19:36:49+00:00 | grok-4.6-direct | code_gen_long | 2 | 5 | ✓ | 30.7s | $0.0156 | Class compiles cleanly, contains all expected methods (__init__, try_acquire, time_until_available) with correct signatures and type hints, uses threading.Lock for thread safety in both public methods |
| 2026-08-13T19:36:44+00:00 | grok-4.6-direct | code_gen_long | 1 | 5 | ✓ | 21.1s | $0.0125 | The class compiles, contains all 3 required methods (__init__, try_acquire, time_until_available) with correct signatures and type hints, uses threading.Lock for thread safety (self._lock with 'with' |
| 2026-08-13T19:36:30+00:00 | grok-4.6-direct | json_strict | 3 | 5 | ✓ | 16.6s | $0.0104 | Response is valid JSON containing exactly 4 objects with all required keys (sku, name, price_usd, in_stock, tags, added_on). All SKUs (CMG001, AUR042, TKL100, BOK512) are 6 characters and uppercase. A |
| 2026-08-13T19:36:07+00:00 | grok-4.6-direct | json_strict | 2 | 5 | ✓ | 18.1s | $0.0102 | Response is valid JSON with 4 objects, each containing all required keys. All SKUs are uppercase 6 characters (CMG001, AUR042, TKL100, BOK512). All dates follow YYYY-MM-DD format. All tags are lowerca |
| 2026-08-13T19:35:50+00:00 | grok-4.6-direct | json_strict | 1 | 5 | ✓ | 14.6s | $0.0082 | Valid JSON array with exactly 4 objects, all having required keys (sku, name, price_usd, in_stock, tags, added_on). All SKUs are uppercase 6 characters (CMG001, AUR042, TKL100, BOK512), all dates are |
| 2026-08-13T19:35:47+00:00 | grok-4.6-direct | code_debug | 3 | 5 | ✓ | 54.0s | $0.0212 | The model correctly identifies line 9 as the buggy line, explains that sorting by message string causes alphabetical instead of chronological order, and provides a valid fix that sorts by timestamp by |
| 2026-08-13T19:34:54+00:00 | grok-4.6-direct | code_debug | 2 | 5 | ✓ | 49.1s | $0.0202 | Buggy line correctly identified as 9, root cause accurately explains the sort key is the message string leading to alphabetical instead of chronological ordering, and the fix correctly sorts by timest |
| 2026-08-13T19:34:54+00:00 | grok-4.6-direct | code_debug | 1 | 5 | ✓ | 49.8s | $0.0226 | All three criteria are met: buggy_line correctly identifies line 9, root_cause correctly explains that the sort key is the message string instead of the timestamp causing alphabetical rather than chro |
| 2026-08-07T18:45:31+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | agentic_tool_use | 3 | 5 | ✓ | 4.5s | — | Valid JSON, plan has 4 items in the correct order (read data.py → write helpers.py → edit data.py → verify), tool_calls has 5 items with correct tool names and plausible args, expected_outcome mention |
| 2026-08-07T18:45:22+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | agentic_tool_use | 2 | 5 | ✓ | 4.5s | — | Valid JSON, plan has exactly 4 items in correct order (read → write helpers → edit → verify), tool_calls has 5 items with correct tool names (read_file, write_file, run_shell) and plausible args, expe |
| 2026-08-07T18:45:12+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | agentic_tool_use | 1 | 4 | ✓ | 4.6s | — | Valid JSON, plan has 4 items in correct order, 5 tool calls with plausible tool names/args, expected_outcome mentions 'ok' print, and 3 realistic failure_modes. Minor issues: redundant duplicate read_ |
| 2026-08-07T18:45:00+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | refactor_existing_code | 3 | 4 | ✓ | 2.0s | — | Valid JSON with 3 identified smells, meaningful refactoring using set for dedup and .get() for safe access, and accurate behavior preservation explanation. Minor deductions: refactored code uses a tra |
| 2026-08-07T18:44:54+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | refactor_existing_code | 2 | 3 | — | 2.0s | — | Valid JSON with identified smells and improvements listed. However, the refactored function does not use a list comprehension (criterion explicitly required), does not demonstrate proper exception han |
| 2026-08-07T18:44:47+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | refactor_existing_code | 1 | 3 | ✗ | 2.1s | — | Valid JSON, correctly identifies 3 smells, and behavior preservation is accurate. However, the refactored function fails the list comprehension criterion (uses explicit for loop instead), the exceptio |
| 2026-08-07T18:44:40+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | agentic_prompt | 3 | 3 | ✗ | 1.8s | — | JSON is valid with correct counts (5 plan items, 2 risks), but there are notable inconsistencies: the plan describes creating a Python script while the dry_run_command references a bash script (/tmp/o |
| 2026-08-07T18:44:36+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | agentic_prompt | 2 | 5 | ✓ | 1.8s | — | All criteria are met: valid JSON, plan has 5 items (4-6 range), risks has 2 items (2-4 range), first_action is a concrete runnable shell command, and dry_run_command is a single shell line. |
| 2026-08-07T18:44:28+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | agentic_prompt | 1 | 4 | ✓ | 1.8s | — | Valid JSON, plan has 5 items (within 4-6), risks has 2 items (within 2-4), dry_run_command is a single shell line. However, first_action is trivially weak (just prints a message) and dry_run_command r |
| 2026-08-07T18:44:21+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | reasoning_multistep | 3 | 5 | ✓ | 4.0s | — | All four steps are correctly shown with accurate calculations (30 km head start, 210 km remaining, 140 km/h combined speed, 1.5 hours), and the final answer of 11:00 AM matches the expected answer. |
| 2026-08-07T18:44:14+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | reasoning_multistep | 2 | 5 | ✓ | 4.0s | — | All four steps are clearly shown with correct calculations: 30 km head start, 210 km remaining, 140 km/h combined speed, 1.5 hours to meet, arriving at 11:00 AM. |
| 2026-08-07T18:44:09+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | reasoning_multistep | 1 | 5 | ✓ | 4.0s | — | All four steps are correctly shown with proper calculations, and the final answer is 11:00 AM matching the expected answer. |
| 2026-08-07T18:43:59+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | creative_write | 3 | 2 | ✗ | 3.7s | — | Word count is within range (~325), first-person present tense is used, named protagonist (Arthur), and all three keywords (lamp, fog, circuit) appear. However, the piece fails on two key criteria: it |
| 2026-08-07T18:43:37+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | creative_write | 2 | 0 | ✗ | 3.7s | — | judge_parse_error: {"score": 2, "reason": "Fails two criteria: the ending is narration ('I close my eyes...') rather than a single line of dialogue, and there are ~7 -ly adverbs (barely, unmistakabl |
| 2026-08-07T18:43:09+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | creative_write | 1 | 2 | ✗ | 3.8s | — | Word count (~308), first-person present, named protagonist (Arthur), lamp/fog/circuit usage, and -ly adverb count (4) are all met, but the piece fails the explicit ending requirement: it concludes wit |
| 2026-08-07T18:42:56+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | summarize | 3 | 5 | ✓ | 1.1s | — | Exactly 5 bullets, each under 20 words (longest is 16), no fluff, clear labels covering Decision/Reason/Trade-off/Metric/Open question with specific details (modular monolith, p99 latency 300ms, servi |
| 2026-08-07T18:42:49+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | summarize | 2 | 5 | ✓ | 1.1s | — | Exactly 5 bullets, each ≤20 words, covering Decision/Reason/Trade-off/Metric/Open Question with no fluff or factual drift. Clean ADR-style format that captures all required points. |
| 2026-08-07T18:42:44+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | summarize | 1 | 5 | ✓ | 1.2s | — | Exactly 5 bullets provided, each under 20 words, covering decision/reason/trade-off/metric/open question with no fluff or factual drift. |
| 2026-08-07T18:42:36+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | code_gen_long | 3 | 5 | ✓ | 4.6s | — | The code meets all criteria: class compiles, has all 3 methods (__init__, try_acquire, time_until_available) with correct signatures and type hints, uses threading.Lock for thread safety, and includes |
| 2026-08-07T18:42:29+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | code_gen_long | 2 | 5 | ✓ | 4.7s | — | The code compiles, contains all 3 methods (__init__, try_acquire, time_until_available) with correct signatures and type hints, uses threading.Lock for thread safety, and includes a __main__ demo bloc |
| 2026-08-07T18:42:21+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | code_gen_long | 1 | 5 | ✓ | 4.8s | — | The code meets all criteria: it compiles, has all required methods (__init__, try_acquire, time_until_available) with correct signatures, uses threading.Lock with context managers, has comprehensive t |
| 2026-08-07T18:42:13+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | json_strict | 3 | 5 | ✓ | 3.0s | — | Valid JSON array with exactly 4 objects, all containing the required keys. All SKUs are uppercase 6-character alphanumeric codes (CMG001, AUR042, TKL100, BOK512). All dates follow YYYY-MM-DD format. A |
| 2026-08-07T18:42:07+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | json_strict | 2 | 5 | ✓ | 3.0s | — | Valid JSON array with exactly 4 objects. All required keys present (sku, name, price_usd, in_stock, tags, added_on). All SKUs are uppercase 6 chars (CMG001, AUR042, TKL100, BOK512). All dates are in Y |
| 2026-08-07T18:42:01+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | json_strict | 1 | 5 | ✓ | 3.2s | — | Response is valid JSON with exactly 4 objects, all required keys present, SKUs are uppercase 6 characters, dates follow YYYY-MM-DD format, and all tags are lowercase. |
| 2026-08-07T18:41:48+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | code_debug | 3 | 2 | ✗ | 0.8s | — | Root cause is correctly identified (sort key is message string instead of timestamp), but the fix `errors.sort(key=lambda e: e['timestamp'])` is broken because `errors` is a list of message strings, n |
| 2026-08-07T18:41:41+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | code_debug | 2 | 2 | ✗ | 0.8s | — | The root cause is correctly identified (sort key is message instead of timestamp), but the line number is off by one (10 vs expected 9), and critically the fix is wrong: `errors` is a list of message |
| 2026-08-07T18:41:29+00:00 | ollama-local/ds-v4pro-qwen35-9b-q4 | code_debug | 1 | 3 | ✓ | 3.6s | — | Root cause correctly identifies the sort key issue (message string vs timestamp), and the fix concept (sort by timestamp) is correct. However, buggy_line is 10 instead of expected 9, and the fix `erro |
| 2026-08-07T18:40:39+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_tool_use | 3 | 5 | ✓ | 3.5s | — | Response is valid JSON with a 4-item plan in correct order (read → write helpers → edit data → verify), 4 plausible tool_calls with reasonable tool names and args, expected_outcome correctly describes |
| 2026-08-07T18:40:29+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_tool_use | 2 | 5 | ✓ | 3.5s | — | Valid JSON, plan has exactly 4 items in the correct order (read data.py → write helpers.py → edit data.py → verify), tool_calls has 4 items with plausible tools and args, expected_outcome correctly de |
| 2026-08-07T18:40:20+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_tool_use | 1 | 5 | ✓ | 3.6s | — | Valid JSON with 4-item plan in correct order (read → write helpers → write data → verify), 4 tool_calls with correct tool names and plausible args, expected_outcome describes 'ok' print, and 3 real fa |
| 2026-08-07T18:40:08+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | refactor_existing_code | 3 | 4 | ✓ | 1.8s | — | Valid JSON with 2-3 real smells identified (bare except, god method). Refactored function uses list comprehension, .get() with default, and set for dedup - but omits exception handling entirely rather |
| 2026-08-07T18:40:01+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | refactor_existing_code | 2 | 3 | ✗ | 1.8s | — | Valid JSON and refactored code meaningfully cleans up the original (list comprehension, .get() with default, set-based dedup). However, the listed smells are generic (DRY, God Method, bare except) and |
| 2026-08-07T18:39:54+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | refactor_existing_code | 1 | 3 | ✗ | 1.9s | — | Response is valid JSON with reasonable smells (bare except), but misses identifying ==True comparison and !=None !='' check explicitly. Refactored code uses list comprehension and .get() with default, |
| 2026-08-07T18:39:46+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_prompt | 3 | 5 | ✓ | 1.4s | — | All criteria are satisfied: valid JSON, plan has 4 items (within 4-6 range), risks has 3 items (within 2-4 range), first_action is a concrete runnable command, and dry_run_command is a single shell li |
| 2026-08-07T18:39:43+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_prompt | 2 | 5 | ✓ | 1.4s | — | All criteria met: valid JSON, plan has 4 items (within 4-6 range), risks has 3 items (within 2-4 range), first_action is concrete and runnable, and dry_run_command is a single shell line. |
| 2026-08-07T18:39:39+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_prompt | 1 | 5 | ✓ | 1.4s | — | All criteria are satisfied: valid JSON, plan has 4 items (within 4-6 range), risks has 3 items (within 2-4 range), first_action is a concrete and immediately runnable shell command, and dry_run_comman |
| 2026-08-07T18:39:34+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | reasoning_multistep | 3 | 5 | ✓ | 3.6s | — | All 4 steps are shown correctly: head start distance (30 km), remaining gap (210 km), closing speed (140 km/h), and time to meet (1.5 hours). Final answer 11:00 AM matches the expected answer exactly. |
| 2026-08-07T18:39:26+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | reasoning_multistep | 2 | 5 | ✓ | 3.6s | — | All 4 steps are shown clearly and computed correctly. Train A's head start (30 km), remaining distance (210 km), combined speed (140 km/h), and meeting time (1.5 hours after 9:30 AM) are all accurate, |
| 2026-08-07T18:39:21+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | reasoning_multistep | 1 | 5 | ✓ | 3.6s | — | All four steps are shown correctly with accurate calculations, and the final answer of 11:00 AM matches the expected answer. |
| 2026-08-07T18:38:58+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | creative_write | 3 | 2 | ✗ | 3.7s | — | The response fails two quantitative criteria: word count is ~481 (max 330), and there are 5 -ly adverbs (slowly, faithfully, unmistakably, intently, faintly) where ≤4 is allowed. Creative quality and |
| 2026-08-07T18:38:30+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | creative_write | 2 | 2 | ✗ | 3.8s | — | Word count is ~450, far exceeding the 270-330 limit. Contains 7 -ly adverbs (slowly, faithfully, unmistakably, perfectly, intently, faintly, softly), well above the ≤4 limit. First-person present, nam |
| 2026-08-07T18:38:17+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | creative_write | 1 | 2 | ✗ | 3.8s | — | The response exceeds the word count limit significantly (~405 words vs 330 max) and contains far too many -ly adverbs (~10 vs max of 4). It does meet criteria for first-person present, named protagoni |
| 2026-08-07T18:38:06+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | summarize | 3 | 4 | ✓ | 0.9s | — | Meets all format criteria: exactly 5 bullets, each well under 20 words, clearly labeled (Decision, Reason, Trade-off, Metric, Open Question), concise with no fluff. Slight uncertainty about whether al |
| 2026-08-07T18:37:58+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | summarize | 2 | 4 | ✓ | 0.9s | — | Format is correct: exactly 5 bullets, each well under 20 words, with no fluff. Structure (Decision, Reason, Trade-off, Metric, Open Question) is coherent and likely covers the 5 required points, but w |
| 2026-08-07T18:37:51+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | summarize | 1 | 4 | ✓ | 0.9s | — | Format is perfect: exactly 5 bullets, each under 20 words, concise and well-categorized (Decision/Reason/Trade-off/Metric/Open Question). Cannot fully verify 'all 5 required points captured' or 'no fa |
| 2026-08-07T18:37:40+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | code_gen_long | 3 | 4 | ✓ | 3.9s | — | All stated criteria are met: class compiles, has 3 methods with correct signatures (try_acquire, time_until_available, __init__), uses threading.Lock, has type hints, and includes a __main__ demo. How |
| 2026-08-07T18:37:21+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | code_gen_long | 2 | 3 | ✓ | 3.9s | — | Structure meets all stated criteria (compiles, 3 methods with correct signatures, threading.Lock, type hints, __main__ demo), but the core logic is fundamentally broken: `_last_update_time()` returns |
| 2026-08-07T18:36:56+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | code_gen_long | 1 | 3 | ✓ | 4.0s | — | All structural criteria are met (compiles, has __init__/try_acquire/time_until_available with correct signatures, uses threading.Lock, has type hints, has __main__ demo). However, there is a critical |
| 2026-08-07T18:36:46+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | json_strict | 3 | 5 | ✓ | 2.3s | — | All criteria met: valid JSON, 4 objects, all 6 keys present in each, SKUs are 6 uppercase chars, dates in YYYY-MM-DD format, and all tags are lowercase. |
| 2026-08-07T18:36:40+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | json_strict | 2 | 5 | ✓ | 2.2s | — | All criteria are met: valid JSON, exactly 4 objects, all required keys present, SKUs are 6 uppercase chars, dates in YYYY-MM-DD format, and tags are lowercase. |
| 2026-08-07T18:36:34+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | json_strict | 1 | 5 | ✓ | 2.4s | — | Valid JSON, 4 objects, all SKUs are uppercase 6 chars (CMG001, AUR042, TKL100, BOK512), all dates are YYYY-MM-DD format, and all tags are lowercase. All criteria met. |
| 2026-08-07T18:36:28+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | code_debug | 3 | 1 | ✗ | 0.6s | — | The root_cause correctly identifies the conceptual issue (sorting by message instead of timestamp), but the buggy_line is wrong (10 instead of 9) and the fix is broken: `m` is a message string, not a |
| 2026-08-07T18:36:22+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | code_debug | 2 | 2 | ✗ | 0.6s | — | Root cause is correctly identified (sort key is message string instead of timestamp), but buggy_line is off by one (10 vs 9), and the fix is syntactically/runtime broken: `m` is a message string, so ` |
| 2026-08-07T18:36:18+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | code_debug | 1 | 2 | ✗ | 0.8s | — | Root cause explanation is correct (sort key is message not timestamp), and fix concept is right (sort by timestamp), but buggy_line is 10 instead of expected 9, and the fix `m['timestamp']` would fail |
| 2026-08-07T18:33:35+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_tool_use | 3 | 0 | ✗ | 47.3s | — | The response is a reasoning monologue rather than the required JSON output. No JSON object with plan, tool_calls, expected_outcome, or failure_modes fields was produced. The output is truncated mid-se |
| 2026-08-07T18:32:43+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_tool_use | 2 | 0 | ✗ | 47.5s | — | The model produced only planning/reasoning text and never output the required JSON object with plan, tool_calls, expected_outcome, and failure_modes fields. The response is truncated mid-sentence and |
| 2026-08-07T18:31:52+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | agentic_tool_use | 1 | 0 | ✗ | 47.3s | — | The response is truncated mid-reasoning and never produces the required JSON object. None of the required fields (plan, tool_calls, expected_outcome, failure_modes) are present, so it completely fails |
| 2026-08-07T18:30:58+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | refactor_existing_code | 3 | 2 | ✗ | 47.3s | — | The response provides substantive analysis identifying all the expected smells (broad except, ==True, !=None/!='', manual dedup) and offers a clean refactored version using list comprehension and dict |
| 2026-08-07T18:30:01+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | refactor_existing_code | 2 | 1 | ✗ | 47.5s | — | The response is not valid JSON; it is a free-form text analysis. While it correctly identifies several real smells (broad except, ==True comparison, !=None !='' check) and provides a cleaner refactore |
| 2026-08-07T18:29:01+00:00 | ollama-local/ds-v4pro-qwen35-4b-q4 | refactor_existing_code | 1 | 1 | ✗ | 47.4s | — | The response is not valid JSON at all—it is free-form prose (truncated mid-sentence) with no structured fields like 'smells', 'refactored', 'behavior_preserved', or 'improvements'. While the analysis |
Showing first 200 trials. Full data: data/all_trials.json in the repo. Download as CSV (23566 rows, includes 1P-est per trial). Linked timestamps open the corresponding published full-response artifact.