This benchmark compares short stories written to the same constrained creative briefs. Separate evaluator models read matched stories in both orders and judge which is better. Their judgments produce relative scores for each writer model.
Higher ratings mean stronger estimated performance within this comparison set. Scores are centered at zero across all writer models in the comparison set.
The ranking combines judgments from earlier and newer evaluator models. It covers 67 writer models, 1,167 direct model pairings, 20,589 distinct story pairs, and 134,964 judgments. Each direct model pairing can include stories from multiple matched prompts. The chart shows 26 selected models; the table lists every rated model with its overall rank.
The shaded ranges show how much scores vary across 300 resamples of stories and evaluators (bootstrap). The table gives the 95% bootstrap intervals. Estimated win chance is a model's average predicted probability of beating another model in the full comparison set.
| Rank | Model | Comparison score | Estimated win chance | 95% bootstrap interval |
|---|---|---|---|---|
| 1 | GPT-6.2 Astra (high) | 4.302 | 96% | 4.223 to 4.379 |
| 2 | Claude Fable 5.2 (high) | 3.564 | 92% | 3.498 to 3.636 |
| 3 | Qwen 4 Max (high)⁂ | 3.548 | 92% | 3.469 to 3.625 |
| 4 | Claude Fable 5.1 (high) | 3.510 | 91% | 3.428 to 3.588 |
| 5 | Claude Opus 5 (xhigh) | 3.443 | 91% | 3.378 to 3.502 |
| 6 | Claude Opus 5.5 (high) | 3.355 | 90% | 3.286 to 3.425 |
| 7 | GPT-6 Astra (high) | 3.166 | 89% | 3.105 to 3.225 |
| 8 | Claude Opus 5 (high) | 3.049 | 88% | 2.945 to 3.148 |
| 9 | GLM-5.3 (max) | 2.613 | 84% | 2.529 to 2.695 |
| 10 | Claude Fable 5 (high)§ | 2.585 | 83% | 2.522 to 2.653 |
| 11 | Kimi K3.1 Pro (high) | 2.366 | 81% | 2.290 to 2.440 |
| 12 | GPT-5.6 Sol (xhigh) | 2.199 | 79% | 2.112 to 2.280 |
| 13 | GPT-5.5 (xhigh) | 2.169 | 79% | 2.112 to 2.241 |
| 14 | Kimi K3 | 2.104 | 78% | 2.034 to 2.173 |
| 15 | Tencent Hy4 (high) | 2.046 | 78% | 1.971 to 2.116 |
| 16 | GPT-5.6 Sol (high) | 2.011 | 77% | 1.935 to 2.083 |
| 17 | GPT-5.4 (medium) | 1.691 | 73% | 1.580 to 1.799 |
| 18 | GPT-5.4 (xhigh) | 1.685 | 73% | 1.580 to 1.789 |
| 19 | Gemini 4 Carbon (high) | 1.618 | 72% | 1.535 to 1.705 |
| 20 | Muse Wonder (high) | 1.609 | 72% | 1.537 to 1.681 |
| 21 | Claude Opus 4.7 (high)† | 1.496 | 71% | 1.389 to 1.602 |
| 22 | Claude Sonnet 4.6 Thinking 16K | 1.385 | 70% | 1.283 to 1.497 |
| 23 | Xiaomi MiMo V2.6 Pro (thinking)※ | 1.191 | 67% | 1.075 to 1.292 |
| 24 | Claude Opus 4.6 Thinking 16K | 0.852 | 62% | 0.679 to 1.041 |
| 25 | Claude Opus 4.8 (xhigh) | 0.500 | 57% | 0.398 to 0.606 |
| 26 | Muse Spark 1.1 (high) | 0.495 | 57% | 0.382 to 0.599 |
| 27 | Gemini 4 Argon (high) | 0.454 | 57% | 0.354 to 0.551 |
| 28 | Muse Spark 1.3 (high) | 0.234 | 54% | 0.151 to 0.309 |
| 29 | GLM-5.2 (max) | 0.182 | 53% | 0.086 to 0.276 |
| 30 | DeepSeek V4 Pro (high) | 0.140 | 52% | 0.054 to 0.229 |
| 31 | GPT-5.2 (medium) | 0.056 | 51% | -0.103 to 0.232 |
| 32 | Grok 4.7 (high) | 0.029 | 51% | -0.146 to 0.178 |
| 33 | Claude Opus 4.8 (high)‡ | -0.013 | 50% | -0.137 to 0.096 |
| 34 | Gemini 3.8 Flash (high) | -0.157 | 48% | -0.243 to -0.061 |
| 35 | Qwen 3.8 Max¶ | -0.174 | 48% | -0.269 to -0.075 |
| 36 | Step 5 Preview (high) | -0.185 | 47% | -0.285 to -0.094 |
| 37 | Kimi K2.6 | -0.191 | 47% | -0.315 to -0.084 |
| 38 | Muse Spark 1.2 (high) | -0.330 | 45% | -0.445 to -0.237 |
| 39 | MiniMax-M3 | -0.389 | 44% | -0.514 to -0.272 |
| 40 | Mistral Medium 3.1 | -0.660 | 40% | -0.784 to -0.520 |
| 41 | ByteDance Seed2.1 Pro | -0.667 | 40% | -0.784 to -0.565 |
| 42 | DeepSeek V4.1 Flash (high) | -0.797 | 38% | -0.882 to -0.701 |
| 43 | DeepSeek V4 Pro Preview | -0.814 | 38% | -0.927 to -0.693 |
| 44 | Qwen3.8-27B^ | -0.956 | 36% | -1.068 to -0.833 |
| 45 | Xiaomi MiMo V2.5 Pro | -0.957 | 36% | -1.079 to -0.819 |
| 46 | Qwen 3 Max Preview | -0.959 | 36% | -1.133 to -0.781 |
| 47 | Gemini 3.7 Flash (high) | -0.995 | 35% | -1.101 to -0.868 |
| 48 | Qwen 3.6 Max Preview | -1.206 | 32% | -1.317 to -1.083 |
| 49 | GLM-5.1 | -1.302 | 31% | -1.471 to -1.119 |
| 50 | Kimi K2.5 Thinking | -1.358 | 30% | -1.533 to -1.194 |
| 51 | Xiaomi MiMo V2 Pro | -1.488 | 29% | -1.678 to -1.265 |
| 52 | Baidu Ernie 5.1 | -1.498 | 28% | -1.612 to -1.384 |
| 53 | Mistral Large 4 (high) | -1.894 | 23% | -1.993 to -1.803 |
| 54 | Mistral Large 3 | -2.046 | 22% | -2.162 to -1.910 |
| 55 | Gemma 4 31B Reasoning | -2.058 | 21% | -2.124 to -1.987 |
| 56 | ByteDance Seed2.0 Pro | -2.180 | 20% | -2.294 to -2.077 |
| 57 | Gemini 3.5 Flash | -2.253 | 19% | -2.335 to -2.175 |
| 58 | Gemini 3.1 Pro Preview | -2.441 | 17% | -2.533 to -2.336 |
| 59 | Qwen 3.6 Plus | -2.553 | 16% | -2.772 to -2.358 |
| 60 | Mistral Medium 3.5 | -2.699 | 15% | -2.802 to -2.601 |
| 61 | Qwen 3.7 Max | -2.931 | 13% | -3.065 to -2.821 |
| 62 | DeepSeek V3.2 | -3.137 | 11% | -3.380 to -2.906 |
| 63 | Grok 4.6 (high) | -3.173 | 11% | -3.327 to -3.014 |
| 64 | GPT-OSS-120B | -3.290 | 10% | -3.398 to -3.197 |
| 65 | MiniMax-M2.7 | -4.026 | 6% | -4.157 to -3.889 |
| 66 | Grok 4.3 | -4.516 | 3% | -4.703 to -4.351 |
| 67 | Grok 4.5 (high) | -5.350 | 1% | -5.443 to -5.262 |
- † Claude Opus 4.7: 347 of 400 stories completed.
- ‡ Claude Opus 4.8 high: 399 of 400 stories completed.
- § Claude Fable 5 high: 395 of 400 stories completed.
- ¶ Qwen 3.8 Max: 398 of 400 stories completed. 20 of these required refusal retries (1 retry round).
- ^ Qwen3.8-27B: 389 of 400 stories completed. 46 of these required refusal retries (1 retry round).
- ※ MiMo V2.6 Pro: 375 of 400 stories completed.
- ⁂ Qwen 4 Max (high): 397 of 400 stories completed.
Striped bars and badges identify incomplete story sets. Quality comparisons concern completed stories. The latest evaluations returned 55,457/55,650 planned judgments; 193 unavailable judgments are excluded from the scores. The Opus 5.5 high versus Opus 5 high comparison covers 50 matched prompts, with 298/300 usable judgments.
Mistral Large 4 (high) completed all 400 prompts. Stories outside 600–800 words were regenerated from their original prompts, retaining the first response within range; 43 prompts required another draw, for 446 total generation calls. Its comparisons cover 26 opponents, including Mistral Large 3, with 20 matched stories per opponent and 3,112/3,120 usable judgments. Comparison coverage.
Muse Wonder (high) completed all 400 prompts. Stories outside 600–800 words were regenerated from their original prompts, retaining the first response within range; 2 prompts required another draw, for 402 total generation calls. Its comparisons cover 21 opponents, including Muse Spark 1.3 (high), with 20 matched stories per opponent and 2,467/2,520 usable judgments. Comparison coverage.
Gemini 4 Argon (high) completed all 400 prompts. Stories outside 600–800 words were regenerated from their original prompts, retaining the first response within range; 0 prompts required another draw, for 410 total generation calls. Its comparisons cover 22 opponents, including Gemini 3.8 Flash (high) and Muse Wonder (high), with 20 matched stories per opponent and 2,577/2,640 usable judgments. Comparison coverage. 10 provider failures returned no story text; a documented retry recovered 10 stories. Those failed calls are included in the generation total and excluded from the length-redraw count.
Kimi K3.1 Pro (high) completed all 400 prompts, accepting the first 600–800-word response to each prompt. There were 400 generation calls and 0 prompts needing another draw; accepted lengths range from 612 to 776 words. Its comparisons cover 22 opponents, including Kimi K3, with 20 matched stories per opponent and 50 against Kimi K3, for 2,818/2,820 usable judgments. Comparison coverage.
Claude Fable 5.2 (high), Tencent Hy4 (high), Step 5 Preview (high), Gemini 4 Carbon (high) each completed all 400 prompts within the 600–800-word target. Their comparisons cover 94 direct model pairings and 1,880 matched story pairs, with 11,278/11,280 usable judgments. Comparison coverage.
Claude Fable 5.2 (high) initially refused 58 of the 400 prompts. Three retry rounds recovered 48, 7 and 3 stories. Each recovered story is the first successful output for its prompt; these 58 later generation draws remain a resampling caveat despite complete story availability.
Qwen 4 Max (high) completed 397/400 prompts; ByteDance Seed2.1 Pro completed 400/400 prompts. All accepted stories meet the 600–800-word target. Their comparisons cover 49 direct model pairings and 980 matched story pairs, with 5,880/5,880 usable judgments. Comparison coverage.
GPT-6.2 Astra (high) completed 400/400 prompts. All accepted stories meet the 600–800-word target. Its initial comparisons covered 25 direct model pairings and 500 matched story pairs, with 2,999/3,000 usable judgments. Comparison coverage.
The five highest-ranked current models before the depth update each have at least 650 matched story pairs. The update adds 207 matched pairs and 1,241/1,242 usable judgments. Comparison depth by model.
Read each cell by row. Red means the row model performed better, blue means the column model performed better, and grey means the models were not directly compared. Near-white cells indicate close results. Both axes follow the leaderboard order. Earlier and newer evaluations contribute to both this chart and the rankings.
The GPT-6.2 Astra (high) versus GPT-6 Astra (high) qualitative comparison covers 50 matched prompts. Astra 6.2 won 50, tied 0 and lost 0 at the story-level margin threshold. The quantitative report uses 299/300 planned judgments. The original 20 prompts and 30 additional prompts each count once in the leaderboard.
The Claude Fable 5.2 (high) versus Claude Fable 5.1 (high) qualitative comparison examines recurring habits, range and narrative choices across 50 matched prompts, with blinded readings from three independent model families and Astra's synthesis. Quantitative comparison: 300/300 usable judgments, retaining the original 20 prompts and adding 30 unused prompts. Fable 5.2 wins 39, ties six and loses five at the story-level margin threshold. These additional comparisons support the focused report and are separate from the main leaderboard's intake sample.
The Kimi K3.1 Pro (high) versus Kimi K3 qualitative comparison examines style, narrative control, recurring habits, and range across 50 matched prompts, with blinded readings from three independent model families. Quantitative comparison: 300/300 usable judgments, retaining the original 20 prompts and adding 30 previously unused prompts. All 50 prompts now contribute to the leaderboard and head-to-head chart.
The Gemini 4 Argon (high) versus Gemini 3.1 Pro Preview comparison covers 50 matched prompts and 294/300 usable judgments. Argon wins all 50 at the story-level margin threshold; six unavailable judgments are excluded. These additional comparisons support the focused report and are separate from the main leaderboard's intake sample.
The matrix includes earlier and newer evaluator models. Each number shows how similarly two evaluators scored the story pairs they both read. Values closer to 1 indicate stronger agreement; 0 means no consistent relationship, and negative values mean opposing scoring patterns. Orange indicates negative correlations, and blue indicates positive correlations. Blank cells lack enough varied judgments to calculate a correlation; the diagonal is omitted. Agreement on these stories does not establish evaluator accuracy.
GLM-5.1 / Muse Spark 1.1 (high): only two shared stories. Shared-story counts accompany every evaluator pair.
Each dot is one completed story; diamonds mark model averages. Thin vertical lines show uncertainty around the averages. The shaded band is the 600–800-word target. The chart includes 10,361 completed stories from all 26 displayed models. This measures length rather than writing quality.
Before the Mistral intake, six stories that exceeded 800 words were regenerated using their original prompts; the first replacement within 600–800 words was retained. The 12 comparisons that used a replaced story were rerun with the same evaluators and both story orders. The rankings and charts use these replacements.
Earlier and newer evaluators use the same instructions and scoring criteria. They have judged 1,200 of the same story pairs, covering all previously tested writer models. Different evaluator versions count separately, with one averaged judgment per evaluator on each story pair. Each story pair receives equal weight in the ranking. Ratings use a Thurstone statistical model with a correction for story presentation order.
The intervals reflect variation in completed comparisons. They do not account for how missing stories or judgments might change the scores, or how results would differ if only newer evaluators were used. Overlapping intervals mean small differences in rank may be uncertain.
Every story must meaningfully incorporate ten required elements:
- character
- object
- concept
- attribute
- action
- method
- setting
- timeframe
- motivation
- tone
Candidate combinations are proposed for coherence and originality, then independently rated. The strongest combination for each seed becomes a fixed brief used by every writer model. A typical brief might combine a neutron-star researcher, butterfly-wing dust, gradual change, a storm-damaged greenhouse, "after the flood," and kindled humility.
Evaluators reward integration rather than keyword inclusion: the required object should affect the plot, the motivation should produce a consequential choice, and the tone should shape the story's development. Both stories in every comparison answer the same brief, holding prompt difficulty constant. Evaluators also consider prose, coherence, character, originality, and overall effectiveness. The public score combines their choices; it is not an average 0-10 grade.
Because the combinations are pre-screened for creative potential, the benchmark measures story construction under deliberately combinable constraints—not completely free-form writing or recovery from arbitrary incoherent prompts.
- Generate stories in the benchmark format.
- Build matched story-comparison prompts for models that wrote to the same required elements.
- Show each pair in both story orders to reduce first- or second-position effects.
- Repeat comparisons across evaluators and combine their choices.
- When evaluator models change, compare their judgments on shared stories before combining results.
- Calculate relative model scores and their uncertainty ranges.
Each report describes its matched stories and evaluator panel. Reports retain the sources and evaluators used at the time.
New Models Compared with Their Predecessors collects sixteen release comparisons, including Gemini 4 Argon versus Gemini 3.1 Pro Preview, on a separate page.
New analyses give recurring writing habits, range and adaptability, and consistency separate treatment, with examples across different prompts. They examine repeated phrasing and story patterns as well as how flexibly each model changes its voice, tone, form, and narrative approach.
Story prompts are available under prompts_wc/, and generated stories under stories_wc/<model>/.
Current ratings and chart data include machine-readable leaderboard scores and uncertainty intervals, direct head-to-head results, matched story-pair results, and evaluator diagnostics.
Earlier benchmark data links to the immutable August 23, 2026 data release. That archive contains the exact evaluator commentary for both story orders, excluded or superseded responses, and the accompanying source story and prompt texts.
Earlier versions of this benchmark used absolute 0-10 rubric ratings rather than direct story comparisons. Those results remain historical context, but the current public quality ranking should use the pairwise comparison results above.
- November 17, 2026: Added GPT-6.2 Astra high.
- November 16, 2026: Added Qwen 4 Max high and ByteDance Seed2.1 Pro.
- November 12, 2026: Added Claude Fable 5.2 high, Tencent Hy4 high, Step 5 Preview high, and Gemini 4 Carbon high.
- October 23, 2026: Added Kimi K3.1 Pro high.
- October 13, 2026: Added Gemini 4 Argon high.
- October 13, 2026: Added Muse Wonder high.
- October 8, 2026: Added Mistral Large 4 high.
- September 26, 2026: Added Claude Opus 5.5 high, Claude Opus 5 high, MiMo V2.6 Pro thinking, DeepSeek V4.1 Flash high, Gemini 3.8 Flash high, and Grok 4.7 high.
- September 5, 2026: Added GPT-6 Astra high, Muse Spark 1.3.
- September 2, 2026: Added Claude Fable 5.1.
- August 23, 2026: Published comparison data and written evaluations.
- August 20, 2026: Added Muse Spark 1.2 high, DeepSeek V4 Pro high, Qwen 3.8 Max, Gemini 3.7 Flash high, and Grok 4.6 high. Added predecessor reports.
- July 25, 2026: Added Claude Opus 5.
- July 18, 2026: Added Kimi K3, updated evaluators.
- July 14, 2026: Added GPT-5.6, Muse Spark 1.1 high, and Grok 4.5.
- July 9, 2026: Added Grok 4.5.
- June 9, 2026: Added Claude Fable 5.
- May 29, 2026: Added Claude Opus 4.8 high and xhigh.
- May 26, 2026: Ernie 5.1, Qwen 3.7 Max, Mistral Medium 3.5, and Grok 4.3 added.
- May 20, 2026: Added Gemini 3.5 Flash.
- Apr 29, 2026: Refreshed the leaderboard with newer models, including GPT-5.5, Kimi K2.6, DeepSeek V4 Pro, Xiaomi MiMo V2.5 Pro, Qwen 3 Max Preview, Gemini 3.1 Pro Preview, ByteDance Seed2.0 Pro, Qwen 3.6 Max Preview, and MiniMax-M2.7.
Multi-agent benchmarks:
- PACT - Benchmarking LLM negotiation skill in multi-round buyer-seller bargaining
- BAZAAR - Evaluating LLMs in Economic Decision-Making within a Competitive Simulated Market
- Buyout Game - Multi-Agent Negotiation and Coalition Benchmark
- LLM Debate Benchmark
- Public Goods Game Benchmark: Contribute & Punish
- Elimination Game: Social Reasoning and Deception Under Pressure
- Step Race: Collaboration vs. Misdirection Under Pressure
- LLM Persuasion Benchmark
Other benchmarks:
- LLM Position Bias Benchmark
- LLM Round-Trip Translation Benchmark
- Extended NYT Connections
- LLM Thematic Generalization Benchmark
- LLM Confabulation/Hallucination Benchmark
- LLM Deceptiveness and Gullibility
- LLM Sycophancy Benchmark
- LLM Divergent Thinking Creativity Benchmark
Follow @lechmazur on X for other benchmarks and updates.



