ChatGPT stories rated 59% higher than human fiction — until readers know the source

Here’s a finding that should make every literary editor squirm: readers can’t tell ChatGPT from Chekhov. In a trio of experiments spanning more than 2,500 participants, people consistently rated AI-generated short stories as more immersive and higher quality than human-written ones. The catch? That preference evaporated the moment they learned a machine was the author.

The study, published in Judgment and Decision Making by researchers Sydney Sears and Deena Skolnick Weisberg, gave 1,682 participants one of six short stories. Three were pulled from respected literary magazines; three were cooked up by ChatGPT 4.0 using prompts mimicking the human stories’ themes and style. When readers were blind to authorship, the AI stories scored a mean quality rating of 1.54 compared to 0.97 for human work on a scale from -3 to +3. Immersion scores showed a similar gap. That’s a decisive win for the silicon storyteller.

But the plot thickens when you look at what happened when participants were told who wrote what. People with positive attitudes toward AI gave even higher marks when they believed ChatGPT was the author. Skeptics did the opposite — penalizing stories they thought were machine-made, even if a human actually wrote them. Two follow-up experiments with 905 participants added a head-to-head challenge: read one of each and guess which is which. Even with direct comparison, people performed no better than chance. Self-reported AI experience correlated with better detection. Years spent reading fiction? Useless.

The researchers offer a sobering explanation that has nothing to do with AI’s literary genius. Machine-generated text tends to be smoother, simpler, and more emotionally pleasant — qualities that make it easier to process. High-quality literary fiction often does the opposite, deliberately making readers work for meaning. The short story format, constrained to about 1,000 words, also plays to AI’s strengths. Coherence over a few pages is a very different beast than sustaining it across three hundred. This aligns with earlier work from Stony Brook and Columbia showing that when models are fine-tuned on an author’s style, professional readers actually prefer the AI imitation eight times more often. The uncomfortable conclusion staring back at us: AI can produce creative work people genuinely enjoy, but we’re still not ready to admit it.