Evaluations of creative stories generated by large language models (LLMs)
often focus on objective properties of the text, such as its style, coherence,
and toxicity. While these metrics are indispensable, they do not speak to a
story's subjective, psychological impact from a reader's perspective. We
introduce the Psychological Depth Scale (PDS), a novel framework rooted in
literary theory that measures an LLM's ability to produce authentic and
narratively complex stories that provoke emotion, empathy, and engagement. We
empirically validate our framework by showing that humans can consistently
evaluate stories based on PDS (0.72 Krippendorff's alpha). We also explore
techniques for automating the PDS to easily scale future analyses. GPT-4o,
combined with a novel Mixture-of-Personas (MoP) prompting strategy, achieves an
average Spearman correlation of $0.51$ with human judgment while Llama-3-70B
scores as high as 0.68 for empathy. Finally, we compared the depth of stories
authored by both humans and LLMs. Surprisingly, GPT-4 stories either surpassed
or were statistically indistinguishable from highly-rated human-written stories
sourced from Reddit. By shifting the focus from text to reader, the
Psychological Depth Scale is a validated, automated, and systematic means of
measuring the capacity of LLMs to connect with humans through the stories they
tell.

通过 Psychological Depth Scale 框架，评估大型语言模型（LLMs）生成的创意故事的心理深度，该框架根植于文学理论，测量 LLMs 生成真实且叙述复杂的故事的能力，能够引发情感、共情和参与度，并验证其自动化与系统性。