We evaluate recent Large language Models (LLMs) on the challenging task of
summarizing short stories, which can be lengthy, and include nuanced subtext or
scrambled timelines. Importantly, we work directly with authors to ensure that
the stories have not been shared online (and therefore are unseen by the
models), and to obtain informed evaluations of summary quality using judgments
from the authors themselves. Through quantitative and qualitative analysis
grounded in narrative theory, we compare GPT-4, Claude-2.1, and LLama-2-70B. We
find that all three models make faithfulness mistakes in over 50% of summaries
and struggle to interpret difficult subtext. However, at their best, the models
can provide thoughtful thematic analysis of stories. We additionally
demonstrate that LLM judgments of summary quality do not match the feedback
from the writers.

最近的大型语言模型在总结短篇小说这一具有挑战性的任务上进行评估，结果显示这些模型在超过 50% 的总结中存在不忠实的错误，并且对于难以解释的含义也有困难，然而在最好的情况下，这些模型可以提供有思考深度的故事主题分析，并且我们还证明了语言模型对总结质量的评判与作者的反馈不一致。