Storytelling is an integral part of human experience and plays a crucial role
in social interactions. Thus, Automatic Story Evaluation (ASE) and Generation
(ASG) could benefit society in multiple ways, but they are challenging tasks
which require high-level human abilities such as creativity, reasoning and deep
understanding. Meanwhile, Large Language Models (LLM) now achieve
state-of-the-art performance on many NLP tasks. In this paper, we study whether
LLMs can be used as substitutes for human annotators for ASE. We perform an
extensive analysis of the correlations between LLM ratings, other automatic
measures, and human annotations, and we explore the influence of prompting on
the results and the explainability of LLM behaviour. Most notably, we find that
LLMs outperform current automatic measures for system-level evaluation but
still struggle at providing satisfactory explanations for their answers.

自动故事评估与生成在社交交互中起到重要作用，研究表明大语言模型能够替代人工评注者，但仍存在解释答案的困难。

语言模型是否喜欢自己的故事？用于自动故事评估的大型语言模型的激励

Do Language Models Enjoy Their Own Stories? Prompting Large Language  Models for Automatic Story Evaluation

Existing automatic story evaluation methods place a premium on story lexical
level coherence, deviating from human preference. We go beyond this limitation
by considering a novel \textbf{Story} \textbf{E}valuation method that mimics
human preference when judging a story, namely \textbf{StoryER}, which consists
of three sub-tasks: \textbf{R}anking, \textbf{R}ating and \textbf{R}easoning.
Given either a machine-generated or a human-written story, StoryER requires the
machine to output 1) a preference score that corresponds to human preference,
2) specific ratings and their corresponding confidences and 3) comments for
various aspects (e.g., opening, character-shaping). To support these tasks, we
introduce a well-annotated dataset comprising (i) 100k ranked story pairs; and
(ii) a set of 46k ratings and comments on various aspects of the story. We
finetune Longformer-Encoder-Decoder (LED) on the collected dataset, with the
encoder responsible for preference score and aspect prediction and the decoder
for comment generation. Our comprehensive experiments result in a competitive
benchmark for each task, showing the high correlation to human preference. In
addition, we have witnessed the joint learning of the preference scores, the
aspect ratings, and the comments brings gain in each single task. Our dataset
and benchmarks are publicly available to advance the research of story
evaluation tasks.\footnote{Dataset and pre-trained model demo are available at
anonymous website \url{this http URL} and
\url{this https URL}}

提出了一种新的故事评估方法 StoryER，包括排序、评分和评论等三个子任务，此方法不仅考虑故事的词汇一致性，还结合了人类偏好进行评估，其基于的公开数据集与经过微调的 Longformer-Encoder-Decoder，通过在每个任务上产生优异的性能表现，展示了其与人类喜好的高相关性。