With the recent advances of open-domain story generation, the lack of
reliable automatic evaluation metrics becomes an increasingly imperative issue
that hinders the fast development of story generation. According to conducted
researches in this regard, learnable evaluation metrics have promised more
accurate assessments by having higher correlations with human judgments. A
critical bottleneck of obtaining a reliable learnable evaluation metric is the
lack of high-quality training data for classifiers to efficiently distinguish
plausible and implausible machine-generated stories. Previous works relied on
\textit{heuristically manipulated} plausible examples to mimic possible system
drawbacks such as repetition, contradiction, or irrelevant content in the text
level, which can be \textit{unnatural} and \textit{oversimplify} the
characteristics of implausible machine-generated stories. We propose to tackle
these issues by generating a more comprehensive set of implausible stories
using {\em plots}, which are structured representations of controllable factors
used to generate stories. Since these plots are compact and structured, it is
easier to manipulate them to generate text with targeted undesirable
properties, while at the same time maintain the grammatical correctness and
naturalness of the generated sentences. To improve the quality of generated
implausible stories, we further apply the adversarial filtering procedure
presented by \citet{zellers2018swag} to select a more nuanced set of
implausible texts. Experiments show that the evaluation metrics trained on our
generated data result in more reliable automatic assessments that correlate
remarkably better with human judgments compared to the baselines.

本文讨论了如何通过使用结构化的故事情节生成更全面的不合理故事来解决自动生成故事的可靠自动评估指标的不足，并应用对抗过滤程序来优化生成的故事。通过使用这种方法，我们可以更精确地评估人工生成与自动生成的故事之间的不同之处。

基于情节导向的对抗样本构建，用于评估开放域故事生成

Plot-guided Adversarial Example Construction for Evaluating Open-domain  Story Generation

Automatic storytelling is challenging since it requires generating long,
coherent natural language to describes a sensible sequence of events. Despite
considerable efforts on automatic story generation in the past, prior work
either is restricted in plot planning, or can only generate stories in a narrow
domain. In this paper, we explore open-domain story generation that writes
stories given a title (topic) as input. We propose a plan-and-write
hierarchical generation framework that first plans a storyline, and then
generates a story based on the storyline. We compare two planning strategies.
The dynamic schema interweaves story planning and its surface realization in
text, while the static schema plans out the entire storyline before generating
stories. Experiments show that with explicit storyline planning, the generated
stories are more diverse, coherent, and on topic than those generated without
creating a full plan, according to both automatic and human evaluations.

本文探讨了根据标题生成故事的开放域故事生成，提出了一种计划和写作的分层式生成框架，比较了两种规划策略，并通过实验表明了明确的故事情节规划会让生成的故事更加多样化、连贯且主题贴切。