While long-context large language models (LLMs) can technically summarize
book-length documents (>100K tokens), the length and complexity of the
documents have so far prohibited evaluations of input-dependent aspects like
faithfulness. In this paper, we conduct the first large-scale human evaluation
of faithfulness and content selection on LLM-generated summaries of fictional
books. Our study mitigates the issue of data contamination by focusing on
summaries of books published in 2023 or 2024, and we hire annotators who have
fully read each book prior to the annotation task to minimize cost and
cognitive burden. We collect FABLES, a dataset of annotations on 3,158 claims
made in LLM-generated summaries of 26 books, at a cost of $5.2K USD, which
allows us to rank LLM summarizers based on faithfulness: Claude-3-Opus
significantly outperforms all closed-source LLMs, while the open-source Mixtral
is on par with GPT-3.5-Turbo. An analysis of the annotations reveals that most
unfaithful claims relate to events and character states, and they generally
require indirect reasoning over the narrative to invalidate. While LLM-based
auto-raters have proven reliable for factuality and coherence in other
settings, we implement several LLM raters of faithfulness and find that none
correlates strongly with human annotations, especially with regard to detecting
unfaithful claims. Our experiments suggest that detecting unfaithful claims is
an important future direction not only for summarization evaluation but also as
a testbed for long-context understanding. Finally, we move beyond faithfulness
by exploring content selection errors in book-length summarization: we develop
a typology of omission errors related to crucial narrative elements and also
identify a systematic over-emphasis on events occurring towards the end of the
book.

通过对虚构书籍的长篇小说生成概述的大规模人类评估，本论文揭示了长篇大语境语言模型在生成摘要时的忠实度和内容选择方面的问题，并且提出了检测生成的摘要中不忠实内容的重要性，同时也探讨了在书籍概述中与关键故事元素相关的遗漏错误以及朝末尾发生事件的系统过度强调。