While long-context large language models (LLMs) can technically summarize
book-length documents (>100K tokens), the length and complexity of the
documents have so far prohibited evaluations of input-dependent aspects like
faithfulness. In this paper, we conduct the first large-scale human evaluation
of faithfulness and content selection on LLM-generated summaries of fictional
books. Our study mitigates the issue of data contamination by focusing on
summaries of books published in 2023 or 2024, and we hire annotators who have
fully read each book prior to the annotation task to minimize cost and
cognitive burden. We collect FABLES, a dataset of annotations on 3,158 claims
made in LLM-generated summaries of 26 books, at a cost of $5.2K USD, which
allows us to rank LLM summarizers based on faithfulness: Claude-3-Opus
significantly outperforms all closed-source LLMs, while the open-source Mixtral
is on par with GPT-3.5-Turbo. An analysis of the annotations reveals that most
unfaithful claims relate to events and character states, and they generally
require indirect reasoning over the narrative to invalidate. While LLM-based
auto-raters have proven reliable for factuality and coherence in other
settings, we implement several LLM raters of faithfulness and find that none
correlates strongly with human annotations, especially with regard to detecting
unfaithful claims. Our experiments suggest that detecting unfaithful claims is
an important future direction not only for summarization evaluation but also as
a testbed for long-context understanding. Finally, we move beyond faithfulness
by exploring content selection errors in book-length summarization: we develop
a typology of omission errors related to crucial narrative elements and also
identify a systematic over-emphasis on events occurring towards the end of the
book.

通过对虚构书籍的长篇小说生成概述的大规模人类评估，本论文揭示了长篇大语境语言模型在生成摘要时的忠实度和内容选择方面的问题，并且提出了检测生成的摘要中不忠实内容的重要性，同时也探讨了在书籍概述中与关键故事元素相关的遗漏错误以及朝末尾发生事件的系统过度强调。

FABLES：评估长篇书籍摘要的忠实性与内容选择

FABLES: Evaluating faithfulness and content selection in book-length  summarization

Summarizing book-length documents (>100K tokens) that exceed the context
window size of large language models (LLMs) requires first breaking the input
document into smaller chunks and then prompting an LLM to merge, update, and
compress chunk-level summaries. Despite the complexity and importance of this
task, it has yet to be meaningfully studied due to the challenges of
evaluation: existing book-length summarization datasets (e.g., BookSum) are in
the pretraining data of most public LLMs, and existing evaluation methods
struggle to capture errors made by modern LLM summarizers. In this paper, we
present the first study of the coherence of LLM-based book-length summarizers
implemented via two prompting workflows: (1) hierarchically merging chunk-level
summaries, and (2) incrementally updating a running summary. We obtain 1193
fine-grained human annotations on GPT-4 generated summaries of 100
recently-published books and identify eight common types of coherence errors
made by LLMs. Because human evaluation is expensive and time-consuming, we
develop an automatic metric, BooookScore, that measures the proportion of
sentences in a summary that do not contain any of the identified error types.
BooookScore has high agreement with human annotations and allows us to
systematically evaluate the impact of many other critical parameters (e.g.,
chunk size, base LLM) while saving $15K and 500 hours in human evaluation
costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce
summaries with higher BooookScore than the oft-repetitive ones generated by
LLaMA 2. Incremental updating yields lower BooookScore but higher level of
detail than hierarchical merging, a trade-off sometimes preferred by human
annotators. We release code and annotations after blind review to spur more
principled research on book-length summarization.

通过两种工作流程 (层级合并、递增更新) 实现基于 LLM 的书籍摘要器的连贯性研究，发现 LLM 会出现八种常见的连贯性错误，并提出了一个自动评估指标 BooookScore，与人工标注结果一致，可用于系统评估其他关键参数的影响，同时节省了人工评估成本。