Summarizing book-length documents (>100K tokens) that exceed the context
window size of large language models (LLMs) requires first breaking the input
document into smaller chunks and then prompting an LLM to merge, update, and
compress chunk-level summaries. Despite the complexity and importance of this
task, it has yet to be meaningfully studied due to the challenges of
evaluation: existing book-length summarization datasets (e.g., BookSum) are in
the pretraining data of most public LLMs, and existing evaluation methods
struggle to capture errors made by modern LLM summarizers. In this paper, we
present the first study of the coherence of LLM-based book-length summarizers
implemented via two prompting workflows: (1) hierarchically merging chunk-level
summaries, and (2) incrementally updating a running summary. We obtain 1193
fine-grained human annotations on GPT-4 generated summaries of 100
recently-published books and identify eight common types of coherence errors
made by LLMs. Because human evaluation is expensive and time-consuming, we
develop an automatic metric, BooookScore, that measures the proportion of
sentences in a summary that do not contain any of the identified error types.
BooookScore has high agreement with human annotations and allows us to
systematically evaluate the impact of many other critical parameters (e.g.,
chunk size, base LLM) while saving $15K and 500 hours in human evaluation
costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce
summaries with higher BooookScore than the oft-repetitive ones generated by
LLaMA 2. Incremental updating yields lower BooookScore but higher level of
detail than hierarchical merging, a trade-off sometimes preferred by human
annotators. We release code and annotations after blind review to spur more
principled research on book-length summarization.

通过两种工作流程 (层级合并、递增更新) 实现基于 LLM 的书籍摘要器的连贯性研究，发现 LLM 会出现八种常见的连贯性错误，并提出了一个自动评估指标 BooookScore，与人工标注结果一致，可用于系统评估其他关键参数的影响，同时节省了人工评估成本。