While human evaluation remains best practice for accurately judging the
faithfulness of automatically-generated summaries, few solutions exist to
address the increased difficulty and workload when evaluating long-form
summaries. Through a survey of 162 papers on long-form summarization, we first
shed light on current human evaluation practices surrounding long-form
summaries. We find that 73% of these papers do not perform any human evaluation
on model-generated summaries, while other works face new difficulties that
manifest when dealing with long documents (e.g., low inter-annotator
agreement). Motivated by our survey, we present LongEval, a set of guidelines
for human evaluation of faithfulness in long-form summaries that addresses the
following challenges: (1) How can we achieve high inter-annotator agreement on
faithfulness scores? (2) How can we minimize annotator workload while
maintaining accurate faithfulness scores? and (3) Do humans benefit from
automated alignment between summary and source snippets? We deploy LongEval in
annotation studies on two long-form summarization datasets in different domains
(SQuALITY and PubMed), and we find that switching to a finer granularity of
judgment (e.g., clause-level) reduces inter-annotator variance in faithfulness
scores (e.g., std-dev from 18.5 to 6.8). We also show that scores from a
partial annotation of fine-grained units highly correlates with scores from a
full annotation workload (0.89 Kendall's tau using 50% judgments). We release
our human judgments, annotation templates, and our software as a Python library
for future research.

本研究针对长文本自动摘要难以进行人工评估的问题，通过对 162 篇长文本摘要论文的调研，提出了一种针对长文本自动摘要信度评估的人工评估指南 LongEval，其中包括如何实现准确的信度得分，减少评估者的工作量，以及人工与自动化对齐等方面的考虑，并通过在两个不同领域的长文本摘要数据集上进行注释研究，发现在精细度方面，如句子的级别，能够显著降低评估者之间的差异。

LongEval：长文摘要人工评估的忠实度指南

LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization

Prior work in document summarization has mainly focused on generating short
summaries of a document. While this type of summary helps get a high-level view
of a given document, it is desirable in some cases to know more detailed
information about its salient points that can't fit in a short summary. This is
typically the case for longer documents such as a research paper, legal
document, or a book. In this paper, we present a new method for generating
extended summaries of long papers. Our method exploits hierarchical structure
of the documents and incorporates it into an extractive summarization model
through a multi-task learning approach. We then present our results on three
long summarization datasets, arXiv-Long, PubMed-Long, and Longsumm. Our method
outperforms or matches the performance of strong baselines. Furthermore, we
perform a comprehensive analysis over the generated results, shedding insights
on future research for long-form summary generation task. Our analysis shows
that our multi-tasking approach can adjust extraction probability distribution
to the favor of summary-worthy sentences across diverse sections. Our datasets,
and codes are publicly available at
this https URL

本研究基于多任务学习方法，利用文件的层次结构生成长篇文献的扩展摘要，并在三个大型数据集上验证其优于其他强基准模型的性能，为长篇文献摘要生成任务的未来研究提供了深入的洞见。

生成长文档的扩展摘要

On Generating Extended Summaries of Long Documents

Most prior work in the sequence-to-sequence paradigm focused on datasets with
input sequence lengths in the hundreds of tokens due to the computational
constraints of common RNN and Transformer architectures. In this paper, we
study long-form abstractive text summarization, a sequence-to-sequence setting
with input sequence lengths up to 100,000 tokens and output sequence lengths up
to 768 tokens. We propose SEAL, a Transformer-based model, featuring a new
encoder-decoder attention that dynamically extracts/selects input snippets to
sparsely attend to for each output segment. Using only the original documents
and summaries, we derive proxy labels that provide weak supervision for
extractive layers simultaneously with regular supervision from abstractive
summaries. The SEAL model achieves state-of-the-art results on existing
long-form summarization tasks, and outperforms strong baseline models on a new
dataset/task we introduce, Search2Wiki, with much longer input text. Since
content selection is explicit in the SEAL model, a desirable side effect is
that the selection can be inspected for enhanced interpretability.

该论文提出了一种基于 Transformer 的模型 SEAL，用于长篇抽象文本摘要，模型具有可解释性，同时可以使用原始文档和摘要提供弱监督学习信号。该模型在现有长篇摘要任务上取得了最佳结果，并在新的数据集 / 任务 Search2Wiki 上优于强基线模型。