Automatic summary assessment is useful for both machine-generated and
human-produced summaries. Automatically evaluating the summary text given the
document enables, for example, summary generation system development and
detection of inappropriate summaries. Summary assessment can be run in a number
of modes: ranking summary generation systems; ranking summaries of a particular
document; and estimating the quality of a document-summary pair on an absolute
scale. Existing datasets with annotation for summary assessment are usually
based on news summarization datasets such as CNN/DailyMail or XSum. In this
work, we describe a new dataset, the podcast summary assessment corpus, a
collection of podcast summaries that were evaluated by human experts at
TREC2020. Compared to existing summary assessment data, this dataset has two
unique aspects: (i) long-input, speech podcast based, documents; and (ii) an
opportunity to detect inappropriate reference summaries in podcast corpus.
First, we examine existing assessment methods, including model-free and
model-based methods, and provide benchmark results for this long-input summary
assessment dataset. Second, with the aim of filtering reference
summary-document pairings for training, we apply summary assessment for data
selection. The experimental results on these two aspects provide interesting
insights on the summary assessment and generation tasks. The podcast summary
assessment data is available.

本文介绍了一种新的数据集，即播客摘要评估语料库，该数据集可用于评估长输入摘要，并提供数据选择功能以筛选参考摘要 - 文档对用作训练。