Previous research in multi-document news summarization has typically
concentrated on collating information that all sources agree upon. However, to
our knowledge, the summarization of diverse information dispersed across
multiple articles about an event has not been previously investigated. The
latter imposes a different set of challenges for a summarization model. In this
paper, we propose a new task of summarizing diverse information encountered in
multiple news articles encompassing the same event. To facilitate this task, we
outlined a data collection schema for identifying diverse information and
curated a dataset named DiverseSumm. The dataset includes 245 news stories,
with each story comprising 10 news articles and paired with a human-validated
reference. Moreover, we conducted a comprehensive analysis to pinpoint the
position and verbosity biases when utilizing Large Language Model (LLM)-based
metrics for evaluating the coverage and faithfulness of the summaries, as well
as their correlation with human assessments. We applied our findings to study
how LLMs summarize multiple news articles by analyzing which type of diverse
information LLMs are capable of identifying. Our analyses suggest that despite
the extraordinary capabilities of LLMs in single-document summarization, the
proposed task remains a complex challenge for them mainly due to their limited
coverage, with GPT-4 only able to cover less than 40% of the diverse
information on average.

在多篇文档新闻摘要研究中，以往研究通常集中于整合各个来源都同意的信息。然而，据我们所知，以往尚未研究过涉及同一事件的多篇文章中分散的多样信息的摘要。后者为摘要模型提出了一组不同的挑战。在本文中，我们提出了一个新任务，即摘要涉及同一事件的多篇新闻文章中的多样信息。为了完成这项任务，我们确定了一个用于识别多样信息的数据收集模式，并创建了一个名为 DiverseSumm 的数据集。该数据集包含 245 个新闻故事，每个故事包括 10 篇新闻文章，并配有人工验证的参考摘要。此外，我们进行了全面的分析，以确定基于大型语言模型（LLM）的度量标准在评估摘要的全面性和忠实度时存在的位置和冗长偏差，以及它们与人工评估的相关性。我们应用我们的发现来研究 LLMs 如何通过分析 LLMs 能够识别哪种类型的多样信息来摘要多篇新闻文章。我们的分析表明，尽管 LLMs 在单篇文档摘要方面具有非凡的能力，但对于他们来说，所提出的任务仍然是一个复杂的挑战，主要是由于他们的覆盖范围有限，GPT-4 只能够平均覆盖不到 40% 的多样信息。

拥抱分歧以获得更丰富的见解：一个多文档摘要基准和一个从新闻文章中概括多样信息的案例研究

Embrace Divergence for Richer Insights: A Multi-document Summarization  Benchmark and a Case Study on Summarizing Diverse Information from News  Articles