Semantic Overlap Summarization (SOS) is a constrained multi-document
summarization task, where the constraint is to capture the common/overlapping
information between two alternative narratives. While recent advancements in
Large Language Models (LLMs) have achieved superior performance in numerous
summarization tasks, a benchmarking study of the SOS task using LLMs is yet to
be performed. As LLMs' responses are sensitive to slight variations in prompt
design, a major challenge in conducting such a benchmarking study is to
systematically explore a variety of prompts before drawing a reliable
conclusion. Fortunately, very recently, the TELeR taxonomy has been proposed
which can be used to design and explore various prompts for LLMs. Using this
TELeR taxonomy and 15 popular LLMs, this paper comprehensively evaluates LLMs
on the SOS Task, assessing their ability to summarize overlapping information
from multiple alternative narratives. For evaluation, we report
well-established metrics like ROUGE, BERTscore, and SEM-F1$ on two different
datasets of alternative narratives. We conclude the paper by analyzing the
strengths and limitations of various LLMs in terms of their capabilities in
capturing overlapping information The code and datasets used to conduct this
study are available at this https URL

应用 TELeR 分类法和 15 种流行的大型语言模型（LLMs），对语义重叠摘要任务进行全面评估，评估它们在从多个备选叙述中总结重叠信息的能力，通过使用 ROUGE、BERTscore 和 SEM-F1 等已建立的指标对两个不同数据集进行评估，并分析各种 LLMs 在捕捉重叠信息方面的优势和局限性。

语义重叠归纳任务的 LLM 性能评估

Benchmarking LLMs on the Semantic Overlap Summarization Task

One of the most important yet onerous tasks in the academic peer-reviewing
process is composing meta-reviews, which involves understanding the core
contributions, strengths, and weaknesses of a scholarly manuscript based on
peer-review narratives from multiple experts and then summarizing those
multiple experts' perspectives into a concise holistic overview. Given the
latest major developments in generative AI, especially Large Language Models
(LLMs), it is very compelling to rigorously study the utility of LLMs in
generating such meta-reviews in an academic peer-review setting. In this paper,
we perform a case study with three popular LLMs, i.e., GPT-3.5, LLaMA2, and
PaLM2, to automatically generate meta-reviews by prompting them with different
types/levels of prompts based on the recently proposed TELeR taxonomy. Finally,
we perform a detailed qualitative study of the meta-reviews generated by the
LLMs and summarize our findings and recommendations for prompting LLMs for this
complex task.

通过案例研究，使用 GPT-3.5、LLaMA2 和 PaLM2 等三种大型语言模型根据 TELeR 分类法的不同类型 / 级别的引导，自动生成学术同行评审中的元评审，然后对元评审进行详细的定性研究，总结了在这一复杂任务中为大型语言模型提供引导的发现和建议。