Automatic summarization generates concise summaries that contain key ideas of
source documents. As the most mainstream datasets for the news sub-domain,
CNN/DailyMail and BBC XSum have been widely used for performance benchmarking.
However, the reference summaries of those datasets turn out to be noisy, mainly
in terms of factual hallucination and information redundancy. To address this
challenge, we first annotate new expert-writing Element-aware test sets
following the "Lasswell Communication Model" proposed by Lasswell (1948),
allowing reference summaries to focus on more fine-grained news elements
objectively and comprehensively. Utilizing the new test sets, we observe the
surprising zero-shot summary ability of LLMs, which addresses the issue of the
inconsistent results between human preference and automatic evaluation metrics
of LLMs' zero-shot summaries in prior work. Further, we propose a Summary
Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step
by step, which helps them integrate more fine-grained details of source
documents into the final summaries that correlate with the human writing
mindset. Experimental results show our method outperforms state-of-the-art
fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L on the two
datasets, respectively. Dataset and code are publicly available at
this https URL

研究发现 CNN / DailyMail 和 BBC XSum 数据集的参考摘要存在噪音，因为这些摘要主要存在于事实幻觉和信息冗余方面。为了解决这个问题，他们提出了 “Summary Chain-of-Thought（SumCoT）” 技术，通过逐步摘要的方式生成细粒度的摘要。利用新的测试集，研究人员观察到 LLM 具有惊人的零 - shot 摘要能力，该技术旨在使他们将更细粒度的细节集成到与人类写作方式相关的最终摘要中。实验结果表明，该方法在两个数据集上的 ROUGE-L 优于现有技术。