Recent work has shown that large language models (LLMs) are capable of
generating summaries zero-shot (i.e., without explicit supervision) that, under
human assessment, are often comparable or even preferred to manually composed
reference summaries. However, this prior work has focussed almost exclusively
on evaluating news article summarization. How do zero-shot summarizers perform
in other (potentially more specialized) domains? In this work we evaluate
zero-shot generated summaries across specialized domains including biomedical
articles, and legal bills (in addition to standard news benchmarks for
reference). We focus especially on the factuality of outputs. We acquire
annotations from domain experts to identify inconsistencies in summaries and
systematically categorize these errors. We analyze whether the prevalence of a
given domain in the pretraining corpus affects extractiveness and faithfulness
of generated summaries of articles in this domain. We release all collected
annotations to facilitate additional research toward measuring and realizing
factually accurate summarization, beyond news articles. The dataset can be
downloaded from this https URL

大型语言模型（LLM）的零摘要生成与人工编写的参考摘要相媲美，我们评估了零摘要生成摘要在生物医学文章等专业领域的实际性，并通过领域专家注释识别总结中的不一致性。

跨领域评估零样本摘要生成器的真实性

Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains

Based on the remarkable achievements of pre-trained language models in
abstractive summarization, the copying mechanism has proved helpful by
improving the factuality, stability, and overall performance. This work
proposes PROM, a new PhRase-level cOpying Mechanism that enhances attention on
n-grams, which can be applied to zero-shot summarization with pre-training.
PROM adds an indicator layer to explicitly pick up tokens in n-gram that can be
copied from the source, and calculates an auxiliary loss for the copying
prediction. Empirical studies show that PROM makes significant improvements in
fine-tuning on benchmarks. In zero-shot setting, PROM is utilized in the
self-supervised pre-training on raw corpora and provides new general baselines
on a wide range of summarization datasets. Further analysis shows that PROM
performs more reasonable copying and contributes to faithfulness.

本文提出了一种新型的词组级别的复制机制 PROM，可用于预训练的零 - shot 摘要生成，能够显著提高摘要性能和忠实度。

PROM: 一种预训练的短语级复制机制用于抽象摘要

PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive  Summarization

Abstractive summarization systems leveraging pre-training language models
have achieved superior results on benchmark datasets. However, such models have
been shown to be more prone to hallucinate facts that are unfaithful to the
input context. In this paper, we propose a method to remedy entity-level
extrinsic hallucinations with Entity Coverage Control (ECC). We first compute
entity coverage precision and prepend the corresponding control code for each
training example, which implicitly guides the model to recognize faithfulness
contents in the training phase. We further extend our method via intermediate
fine-tuning on large but noisy data extracted from Wikipedia to unlock
zero-shot summarization. We show that the proposed method leads to more
faithful and salient abstractive summarization in supervised fine-tuning and
zero-shot settings according to our experimental results on three benchmark
datasets XSum, Pubmed, and SAMSum of very different domains and styles.

本文提出了一种使用 “实体覆盖控制（Entity Coverage Control，ECC）” 方法来解决预训练语言模型产生的 “幻觉（hallucinate）” 现象的方案，该方法不仅更好地保持了上下文的内容准确性，而且在使用大量噪声数据进行中间优调（intermediate fine-tuning）时，还可以解决无需监督的情况下的提要抽象生成（zero-shot summarization）问题，并证明了该方法在三个不同领域和风格的基准数据集上比其他方法更为有效和鲜明。