Current metrics for evaluating factuality for abstractive document
summarization have achieved high correlations with human judgment, but they do
not account for the vision modality and thus are not adequate for
vision-and-language summarization. We propose CLIPBERTScore, a simple weighted
combination of CLIPScore and BERTScore to leverage the robustness and strong
factuality detection performance between image-summary and document-summary,
respectively. Next, due to the lack of meta-evaluation benchmarks to evaluate
the quality of multimodal factuality metrics, we collect human judgments of
factuality with respect to documents and images. We show that this simple
combination of two metrics in the zero-shot setting achieves higher
correlations than existing factuality metrics for document summarization,
outperforms an existing multimodal summarization metric, and performs
competitively with strong multimodal factuality metrics specifically fine-tuned
for the task. Our thorough analysis demonstrates the robustness and high
correlation of CLIPBERTScore and its components on four factuality
metric-evaluation benchmarks. Finally, we demonstrate two practical downstream
applications of our CLIPBERTScore metric: for selecting important images to
focus on during training, and as a reward for reinforcement learning to improve
factuality of multimodal summary generation w.r.t automatic and human
evaluation. Our data and code are publicly available at
this https URL

我们提出了 CLIPBERTScore，一种结合 CLIPScore 和 BERTScore 的加权组合度量方法，用于多模式事实性摘要的评估，并在多个数据集上证明了其效果。

多模态抽象摘要中事实性的评估与提高

Evaluating and Improving Factuality in Multimodal Abstractive Summarization

It is well known that the standard likelihood training and approximate
decoding objectives in neural text generation models lead to less human-like
responses for open-ended tasks such as language modeling and story generation.
In this paper we have analyzed limitations of these models for abstractive
document summarization and found that these models are highly prone to
hallucinate content that is unfaithful to the input document. We conducted a
large scale human evaluation of several neural abstractive summarization
systems to better understand the types of hallucinations they produce. Our
human annotators found substantial amounts of hallucinated content in all model
generated summaries. However, our analysis does show that pretrained models are
better summarizers not only in terms of raw metrics, i.e., ROUGE, but also in
generating faithful and factual summaries as evaluated by humans. Furthermore,
we show that textual entailment measures better correlate with faithfulness
than standard metrics, potentially leading the way to automatic evaluation
metrics as well as training and decoding criteria.

分析了神经文本生成模型在抽象文档摘要生成方面的局限性，并发现这些模型非常容易产生虚假内容。人类评估显示，预训练模型生成的摘要不仅在原始度量标准（即 ROUGE）方面更好，而且在生成忠实和真实摘要方面也更好，文本蕴含度量更好地相关于忠实度，有可能成为自动评估度量标准以及训练和解码标准的前景。

论抽象化摘要中的忠实度与客观性

On Faithfulness and Factuality in Abstractive Summarization

Abstractive document summarization is usually modeled as a
sequence-to-sequence (Seq2Seq) learning problem. Unfortunately, training large
Seq2Seq based summarization models on limited supervised summarization data is
challenging. This paper presents three pre-training objectives which allow us
to pre-train a Seq2Seq based abstractive summarization model on unlabeled text.
The main idea is that, given an input text artificially constructed from a
document, a model is pre-trained to reinstate the original document. These
objectives include sentence reordering, next sentence generation, and masked
document generation, which have close relations with the abstractive document
summarization task. Experiments on two benchmark summarization datasets (i.e.,
CNN/DailyMail and New York Times) show that all three objectives can improve
performance upon baselines. Compared to models pre-trained on large-scale data
(more than 160GB), our method, with only 19GB text for pre-training, achieves
comparable results, which demonstrates its effectiveness.

该论文提出了三个预训练目标，并基于未标记的文本对 Seq2Seq based abstractive summarization model 进行预训练，从而在有限的监督 summarization data 上改进其性能，这三个目标与 abstractive document summarization task 有密切关联。实验结果表明，这三个目标均能改进模型性能，且仅使用 19GB 的文本进行预训练即可与使用大规模数据集预训练的模型达到可比较的效果。