Evaluating the factual consistency of automatically generated summaries is
essential for the progress and adoption of reliable summarization systems.
Despite recent advances, existing factuality evaluation models are not robust,
being especially prone to entity and relation errors in new domains. We propose
FactKB, a simple new approach to factuality evaluation that is generalizable
across domains, in particular with respect to entities and relations. FactKB is
based on language models pretrained using facts extracted from external
knowledge bases. We introduce three types of complementary factuality
pretraining objectives based on direct entity facts, facts grounded in
auxiliary knowledge about entities, and facts constructed compositionally
through knowledge base walks. The resulting factuality evaluation model
achieves state-of-the-art performance on two in-domain news summarization
benchmarks as well as on three out-of-domain scientific literature datasets.
Further analysis of FactKB shows improved ability to detect erroneous entities
and relations in summaries and is robust and generalizable across domains.

文章提出了一种新的事实评估方法 FactKB，该方法使用基于预先抽取的实体知识的语言模型，以达到跨领域的通用性并解决现有模型在新领域中存在的实体和关系错误问题。通过在两个领域内的新闻汇总测试数据和三个跨领域的科学文献数据上测试，FactKB 的事实性评估模型达到了最先进的性能水平，并表现出在摘要中检测错误实体和关系的能力显著提高，从而进一步证明其在领域通用性和鲁棒性方面的特点。

FactKB：使用增强事实知识的语言模型进行可泛化事实性评估

FactKB: Generalizable Factuality Evaluation using Language Models  Enhanced with Factual Knowledge

State-of-the-art summarization models still struggle to be factually
consistent with the input text. A model-agnostic way to address this problem is
post-editing the generated summaries. However, existing approaches typically
fail to remove entity errors if a suitable input entity replacement is not
available or may insert erroneous content. In our work, we focus on removing
extrinsic entity errors, or entities not in the source, to improve consistency
while retaining the summary's essential information and form. We propose to use
sentence-compression data to train the post-editing model to take a summary
with extrinsic entity errors marked with special tokens and output a
compressed, well-formed summary with those errors removed. We show that this
model improves factual consistency while maintaining ROUGE, improving entity
precision by up to 30% on XSum, and that this model can be applied on top of
another post-editor, improving entity precision by up to a total of 38%. We
perform an extensive comparison of post-editing approaches that demonstrate
trade-offs between factual consistency, informativeness, and grammaticality,
and we analyze settings where post-editors show the largest improvements.

本文提出了一种基于句子压缩数据的模型无关方法，旨在在保留要点和格式要求的同时消除外在实体错误，从而提高信息摘要生成中的事实一致性，并使用 XSum 数据集进行评估， 实验结果表明，在不影响 ROUGE 的情况下，本文所提出的方法可以提高实体准确性高达 30％。