Evaluating the factual consistency of automatically generated summaries is
essential for the progress and adoption of reliable summarization systems.
Despite recent advances, existing factuality evaluation models are not robust,
being especially prone to entity and relation errors in new domains. We propose
FactKB, a simple new approach to factuality evaluation that is generalizable
across domains, in particular with respect to entities and relations. FactKB is
based on language models pretrained using facts extracted from external
knowledge bases. We introduce three types of complementary factuality
pretraining objectives based on direct entity facts, facts grounded in
auxiliary knowledge about entities, and facts constructed compositionally
through knowledge base walks. The resulting factuality evaluation model
achieves state-of-the-art performance on two in-domain news summarization
benchmarks as well as on three out-of-domain scientific literature datasets.
Further analysis of FactKB shows improved ability to detect erroneous entities
and relations in summaries and is robust and generalizable across domains.

文章提出了一种新的事实评估方法 FactKB，该方法使用基于预先抽取的实体知识的语言模型，以达到跨领域的通用性并解决现有模型在新领域中存在的实体和关系错误问题。通过在两个领域内的新闻汇总测试数据和三个跨领域的科学文献数据上测试，FactKB 的事实性评估模型达到了最先进的性能水平，并表现出在摘要中检测错误实体和关系的能力显著提高，从而进一步证明其在领域通用性和鲁棒性方面的特点。