Factual consistency is an essential quality of text summarization models in
practical settings. Existing work in evaluating this dimension can be broadly
categorized into two lines of research, entailment-based and question answering
(QA)-based metrics, and different experimental setups often lead to contrasting
conclusions as to which paradigm performs the best. In this work, we conduct an
extensive comparison of entailment and QA-based metrics, demonstrating that
carefully choosing the components of a QA-based metric, especially question
generation and answerability classification, is critical to performance.
Building on those insights, we propose an optimized metric, which we call
QAFactEval, that leads to a 14% average improvement over previous QA-based
metrics on the SummaC factual consistency benchmark, and also outperforms the
best-performing entailment-based metric. Moreover, we find that QA-based and
entailment-based metrics can offer complementary signals and be combined into a
single metric for a further performance boost.

本文针对文本摘要模型中的事实一致性问题展开研究，对比了基于蕴含和基于问答的度量方法，并且提出了一种优化的基于问答的度量方法 QAFactEval，相较之前的方法平均提高了 14% 的 SummaC 事实一致性基准测试性能，并且在最好的基于蕴含的方法之上表现更佳。此外，我们发现基于问答和基于蕴含的度量方法可以互补，结合起来可以提高摘要模型的性能