Many text generation applications require the generated text to be factually
consistent with input information. Automatic evaluation of factual consistency
is challenging. Previous work has developed various metrics that often depend
on specific functions, such as natural language inference (NLI) or question
answering (QA), trained on limited data. Those metrics thus can hardly assess
diverse factual inconsistencies (e.g., contradictions, hallucinations) that
occur in varying inputs/outputs (e.g., sentences, documents) from different
tasks. In this paper, we propose AlignScore, a new holistic metric that applies
to a variety of factual inconsistency scenarios as above. AlignScore is based
on a general function of information alignment between two arbitrary text
pieces. Crucially, we develop a unified training framework of the alignment
function by integrating a large diversity of data sources, resulting in 4.7M
training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact
verification, information retrieval, semantic similarity, and summarization).
We conduct extensive experiments on large-scale benchmarks including 22
evaluation datasets, where 19 of the datasets were never seen in the alignment
training. AlignScore achieves substantial improvement over a wide range of
previous metrics. Moreover, AlignScore (355M parameters) matches or even
outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude
larger.

本文提出了一个新的综合指标 AlignScore，基于信息对齐来评估各种实际不一致性场景，包含了 7 个知名任务的 4.7M 数据来源的统一训练框架，重点在于我们进行了大量的实验，得出 AlignScore 在测试中的显著优势，且能够匹敌比其大数个数量级的 ChatGPT 和 GPT-4 等度量标准。