Assessing the factual consistency of automatically generated texts in
relation to source context is crucial for developing reliable natural language
generation applications. Recent literature proposes AlignScore which uses a
unified alignment model to evaluate factual consistency and substantially
outperforms previous methods across many benchmark tasks. In this paper, we
take a closer look of datasets used in AlignScore and uncover an unexpected
finding: utilizing a smaller number of data points can actually improve
performance. We process the original AlignScore training dataset to remove
noise, augment with robustness-enhanced samples, and utilize a subset
comprising 10\% of the data to train an improved factual consistency evaluation
model, we call LIM-RA (Less Is More for Robust AlignScore). LIM-RA demonstrates
superior performance, consistently outperforming AlignScore and other strong
baselines like ChatGPT across four benchmarks (two utilizing traditional
natural language generation datasets and two focused on large language model
outputs). Our experiments show that LIM-RA achieves the highest score on 24 of
the 33 test datasets, while staying competitive on the rest, establishing the
new state-of-the-art benchmarks.

我们研究了 AlignScore 数据集，并发现使用较少的数据点实际上可以提高性能，因此我们使用 10％的数据集训练了一个改进的模型 LIM-RA，该模型在四个基准测试中表现出卓越的性能，成为新的最先进基准模型。

简洁有利于提高事实一致性的自动评价

Less is More for Improving Automatic Evaluation of Factual Consistency

Many text generation applications require the generated text to be factually
consistent with input information. Automatic evaluation of factual consistency
is challenging. Previous work has developed various metrics that often depend
on specific functions, such as natural language inference (NLI) or question
answering (QA), trained on limited data. Those metrics thus can hardly assess
diverse factual inconsistencies (e.g., contradictions, hallucinations) that
occur in varying inputs/outputs (e.g., sentences, documents) from different
tasks. In this paper, we propose AlignScore, a new holistic metric that applies
to a variety of factual inconsistency scenarios as above. AlignScore is based
on a general function of information alignment between two arbitrary text
pieces. Crucially, we develop a unified training framework of the alignment
function by integrating a large diversity of data sources, resulting in 4.7M
training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact
verification, information retrieval, semantic similarity, and summarization).
We conduct extensive experiments on large-scale benchmarks including 22
evaluation datasets, where 19 of the datasets were never seen in the alignment
training. AlignScore achieves substantial improvement over a wide range of
previous metrics. Moreover, AlignScore (355M parameters) matches or even
outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude
larger.

本文提出了一个新的综合指标 AlignScore，基于信息对齐来评估各种实际不一致性场景，包含了 7 个知名任务的 4.7M 数据来源的统一训练框架，重点在于我们进行了大量的实验，得出 AlignScore 在测试中的显著优势，且能够匹敌比其大数个数量级的 ChatGPT 和 GPT-4 等度量标准。