Despite tremendous improvements in natural language generation, summarization
models still suffer from the unfaithfulness issue. Previous work evaluates
faithfulness either using models trained on the other tasks or in-domain
synthetic data, or prompting a large model such as ChatGPT. This paper proposes
to do zero-shot faithfulness evaluation simply with a moderately-sized
foundation language model. We introduce a new metric FFLM, which is a
combination of probability changes based on the intuition that prefixing a
piece of text that is consistent with the output will increase the probability
of predicting the output. Experiments show that FFLM performs competitively
with or even outperforms ChatGPT on both inconsistency detection and
faithfulness rating with 24x fewer parameters. FFLM also achieves improvements
over other strong baselines.

提出一种新指标 FFLM，结合基于概率变化的方法评估生成模型的忠实度，相比于强基准模型 ChatGPT 拥有更少的参数，并在不一致性检测和忠实度评分方面表现出竞争性和优越性的改进。