While large pretrained language models (PLMs) demonstrate incredible fluency
and performance on many natural language tasks, recent work has shown that
well-performing PLMs are very sensitive to what prompts are feed into them.
Even when prompts are semantically identical, language models may give very
different answers. When considering safe and trustworthy deployments of PLMs we
would like their outputs to be consistent under prompts that mean the same
thing or convey the same intent. While some work has looked into how
state-of-the-art PLMs address this need, they have been limited to only
evaluating lexical equality of single- or multi-word answers and do not address
consistency of generative text sequences. In order to understand consistency of
PLMs under text generation settings, we develop a measure of semantic
consistency that allows the comparison of open-ended text outputs. We implement
several versions of this consistency metric to evaluate the performance of a
number of PLMs on paraphrased versions of questions in the TruthfulQA dataset,
we find that our proposed metrics are considerably more consistent than
traditional metrics embodying lexical consistency, and also correlate with
human evaluation of output consistency to a higher degree.

本研究针对更新的预训练语言模型（PLMs）在语义一致性方面的问题，提出了一种衡量语义一致性的度量标准，并在 TruthfulQA 数据集上评估多个 PLMs 的性能，发现我们提出的语义一致性度量标准比传统的基于词汇一致性的度量标准更可靠，也与人类评估输出一致性的程度更为相关。