Large language models have been shown to behave inconsistently in response to
meaning-preserving paraphrastic inputs. At the same time, researchers evaluate
the knowledge and reasoning abilities of these models with test evaluations
that do not disaggregate the effect of paraphrastic variability on performance.
We propose a metric for evaluating the paraphrastic consistency of natural
language reasoning models based on the probability of a model achieving the
same correctness on two paraphrases of the same problem. We mathematically
connect this metric to the proportion of a model's variance in correctness
attributable to paraphrasing. To estimate paraphrastic consistency, we collect
ParaNLU, a dataset of 7,782 human-written and validated paraphrased reasoning
problems constructed on top of existing benchmark datasets for defeasible and
abductive natural language inference. Using ParaNLU, we measure the
paraphrastic consistency of several model classes and show that consistency
dramatically increases with pretraining but not finetuning. All models tested
exhibited room for improvement in paraphrastic consistency.

大型语言模型在对意义保持改写的输入作出回应时表现不一致。我们提出了一种度量自然语言推理模型改写一致性的评估指标，该指标基于模型在同一问题的两个改写上的正确性概率，并将其与改写相关的正确性变化的比例数学上相连。通过收集 ParaNLU 数据集，我们测量了几种模型类别的改写一致性，并展示了一致性随预训练而显著增加，而微调则没有。所有测试模型在改写一致性方面都有改进的空间。