Language models (LMs) may appear insensitive to word order changes in natural
language understanding (NLU) tasks. In this paper, we propose that linguistic
redundancy can explain this phenomenon, whereby word order and other linguistic
cues such as case markers provide overlapping and thus redundant information.
Our hypothesis is that models exhibit insensitivity to word order when the
order provides redundant information, and the degree of insensitivity varies
across tasks. We quantify how informative word order is using mutual
information (MI) between unscrambled and scrambled sentences. Our results show
the effect that the less informative word order is, the more consistent the
model's predictions are between unscrambled and scrambled sentences. We also
find that the effect varies across tasks: for some tasks, like SST-2, LMs'
prediction is almost always consistent with the original one even if the
Pointwise-MI (PMI) changes, while for others, like RTE, the consistency is near
random when the PMI gets lower, i.e., word order is really important.

通过研究语言模型的敏感度，提出语言冗余可以解释语言模型对自然语言理解任务中的词序变化不敏感的现象，即词序和其他语言线索（如格标记）提供了重叠和冗余的信息。利用互信息量化词序的信息量，结果显示当词序信息量较低时，模型对未乱序和乱序句子的预测更加一致。此外，发现该影响在不同任务中有所变化，对于某些任务（如 SST-2），语言模型的预测与原始预测几乎始终一致，尽管点互信息（PMI）发生变化，而对于其他任务（如 RTE），当 PMI 较低时，预测的一致性接近随机，即词序真正重要。