Recent studies have shown that language models pretrained and/or fine-tuned
on randomly permuted sentences exhibit competitive performance on GLUE, putting
into question the importance of word order information. Somewhat
counter-intuitively, some of these studies also report that position embeddings
appear to be crucial for models' good performance with shuffled text. We probe
these language models for word order information and investigate what position
embeddings learned from shuffled text encode, showing that these models retain
information pertaining to the original, naturalistic word order. We show this
is in part due to a subtlety in how shuffling is implemented in previous work
-- before rather than after subword segmentation. Surprisingly, we find even
Language models trained on text shuffled after subword segmentation retain some
semblance of information about word order because of the statistical
dependencies between sentence length and unigram probabilities. Finally, we
show that beyond GLUE, a variety of language understanding tasks do require
word order information, often to an extent that cannot be learned through
fine-tuning.

研究发现，经过随即排列的句子进行预训练和 / 或微调的语言模型表现出与 GLUE 上竞争力的表现，这给单词顺序信息的重要性带来了质疑。尽管有一些研究表明位置嵌入对于模型在混乱的文本上表现良好似乎很让人费解，但我们对这些语言模型进行了单词顺序信息的探究并研究了从混淆的文本中学习的位置嵌入编码的内容，表明这些模型保留了与原始的自然顺序有关的信息。我们表明，这在某种程度上是由于先前的研究中所实现的混淆的微妙之处 —— 而不是在子单词分段之后而是在之前。但令人惊讶的是，我们发现即使是在子单词分段之后洗牌的文本上训练出的语言模型也保留了单词顺序信息的某些方面，因为句子长度和单字概率之间存在统计依赖关系。最后，我们证明除了 GLUE，各种语言理解任务确实需要单词顺序信息，这往往不能通过微调来学习。