Word order is an important concept in natural language, and in this work, we
study how word order affects the induction of world knowledge from raw text
using language models. We use word analogies to probe for such knowledge.
Specifically, in addition to the natural word order, we first respectively
extract texts of six fixed word orders from five languages and then pretrain
the language models on these texts. Finally, we analyze the experimental
results of the fixed word orders on word analogies and show that i) certain
fixed word orders consistently outperform or underperform others, though the
specifics vary across languages, and ii) the Wov2Lex hypothesis is not hold in
pre-trained language models, and the natural word order typically yields
mediocre results. The source code will be made publicly available at
this https URL

研究通过语言模型从原始文本中诱导世界知识时，词序对其的影响，使用词类比来探究这种知识。从五种语言分别提取六种固定词序的文本，对这些文本进行预训练。最终，分析了固定词序在词类比任务上的实验结果，发现某些特定的固定词序始终优于或不如其他词序，并且这些具体情况因语言而异。另外，发现 Wov2Lex 假设在预训练语言模型中不成立，而自然的词序通常导致中等水平的结果。源代码将在以下网址公开。