Word order, an essential property of natural languages, is injected in
Transformer-based neural language models using position encoding. However,
recent experiments have shown that explicit position encoding is not always
useful, since some models without such feature managed to achieve state-of-the
art performance on some tasks. To understand better this phenomenon, we examine
the effect of removing position encodings on the pre-training objective itself
(i.e., masked language modelling), to test whether models can reconstruct
position information from co-occurrences alone. We do so by controlling the
amount of masked tokens in the input sentence, as a proxy to affect the
importance of position information for the task. We find that the necessity of
position information increases with the amount of masking, and that masked
language models without position encodings are not able to reconstruct this
information on the task. These findings point towards a direct relationship
between the amount of masking and the ability of Transformers to capture
order-sensitive aspects of language using position encoding.

本研究探讨了在 Transformer-based 神经语言模型中移除位置编码的影响，发现掩码语言建模任务中位置信息的重要性随着掩码数量的增加而增加，并且没有位置编码的模型不能完成该任务，这揭示了 Transformers 通过位置编码捕捉语言的顺序敏感方面的直接关系。