Positional Encodings (PEs) are used to inject word-order information into
transformer-based language models. While they can significantly enhance the
quality of sentence representations, their specific contribution to language
models is not fully understood, especially given recent findings that various
positional encodings are insensitive to word order. In this work, we conduct a
systematic study of positional encodings in \textbf{Bidirectional Masked
Language Models} (BERT-style) , which complements existing work in three
aspects: (1) We uncover the core function of PEs by identifying two common
properties, Locality and Symmetry; (2) We show that the two properties are
closely correlated with the performances of downstream tasks; (3) We quantify
the weakness of current PEs by introducing two new probing tasks, on which
current PEs perform poorly. We believe that these results are the basis for
developing better PEs for transformer-based language models. The code is
available at \faGithub~ https://github.com/tigerchen52/locality\_symmetry

位置编码（PEs）用于将单词顺序信息注入基于 Transformer 的语言模型中。本研究对双向遮蔽语言模型（BERT-style）中的位置编码进行了系统研究，揭示了其核心功能和与下游任务性能相关的两个常见特性（局部性和对称性），并对当前位置编码的弱点进行了定量评估。这些结果可为开发基于 Transformer 的语言模型的更好位置编码提供基础。