In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.

利用自然语言序列的内在分割，设计了一种名为Bilevel Positional Encoding（BiPE）的新的位置编码方法。每个位置的BiPE结合了分段内编码和分段间编码，从而能够捕捉语义信息，并通过相对位置编码来改善外推能力。理论分析表明，位置信息的解缠可以使学习更加有效。实证结果还表明，BiPE在不同文本模态任务的广泛范围内具有出色的长度外推能力。

双石击鸟：双层位置编码实现更好的长度外推