To capture user preference, transformer models have been widely applied to
model sequential user behavior data. The core of transformer architecture lies
in the self-attention mechanism, which computes the pairwise attention scores
in a sequence. Due to the permutation-equivariant nature, positional encoding
is used to enhance the attention between token representations. In this
setting, the pairwise attention scores can be derived by both semantic
difference and positional difference. However, prior studies often model the
two kinds of difference measurements in different ways, which potentially
limits the expressive capacity of sequence modeling. To address this issue,
this paper proposes a novel transformer variant with complex vector attention,
named EulerFormer, which provides a unified theoretical framework to formulate
both semantic difference and positional difference. The EulerFormer involves
two key technical improvements. First, it employs a new transformation function
for efficiently transforming the sequence tokens into polar-form complex
vectors using Euler's formula, enabling the unified modeling of both semantic
and positional information in a complex rotation form.Secondly, it develops a
differential rotation mechanism, where the semantic rotation angles can be
controlled by an adaptation function, enabling the adaptive integration of the
semantic and positional information according to the semantic
contexts.Furthermore, a phase contrastive learning task is proposed to improve
the anisotropy of contextual representations in EulerFormer. Our theoretical
framework possesses a high degree of completeness and generality. It is more
robust to semantic variations and possesses moresuperior theoretical properties
in principle. Extensive experiments conducted on four public datasets
demonstrate the effectiveness and efficiency of our approach.

该研究提出了一种名为 EulerFormer 的新型变体 Transformer 模型，利用复向量注意力机制动态地整合语义差异和位置差异，并通过相位对比学习任务来改善上下文表示的各向异性。在四个公共数据集上进行的广泛实验证明了该方法的有效性和效率。