Self-attention-based networks have achieved remarkable performance in
sequential recommendation tasks. A crucial component of these models is
positional encoding. In this study, we delve into the learned positional
embedding, demonstrating that it often captures the distance between tokens.
Building on this insight, we introduce novel attention models that directly
learn positional relations. Extensive experiments reveal that our proposed
models, \textbf{PARec} and \textbf{FPARec} outperform previous
self-attention-based approaches.Our code is available at the link for anonymous
review: this https URL r/FPARec-2C55/

自注意力机制网络在顺序推荐任务中取得了显著的性能表现。本研究探讨了学习到的位置编码，发现其通常捕捉到令牌之间的距离。基于这一发现，我们提出了直接学习位置关系的新型注意力模型。大量实验证明了我们提出的模型 PARec 和 FPARec 优于之前的自注意力方法。

序列推荐中的位置注意力学习

Learning Positional Attention for Sequential Recommendation

Recent advances on Vision Transformer (ViT) and its improved variants have
shown that self-attention-based networks surpass traditional Convolutional
Neural Networks (CNNs) in most vision tasks. However, existing ViTs focus on
the standard accuracy and computation cost, lacking the investigation of the
intrinsic influence on model robustness and generalization. In this work, we
conduct systematic evaluation on components of ViTs in terms of their impact on
robustness to adversarial examples, common corruptions and distribution shifts.
We find some components can be harmful to robustness. By using and combining
robust components as building blocks of ViTs, we propose Robust Vision
Transformer (RVT), which is a new vision transformer and has superior
performance with strong robustness. We further propose two new plug-and-play
techniques called position-aware attention scaling and patch-wise augmentation
to augment our RVT, which we abbreviate as RVT*. The experimental results on
ImageNet and six robustness benchmarks show the advanced robustness and
generalization ability of RVT compared with previous ViTs and state-of-the-art
CNNs. Furthermore, RVT-S* also achieves Top-1 rank on multiple robustness
leaderboards including ImageNet-C and ImageNet-Sketch. The code will be
available at https://github.com/alibaba/easyrobust.

研究发现，很多 ViT 组件对鲁棒性有害，因此提出使用鲁棒组件构建 Robust Vision Transformer（RVT）网络，并进一步提出 position-aware attention scaling 和 patch-wise augmentation 两种方法增强其性能，实验结果显示 RVT 在多项鲁棒性测试中表现优秀。