BriefGPT.xyz
Sep, 2021
SHAPE: 基于位移的绝对位置编码用于Transformers
SHAPE: Shifted Absolute Position Embedding for Transformers
HTML
PDF
Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, Kentaro Inui
TL;DR
本论文通过随机平移绝对位置来构建shift invariance的position representation SHAPE,其计算效率和性能与其他方法相当且更为简单。
Abstract
position representation
is crucial for building position-aware representations in
transformers
. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high comp
→