BriefGPT.xyz
Feb, 2025
学习RoPEs:利用STRING改进2D和3D位置编码
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
HTML
PDF
Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie...
TL;DR
本研究解决了在大型语言模型中使用的旋转位置编码的局限性,提出了一种新的可分离平移不变位置编码方法STRING。STRING在保持低计算开销的同时,提供了精确的平移不变性,并在视觉变换器中应用,尤其在开放词汇物体检测和机器人控制方面实现了显著提升。
Abstract
We introduce STRING: Separable Translationally Invariant
Position Encodings
. STRING extends Rotary
Position Encodings
, a recently proposed and widely used algorithm in large language models, via a unifying theore
→