As the rapid progression of practical applications based on Large Language
Models continues, the importance of extrapolating performance has grown
exponentially in the research domain. In our study, we identified an anomalous
behavior in Transformer models that had been previously overlooked, leading to
a chaos around closest tokens which carried the most important information.
We've coined this discovery the "headache of Transformers". To address this at
its core, we introduced a novel self-attention structure named Collinear
Constrained Attention (CoCA). This structure can be seamlessly integrated with
existing extrapolation, interpolation methods, and other optimization
strategies designed for traditional Transformer models. We have achieved
excellent extrapolating performance even for 16 times to 24 times of sequence
lengths during inference without any fine-tuning on our model. We have also
enhanced CoCA's computational and spatial efficiency to ensure its
practicality. We plan to open-source CoCA shortly. In the meantime, we've made
our code available in the appendix for reappearing experiments.

我们的研究发现了 Transformer 模型中一个被忽视的异常行为，称之为 “Transformers 的头痛”，并引入了一种名为 Collinear Constrained Attention (CoCA) 的新型自注意结构，以解决该问题，并实现优秀的外推性能，同时提高了 CoCA 的计算和空间效率。