Pairwise dot-product self-attention is key to the success of transformers
that achieve state-of-the-art performance across a variety of applications in
language and vision. This dot-product self-attention computes attention weights
among the input tokens using Euclidean distance, which makes the model prone to
representation collapse and vulnerable to contaminated samples. In this paper,
we propose using a Mahalanobis distance metric for computing the attention
weights to stretch the underlying feature space in directions of high
contextual relevance. In particular, we define a hyper-ellipsoidal neighborhood
around each query to increase the attention weights of the tokens lying in the
contextually important directions. We term this novel class of attention
Elliptical Attention. Our Elliptical Attention provides two benefits: 1)
reducing representation collapse and 2) enhancing the model's robustness as the
Elliptical Attention pays more attention to contextually relevant information
rather than focusing on some small subset of informative features. We
empirically demonstrate the advantages of Elliptical Attention over the
baseline dot-product attention and state-of-the-art attention methods on
various practical tasks, including object classification, image segmentation,
and language modeling across different data modalities.

应用 Mahalanobis 距离计算注意力权重，将模型的特征空间在高上下文相关性方向进行拉伸，从而达到减少特征塌缩和提升模型鲁棒性的效果，验证了在多个实际任务中 Elliptical Attention 相对于基线 dot-product attention 和最先进的方法的优越性。