Pairwise dot-product self-attention is key to the success of transformers
that achieve state-of-the-art performance across a variety of applications in
language and vision. This dot-product self-attention computes attention weights
among the input tokens using Euclidean distance, which makes the model prone to
representation collapse and vulnerable to contaminated samples. In this paper,
we propose using a Mahalanobis distance metric for computing the attention
weights to stretch the underlying feature space in directions of high
contextual relevance. In particular, we define a hyper-ellipsoidal neighborhood
around each query to increase the attention weights of the tokens lying in the
contextually important directions. We term this novel class of attention
Elliptical Attention. Our Elliptical Attention provides two benefits: 1)
reducing representation collapse and 2) enhancing the model's robustness as the
Elliptical Attention pays more attention to contextually relevant information
rather than focusing on some small subset of informative features. We
empirically demonstrate the advantages of Elliptical Attention over the
baseline dot-product attention and state-of-the-art attention methods on
various practical tasks, including object classification, image segmentation,
and language modeling across different data modalities.

应用 Mahalanobis 距离计算注意力权重，将模型的特征空间在高上下文相关性方向进行拉伸，从而达到减少特征塌缩和提升模型鲁棒性的效果，验证了在多个实际任务中 Elliptical Attention 相对于基线 dot-product attention 和最先进的方法的优越性。

椭圆形注意力

Elliptical Attention

Most sparse linear representation-based trackers need to solve a
computationally expensive L1-regularized optimization problem. To address this
problem, we propose a visual tracker based on non-sparse linear
representations, which admit an efficient closed-form solution without
sacrificing accuracy. Moreover, in order to capture the correlation information
between different feature dimensions, we learn a Mahalanobis distance metric in
an online fashion and incorporate the learned metric into the optimization
problem for obtaining the linear representation. We show that online metric
learning using proximity comparison significantly improves the robustness of
the tracking, especially on those sequences exhibiting drastic appearance
changes. Furthermore, in order to prevent the unbounded growth in the number of
training samples for the metric learning, we design a time-weighted reservoir
sampling method to maintain and update limited-sized foreground and background
sample buffers for balancing sample diversity and adaptability. Experimental
results on challenging videos demonstrate the effectiveness and robustness of
the proposed tracker.

提出一种基于非稀疏线性表示的视觉跟踪器，采用有效的闭式解决方案解决了计算昂贵的 L1 正则化优化问题，并采用在线度量学习来捕捉不同特征维度之间的相关信息，从而获得更好的跟踪效果。

基于在线水库度量学习的视觉跟踪非稀疏线性表示

Non-sparse Linear Representations for Visual Tracking with Online  Reservoir Metric Learning

For many machine learning algorithms such as $k$-Nearest Neighbor ($k$-NN)
classifiers and $ k $-means clustering, often their success heavily depends on
the metric used to calculate distances between different data points.
An effective solution for defining such a metric is to learn it from a set of
labeled training samples. In this work, we propose a fast and scalable
algorithm to learn a Mahalanobis distance metric. By employing the principle of
margin maximization to achieve better generalization performances, this
algorithm formulates the metric learning as a convex optimization problem and a
positive semidefinite (psd) matrix is the unknown variable. a specialized
gradient descent method is proposed. our algorithm is much more efficient and
has a better performance in scalability compared with existing methods.
Experiments on benchmark data sets suggest that, compared with state-of-the-art
metric learning algorithms, our algorithm can achieve a comparable
classification accuracy with reduced computational complexity.

本文提出一种快速且可扩展的机器学习算法，用于学习马氏距离指标，并利用凸优化原理和梯度下降方法解决距离度量问题，实现在计算复杂性方面的显著提高，达到与现有方法相当的分类准确度。

可扩展的大边界马氏距离度量学习

Scalable Large-Margin Mahalanobis Distance Metric Learning

The learning of appropriate distance metrics is a critical problem in image
classification and retrieval. In this work, we propose a boosting-based
technique, termed \BoostMetric, for learning a Mahalanobis distance metric. One
of the primary difficulties in learning such a metric is to ensure that the
Mahalanobis matrix remains positive semidefinite. Semidefinite programming is
sometimes used to enforce this constraint, but does not scale well.
\BoostMetric is instead based on a key observation that any positive
semidefinite matrix can be decomposed into a linear positive combination of
trace-one rank-one matrices. \BoostMetric thus uses rank-one positive
semidefinite matrices as weak learners within an efficient and scalable
boosting-based learning process. The resulting method is easy to implement,
does not require tuning, and can accommodate various types of constraints.
Experiments on various datasets show that the proposed algorithm compares
favorably to those state-of-the-art methods in terms of classification accuracy
and running time.

本文提出了一种基于提升法的技术，称为 BoostMetric，用于学习 Mahalanobis 距离度量标准，通过提升算法中有效和可扩展的学习过程，缓解了学习这种度量标准的主要困难之一，即确保 Mahalanobis 矩阵是正定半定的。在各种数据集上的实验表明，该算法在分类准确性和运行时间方面优于那些最先进的方法。