Self-attention is key to the remarkable success of transformers in sequence
modeling tasks including many applications in natural language processing and
computer vision. Like neural network layers, these attention mechanisms are
often developed by heuristics and experience. To provide a principled framework
for constructing attention layers in transformers, we show that the
self-attention corresponds to the support vector expansion derived from a
support vector regression problem, whose primal formulation has the form of a
neural network layer. Using our framework, we derive popular attention layers
used in practice and propose two new attentions: 1) the Batch Normalized
Attention (Attention-BN) derived from the batch normalization layer and 2) the
Attention with Scaled Head (Attention-SH) derived from using less training data
to fit the SVR model. We empirically demonstrate the advantages of the
Attention-BN and Attention-SH in reducing head redundancy, increasing the
model's accuracy, and improving the model's efficiency in a variety of
practical applications including image and time-series classification.

自注意力机制是 transformers 在序列建模任务中取得卓越成功的关键，本研究提出了一种基于支持向量回归问题的自注意力构建方法，推导出常用的注意力层，并提出了两种新型注意力机制：1) 批正则注意力，2) 缩放头注意力，通过实证研究证明这两种注意力机制在提高模型准确性和效率方面的优势。