Vision transformers using self-attention or its proposed alternatives have
demonstrated promising results in many image related tasks. However, the
underpinning inductive bias of attention is not well understood. To address
this issue, this paper analyzes attention through the lens of convex duality.
For the non-linear dot-product self-attention, and alternative mechanisms such
as MLP-mixer and Fourier Neural Operator (FNO), we derive equivalent
finite-dimensional convex problems that are interpretable and solvable to
global optimality. The convex programs lead to {\it block nuclear-norm
regularization} that promotes low rank in the latent feature and token
dimensions. In particular, we show how self-attention networks implicitly
clusters the tokens, based on their latent similarity. We conduct experiments
for transferring a pre-trained transformer backbone for CIFAR-100
classification by fine-tuning a variety of convex attention heads. The results
indicate the merits of the bias induced by attention compared with the existing
MLP or linear heads.

本文通过凸对偶性的视角分析了自我注意力及其替代方案的基础归纳偏差问题，并为非线性点积自我注意力和替代机制提出了有解释性且可全局最优求解的等效有限维凸问题。通过实验证明，与现有的 MLP 或线性 heuads 相比，自我注意网络的偏差对于 CIFAR-100 分类任务具有吸引力。