Vision transformers using self-attention or its proposed alternatives have
demonstrated promising results in many image related tasks. However, the
underpinning inductive bias of attention is not well understood. To address
this issue, this paper analyzes attention through the lens of convex duality.
For the non-linear dot-product self-attention, and alternative mechanisms such
as MLP-mixer and Fourier Neural Operator (FNO), we derive equivalent
finite-dimensional convex problems that are interpretable and solvable to
global optimality. The convex programs lead to {\it block nuclear-norm
regularization} that promotes low rank in the latent feature and token
dimensions. In particular, we show how self-attention networks implicitly
clusters the tokens, based on their latent similarity. We conduct experiments
for transferring a pre-trained transformer backbone for CIFAR-100
classification by fine-tuning a variety of convex attention heads. The results
indicate the merits of the bias induced by attention compared with the existing
MLP or linear heads.

本文通过凸对偶性的视角分析了自我注意力及其替代方案的基础归纳偏差问题，并为非线性点积自我注意力和替代机制提出了有解释性且可全局最优求解的等效有限维凸问题。通过实验证明，与现有的 MLP 或线性 heuads 相比，自我注意网络的偏差对于 CIFAR-100 分类任务具有吸引力。

透过凸优化解释注意力：关于视觉 Transformer 的分析和解读

Unraveling Attention via Convex Duality: Analysis and Interpretations of  Vision Transformers

We propose the Limited Multi-Label (LML) projection layer as a new primitive
operation for end-to-end learning systems. The LML layer provides a
probabilistic way of modeling multi-label predictions limited to having exactly
k labels. We derive efficient forward and backward passes for this layer and
show how the layer can be used to optimize the top-k recall for multi-label
tasks with incomplete label information. We evaluate LML layers on top-k
CIFAR-100 classification and scene graph generation. We show that LML layers
add a negligible amount of computational overhead, strictly improve the model's
representational capacity, and improve accuracy. We also revisit the truncated
top-k entropy method as a competitive baseline for top-k classification.

本文提出了 Limited Multi-Label（LML）投影层作为端到端学习系统的一个新的原语操作，这个层提供了一个概率建模有 k 个标签的多标签预测的方法，并提供了 LML 层的前向和反向传递的有效方法，我们在 top-k CIFAR-100 分类和场景图生成上评估 LML 层，并表明 LML 层增加的计算开销可忽略不计，严格提高了模型的表示能力和准确性，同时重新考虑了截断的 top-k 熵方法作为 top-k 分类的竞争基线。