Vision transformers using self-attention or its proposed alternatives have
demonstrated promising results in many image related tasks. However, the
underpinning inductive bias of attention is not well understood. To address
this issue, this paper analyzes attention through the lens of convex duality.
For the non-linear dot-product self-attention, and alternative mechanisms such
as MLP-mixer and Fourier Neural Operator (FNO), we derive equivalent
finite-dimensional convex problems that are interpretable and solvable to
global optimality. The convex programs lead to {\it block nuclear-norm
regularization} that promotes low rank in the latent feature and token
dimensions. In particular, we show how self-attention networks implicitly
clusters the tokens, based on their latent similarity. We conduct experiments
for transferring a pre-trained transformer backbone for CIFAR-100
classification by fine-tuning a variety of convex attention heads. The results
indicate the merits of the bias induced by attention compared with the existing
MLP or linear heads.

本文通过凸对偶性的视角分析了自我注意力及其替代方案的基础归纳偏差问题，并为非线性点积自我注意力和替代机制提出了有解释性且可全局最优求解的等效有限维凸问题。通过实验证明，与现有的 MLP 或线性 heuads 相比，自我注意网络的偏差对于 CIFAR-100 分类任务具有吸引力。

透过凸优化解释注意力：关于视觉 Transformer 的分析和解读

Unraveling Attention via Convex Duality: Analysis and Interpretations of  Vision Transformers

The families of $f$-divergences (e.g. the Kullback-Leibler divergence) and
Integral Probability Metrics (e.g. total variation distance or maximum mean
discrepancies) are widely used to quantify the similarity between probability
distributions. In this work, we systematically study the relationship between
these two families from the perspective of convex duality. Starting from a
tight variational representation of the $f$-divergence, we derive a
generalization of the moment-generating function, which we show exactly
characterizes the best lower bound of the $f$-divergence as a function of a
given IPM. Using this characterization, we obtain new bounds while also
recovering in a unified manner well-known results, such as Hoeffding's lemma,
Pinsker's inequality and its extension to subgaussian functions, and the
Hammersley-Chapman-Robbins bound. This characterization also allows us to prove
new results on topological properties of the divergence which may be of
independent interest.

该研究系统研究了从凸性对偶的角度出发将 $f$-divergences 和 Integral Probability Metrics 两个家族联系在一起的关系，派生出一种广义的矩生成函数并在此基础上得出了许多针对 $f$-divergences 的新的下限。此外，研究还证明了衍生出的这种下限的不同拓扑性质。

在 $f$- 散度和积分概率度量之间的最优界限

Optimal Bounds between $f$-Divergences and Integral Probability Metrics

We review basic concepts of convex duality, focusing on the very general and
supremely useful Fenchel-Rockafellar duality. We summarize how this duality may
be applied to a variety of reinforcement learning (RL) settings, including
policy evaluation or optimization, online or offline learning, and discounted
or undiscounted rewards. The derivations yield a number of intriguing results,
including the ability to perform policy evaluation and on-policy policy
gradient with behavior-agnostic offline data and methods to learn a policy via
max-likelihood optimization. Although many of these results have appeared
previously in various forms, we provide a unified treatment and perspective on
these results, which we hope will enable researchers to better use and apply
the tools of convex duality to make further progress in RL.

本论文主要探讨凸对偶性的基本概念，重点关注最通用且超级有用的 Fenchel-Rockafellar 对偶性，总结了如何将此对偶性应用于各种强化学习（RL）场景中，包括策略评估或优化、在线或离线学习，以及折现或未折现奖励。