Adam with decoupled weight decay, also known as AdamW, is widely acclaimed
for its superior performance in language modeling tasks, surpassing Adam with
$\ell_2$ regularization in terms of generalization and optimization. However,
this advantage is not theoretically well-understood. One challenge here is that
though intuitively Adam with $\ell_2$ regularization optimizes the $\ell_2$
regularized loss, it is not clear if AdamW optimizes a specific objective. In
this work, we make progress toward understanding the benefit of AdamW by
showing that it implicitly performs constrained optimization. More concretely,
we show in the full-batch setting, if AdamW converges with any non-increasing
learning rate schedule whose partial sum diverges, it must converge to a KKT
point of the original loss under the constraint that the $\ell_\infty$ norm of
the parameter is bounded by the inverse of the weight decay factor. This result
is built on the observation that Adam can be viewed as a smoothed version of
SignGD, which is the normalized steepest descent with respect to $\ell_\infty$
norm, and a surprising connection between normalized steepest descent with
weight decay and Frank-Wolfe.

AdamW 在语言建模任务中表现出优越的性能，优于具有正则化 l2 项的 Adam 算法，本文通过分析表明 AdamW 隐式地进行了约束优化。

AdamW 的隐式偏差: l∞范数约束优化

Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization

Lion (Evolved Sign Momentum), a new optimizer discovered through program
search, has shown promising results in training large AI models. It performs
comparably or favorably to AdamW but with greater memory efficiency. As we can
expect from the results of a random search program, Lion incorporates elements
from several existing algorithms, including signed momentum, decoupled weight
decay, Polak, and Nesterov momentum, but does not fit into any existing
category of theoretically grounded optimizers. Thus, even though Lion appears
to perform well as a general-purpose optimizer for a wide range of tasks, its
theoretical basis remains uncertain. This lack of theoretical clarity limits
opportunities to further enhance and expand Lion's efficacy.
This work aims to demystify Lion. Based on both continuous-time and
discrete-time analysis, we demonstrate that Lion is a theoretically novel and
principled approach for minimizing a general loss function $f(x)$ while
enforcing a bound constraint $\|x\|_\infty \leq 1/\lambda$. Lion achieves this
through the incorporation of decoupled weight decay, where $\lambda$ represents
the weight decay coefficient. Our analysis is made possible by the development
of a new Lyapunov function for the Lion updates. It applies to a broader family
of Lion-$\kappa$ algorithms, where the $\text{sign}(\cdot)$ operator in Lion is
replaced by the subgradient of a convex function $\kappa$, leading to the
solution of a general composite optimization problem of $\min_x f(x) +
\kappa^*(x)$. Our findings provide valuable insights into the dynamics of Lion
and pave the way for further improvements and extensions of Lion-related
algorithms.

Lion 是一种优化器模型，通过程序搜索发现，它在训练大型 AI 模型上展示了有希望的结果，在内存效率上与 AdamW 相当或更好。通过连续时间和离散时间分析，我们证明了 Lion 在最小化一般损失函数并强制执行边界约束时是理论上新颖和有原则的方法， LION 综合了多种现有算法的要素，但其理论基础仍然不确定。