Despite the success of the Adam optimizer in practice, the theoretical
understanding of its algorithmic components still remains limited. In
particular, most existing analyses of Adam show the convergence rate that can
be simply achieved by non-adative algorithms like SGD. In this work, we provide
a different perspective based on online learning that underscores the
importance of Adam's algorithmic components. Inspired by Cutkosky et al.
(2023), we consider the framework called online learning of updates, where we
choose the updates of an optimizer based on an online learner. With this
framework, the design of a good optimizer is reduced to the design of a good
online learner. Our main observation is that Adam corresponds to a principled
online learning framework called Follow-the-Regularized-Leader (FTRL). Building
on this observation, we study the benefits of its algorithmic components from
the online learning perspective.

本研究从在线学习的角度出发，对 Adam 优化器的算法组成进行了研究，通过研究 Adam 与 Follow-the-Regularized-Leader 在线学习框架的对应关系，揭示了其算法组成的益处。

通过在线学习更新理解 Adam 优化器：Adam 即 FTRL 的伪装

Understanding Adam Optimizer via Online Learning of Updates: Adam is  FTRL in Disguise

The integration of algorithmic components into neural architectures has
gained increased attention recently, as it allows training neural networks with
new forms of supervision such as ordering constraints or silhouettes instead of
using ground truth labels. Many approaches in the field focus on the continuous
relaxation of a specific task and show promising results in this context. But
the focus on single tasks also limits the applicability of the proposed
concepts to a narrow range of applications. In this work, we build on those
ideas to propose an approach that allows to integrate algorithms into
end-to-end trainable neural network architectures based on a general
approximation of discrete conditions. To this end, we relax these conditions in
control structures such as conditional statements, loops, and indexing, so that
resulting algorithms are smoothly differentiable. To obtain meaningful
gradients, each relevant variable is perturbed via logistic distributions and
the expectation value under this perturbation is approximated. We evaluate the
proposed continuous relaxation model on four challenging tasks and show that it
can keep up with relaxations specifically designed for each individual task.

本文提出了一种将算法组件集成到端到端可训练的神经网络体系结构中的方法，通过条件语句、循环和索引来松弛离散条件，获得连续可微的算法。结果表明，所提出的连续松弛模型可以应用于多种任务。

通过连续松弛进行算法监督的学习

Learning with Algorithmic Supervision via Continuous Relaxations

Recent research has seen several advances relevant to black-box VI, but the
current state of automatic posterior inference is unclear. One such advance is
the use of normalizing flows to define flexible posterior densities for deep
latent variable models. Another direction is the integration of Monte-Carlo
methods to serve two purposes; first, to obtain tighter variational objectives
for optimization, and second, to define enriched variational families through
sampling. However, both flows and variational Monte-Carlo methods remain
relatively unexplored for black-box VI. Moreover, on a pragmatic front, there
are several optimization considerations like step-size scheme, parameter
initialization, and choice of gradient estimators, for which there are no clear
guidance in the existing literature. In this paper, we postulate that black-box
VI is best addressed through a careful combination of numerous algorithmic
components. We evaluate components relating to optimization, flows, and
Monte-Carlo methods on a benchmark of 30 models from the Stan model library.
The combination of these algorithmic components significantly advances the
state-of-the-art "out of the box" variational inference.

通过在 30 个 Stan 模型库中的基准测试中评估与优化，normalizing flows 和 Monte-Carlo 方法的组合显着推进了黑盒 VI 的状态。