Since its invention in 2014, the Adam optimizer has received tremendous
attention. On one hand, it has been widely used in deep learning and many
variants have been proposed, while on the other hand their theoretical
convergence property remains to be a mystery. It is far from satisfactory in
the sense that some studies require strong assumptions about the updates, which
are not necessarily applicable in practice, while other studies still follow
the original problematic convergence analysis of Adam, which was shown to be
not sufficient to ensure convergence. Although rigorous convergence analysis
exists for Adam, they impose specific requirements on the update of the
adaptive step size, which are not generic enough to cover many other variants
of Adam. To address theses issues, in this extended abstract, we present a
simple and generic proof of convergence for a family of Adam-style methods
(including Adam, AMSGrad, Adabound, etc.). Our analysis only requires an
increasing or large "momentum" parameter for the first-order moment, which is
indeed the case used in practice, and a boundness condition on the adaptive
factor of the step size, which applies to all variants of Adam under mild
conditions of stochastic gradients. We also establish a variance diminishing
result for the used stochastic gradient estimators. Indeed, our analysis of
Adam is so simple and generic that it can be leveraged to establish the
convergence for solving a broader family of non-convex optimization problems,
including min-max, compositional, and bilevel optimization problems. For the
full (earlier) version of this extended abstract, please refer to
arXiv:2104.14840.

本研究通过对 Adam 优化器家族进行分析，提出了一种适用于包括 min-max、组合和双层优化问题的简单且通用的渐进收敛证明方法，并证明了使用随机梯度估计器的方差减少结果。

Adam 系列算法的新收敛分析

A Novel Convergence Analysis for Algorithms of the Adam Family

We present a unified framework for analyzing local SGD methods in the convex
and strongly convex regimes for distributed/federated training of supervised
machine learning models. We recover several known methods as a special case of
our general framework, including Local-SGD/FedAvg, SCAFFOLD, and several
variants of SGD not originally designed for federated learning. Our framework
covers both the identical and heterogeneous data settings, supports both random
and deterministic number of local steps, and can work with a wide array of
local stochastic gradient estimators, including shifted estimators which are
able to adjust the fixed points of local iterations for faster convergence. As
an application of our framework, we develop multiple novel FL optimizers which
are superior to existing methods. In particular, we develop the first linearly
converging local SGD method which does not require any data homogeneity or
other strong assumptions.

该论文提出了一种统一框架，用于在凸性和强凸性条件下分析本地 SGD 方法，适用于监督机器学习模型的分布式 / 联邦训练。作为该框架的应用，作者开发了多个新型优化器，特别是开发了第一个线性收敛的本地 SGD 方法，不需要任何数据同质性或其他强假设。

局部 SGD：统一理论和新高效方法

Local SGD: Unified Theory and New Efficient Methods

Recent variational inference methods use stochastic gradient estimators whose
variance is not well understood. Theoretical guarantees for these estimators
are important to understand when these methods will or will not work. This
paper gives bounds for the common "reparameterization" estimators when the
target is smooth and the variational family is a location-scale distribution.
These bounds are unimprovable and thus provide the best possible guarantees
under the stated assumptions.

本文通过研究在目标平滑、变分族为位置 - 尺度分布情况下的重参数估计器，为基于随机梯度估计器的最新变分推断方法提供了不可改善的界限。