In previous literature, backward error analysis was used to find ordinary
differential equations (ODEs) approximating the gradient descent trajectory. It
was found that finite step sizes implicitly regularize solutions because terms
appearing in the ODEs penalize the two-norm of the loss gradients. We prove
that the existence of similar implicit regularization in RMSProp and Adam
depends on their hyperparameters and the training stage, but with a different
"norm" involved: the corresponding ODE terms either penalize the (perturbed)
one-norm of the loss gradients or, on the contrary, hinder its decrease (the
latter case being typical). We also conduct numerical experiments and discuss
how the proven facts can influence generalization.

前人的研究表明，通过反向误差分析可以找到逼近梯度下降轨迹的常微分方程（ODEs）。本文证明 RMSProp 和 Adam 中存在类似的隐式正则化现象，取决于超参数和训练阶段，并与之前的研究有所不同。我们还进行了数值实验，并讨论了这些事实如何影响泛化能力。

Adam 隐式偏差研究

On the Implicit Bias of Adam

Recent years have seen many insights on deep learning optimisation being
brought forward by finding implicit regularisation effects of commonly used
gradient-based optimisers. Understanding implicit regularisation can not only
shed light on optimisation dynamics, but it can also be used to improve
performance and stability across problem domains, from supervised learning to
two-player games such as Generative Adversarial Networks. An avenue for finding
such implicit regularisation effects has been quantifying the discretisation
errors of discrete optimisers via continuous-time flows constructed by backward
error analysis (BEA). The current usage of BEA is not without limitations,
since not all the vector fields of continuous-time flows obtained using BEA can
be written as a gradient, hindering the construction of modified losses
revealing implicit regularisers. In this work, we provide a novel approach to
use BEA, and show how our approach can be used to construct continuous-time
flows with vector fields that can be written as gradients. We then use this to
find previously unknown implicit regularisation effects, such as those induced
by multiple stochastic gradient descent steps while accounting for the exact
data batches used in the updates, and in generally differentiable two-player
games.

本篇研究提出了一种基于离散优化器的连续时间流的后向误差分析方法，用于构建可以被写成梯度形式的矢量场的流，从而找到以前未知的隐式正则化效应。

随机梯度下降中的隐式规则化：从单目标到两人博弈

Implicit regularisation in stochastic gradient descent: from  single-objective to two-player games

Gradient descent can be surprisingly good at optimizing deep neural networks
without overfitting and without explicit regularization. We find that the
discrete steps of gradient descent implicitly regularize models by penalizing
gradient descent trajectories that have large loss gradients. We call this
Implicit Gradient Regularization (IGR) and we use backward error analysis to
calculate the size of this regularization. We confirm empirically that implicit
gradient regularization biases gradient descent toward flat minima, where test
errors are small and solutions are robust to noisy parameter perturbations.
Furthermore, we demonstrate that the implicit gradient regularization term can
be used as an explicit regularizer, allowing us to control this gradient
regularization directly. More broadly, our work indicates that backward error
analysis is a useful theoretical approach to the perennial question of how
learning rate, model size, and parameter regularization interact to determine
the properties of overparameterized models optimized with gradient descent.

本文研究了梯度下降算法在优化神经网络时的表现，发现梯度下降中的离散步骤隐含地通过惩罚大损失梯度轨迹的方式实现了模型的正则化，这种 “隐性梯度正则化” 导致梯度下降趋向于平坦的最小值，使解决方案对噪声参数扰动有很好的鲁棒性，这一理论有助于解决过拟合问题。