The success of modern machine learning is due in part to the adaptive
optimization methods that have been developed to deal with the difficulties of
training large models over complex datasets. One such method is gradient
clipping: a practical procedure with limited theoretical underpinnings. In this
work, we study clipping in a least squares problem under streaming SGD. We
develop a theoretical analysis of the learning dynamics in the limit of large
intrinsic dimension-a model and dataset dependent notion of dimensionality. In
this limit we find a deterministic equation that describes the evolution of the
loss. We show that with Gaussian noise clipping cannot improve SGD performance.
Yet, in other noisy settings, clipping can provide benefits with tuning of the
clipping threshold. In these cases, clipping biases updates in a way beneficial
to training which cannot be recovered by SGD under any schedule. We conclude
with a discussion about the links between high-dimensional clipping and neural
network training.

通过研究剪裁在流式随机梯度下降中的应用，发现剪裁在某些噪声环境中可以提供性能优势，并讨论了高维剪裁与神经网络训练之间的联系。

SGD 在高维度中的梯度剪裁动力学

A Clipped Trip: the Dynamics of SGD with Gradient Clipping in  High-Dimensions

Adaptive optimization methods are widely recognized as among the most popular
approaches for training Deep Neural Networks (DNNs). Techniques such as Adam,
AdaGrad, and AdaHessian utilize a preconditioner that modifies the search
direction by incorporating information about the curvature of the objective
function. However, despite their adaptive characteristics, these methods still
require manual fine-tuning of the step-size. This, in turn, impacts the time
required to solve a particular problem. This paper presents an optimization
framework named SANIA to tackle these challenges. Beyond eliminating the need
for manual step-size hyperparameter settings, SANIA incorporates techniques to
address poorly scaled or ill-conditioned problems. We also explore several
preconditioning methods, including Hutchinson's method, which approximates the
Hessian diagonal of the loss function. We conclude with an extensive empirical
examination of the proposed techniques across classification tasks, covering
both convex and non-convex contexts.

SANIA 是一个优化框架，通过自适应优化方法和预条件方法，解决了在训练深度神经网络中需要手动微调步长和处理问题缩放或病态的挑战。在分类任务中，包括凸和非凸情境下进行了广泛实证研究。

SANIA：Polyak 类型优化框架导致尺度不变的随机算法

SANIA: Polyak-type Optimization Framework Leads to Scale Invariant  Stochastic Algorithms

Despite superior training outcomes, adaptive optimization methods such as
Adam, Adagrad or RMSprop have been found to generalize poorly compared to
Stochastic gradient descent (SGD). These methods tend to perform well in the
initial portion of training but are outperformed by SGD at later stages of
training. We investigate a hybrid strategy that begins training with an
adaptive method and switches to SGD when appropriate. Concretely, we propose
SWATS, a simple strategy which switches from Adam to SGD when a triggering
condition is satisfied. The condition we propose relates to the projection of
Adam steps on the gradient subspace. By design, the monitoring process for this
condition adds very little overhead and does not increase the number of
hyperparameters in the optimizer. We report experiments on several standard
benchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR-10 and
CIFAR-100 data sets, ResNet on the tiny-ImageNet data set and language modeling
with recurrent networks on the PTB and WT2 data sets. The results show that our
strategy is capable of closing the generalization gap between SGD and Adam on a
majority of the tasks.

提出一种混合方法 SWATS 进行训练，开头使用自适应方法 Adam，后期如果符合一定条件则切换至 SGD。实验证明，SWATS 能够缩短自适应方法和 SGD 之间的泛化差距，在多数任务上表现良好。

由 Adam 优化器转为 SGD 优化器提升泛化性能

Improving Generalization Performance by Switching from Adam to SGD

Adaptive optimization methods, which perform local optimization with a metric
constructed from the history of iterates, are becoming increasingly popular for
training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We
show that for simple overparameterized problems, adaptive methods often find
drastically different solutions than gradient descent (GD) or stochastic
gradient descent (SGD). We construct an illustrative binary classification
problem where the data is linearly separable, GD and SGD achieve zero test
error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to
half. We additionally study the empirical generalization capability of adaptive
methods on several state-of-the-art deep learning models. We observe that the
solutions found by adaptive methods generalize worse (often significantly
worse) than SGD, even when these solutions have better training performance.
These results suggest that practitioners should reconsider the use of adaptive
methods to train neural networks.

本文研究了使用自适应优化方法训练深度神经网络的表现，提出了一些简单超参数问题，发现自适应方法得到的结果往往比梯度下降方法差，甚至可能导致结果更糟糕，建议实践者重新考虑使用自适应方法训练神经网络。