Methods with adaptive stepsizes, such as AdaGrad and Adam, are essential for
training modern Deep Learning models, especially Large Language Models.
Typically, the noise in the stochastic gradients is heavy-tailed for the later
ones. Gradient clipping provably helps to achieve good high-probability
convergence for such noises. However, despite the similarity between
AdaGrad/Adam and Clip-SGD, the high-probability convergence of AdaGrad/Adam has
not been studied in this case. In this work, we prove that AdaGrad (and its
delayed version) can have provably bad high-probability convergence if the
noise is heavy-tailed. To fix this issue, we propose a new version of AdaGrad
called Clip-RAdaGradD (Clipped Reweighted AdaGrad with Delay) and prove its
high-probability convergence bounds with polylogarithmic dependence on the
confidence level for smooth convex/non-convex stochastic optimization with
heavy-tailed noise. Our empirical evaluations, including NLP model fine-tuning,
highlight the superiority of clipped versions of AdaGrad/Adam in handling the
heavy-tailed noise.

本文通过证明，在处理偏重尾噪声时，AdaGrad 和 Adam 具有很差的高概率收敛性，提出了一种名为 Clip-RAdaGradD（Clipped Reweighted AdaGrad with Delay）的新版本 AdaGrad，并证明了它在处理偏重尾噪声时具有多对数相关性的高概率收敛边界。经验评估揭示出剪裁版本的 AdaGrad/Adam 在处理偏重尾噪声时具有卓越优势。

在重尾噪声存在时，梯度剪裁改进了 AdaGrad

Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed

Stochastic gradient descent is the method of choice for large scale
optimization of machine learning objective functions. Yet, its performance is
greatly variable and heavily depends on the choice of the stepsizes. This has
motivated a large body of research on adaptive stepsizes. However, there is
currently a gap in our theoretical understanding of these methods, especially
in the non-convex setting. In this paper, we start closing this gap: we
theoretically analyze in the convex and non-convex settings a generalized
version of the AdaGrad stepsizes. We show sufficient conditions for these
stepsizes to achieve almost sure asymptotic convergence of the gradients to
zero, proving the first guarantee for generalized AdaGrad stepsizes in the
non-convex setting. Moreover, we show that these stepsizes allow to
automatically adapt to the level of noise of the stochastic gradients in both
the convex and non-convex settings, interpolating between $O(1/T)$ and
$O(1/\sqrt{T})$, up to logarithmic terms.

通过研究广义 AdaGrad 步长在凸和非凸设置中，本文证明了这些步长实现梯度渐近收敛于零的充分条件，从而填补了这些方法理论上的空白。此外，本文表明这些步长允许自动适应随机梯度噪声级别在凸和非凸情况下，实现 O（1/T）到 O（1 / 根号 T）的插值（带有对数项）。