Adaptive gradient algorithms have been widely adopted in training large-scale
deep neural networks, especially large foundation models. Despite their huge
success in practice, their theoretical advantages over stochastic gradient
descent (SGD) have not been fully understood, especially in the large
batch-size setting commonly used in practice. This is because the only
theoretical result that can demonstrate the benefit of Adagrad over SGD was
obtained in the original paper of Adagrad for nonsmooth objective functions.
However, for nonsmooth objective functions, there can be a linear slowdown of
convergence when batch size increases, and thus a convergence analysis based on
nonsmooth assumption cannot be used for large batch algorithms. In this work,
we resolve this gap between theory and practice by providing a new analysis of
Adagrad on both convex and nonconvex smooth objectives suitable for the large
batch setting. It is shown that under the anisotropic smoothness and noise
conditions, increased batch size does not slow down convergence for Adagrad,
and thus it can still achieve a faster convergence guarantee over SGD even in
the large batch setting. We present detailed comparisons between SGD and
Adagrad to provide a better understanding of the benefits of adaptive gradient
methods. Experiments in logistic regression and instruction following
fine-tuning tasks provide strong evidence to support our theoretical analysis.

通过在大批量设置下对 Adagrad 进行新分析，证明了它在凸平滑目标和非凸平滑目标上不会因批量大小增加而收敛减慢，因此在大批量设置中仍然可以比 SGD 更快地实现收敛，进而解决了理论和实践之间的差距。