Modern deep neural networks often require distributed training with many
workers due to their large size. As worker numbers increase, communication
overheads become the main bottleneck in data-parallel minibatch stochastic
gradient methods with per-iteration gradient synchronization. Local gradient
methods like Local SGD reduce communication by only syncing after several local
steps. Despite understanding their convergence in i.i.d. and heterogeneous
settings and knowing the importance of batch sizes for efficiency and
generalization, optimal local batch sizes are difficult to determine. We
introduce adaptive batch size strategies for local gradient methods that
increase batch sizes adaptively to reduce minibatch gradient variance. We
provide convergence guarantees under homogeneous data conditions and support
our claims with image classification experiments, demonstrating the
effectiveness of our strategies in training and generalization.

现代深度神经网络通常需要分布式训练以应对其巨大的规模，但当工作节点数量增加时，通过每次迭代梯度同步的数据并行小批量随机梯度方法中的通信开销成为主要瓶颈。本文引入了适应性批量大小策略，用于局部梯度方法，通过自适应地增加批量大小来减小小批量梯度的方差，提供了在均匀数据条件下的收敛性保证，并通过图像分类实验支持我们的说法，证明了我们的策略在训练和泛化中的有效性。

分布式本地梯度方法的通信高效自适应批量大小策略

Communication-Efficient Adaptive Batch Size Strategies for Distributed  Local Gradient Methods

The choice of batch sizes in stochastic gradient optimizers is critical for
model training. However, the practice of varying batch sizes throughout the
training process is less explored compared to other hyperparameters. We
investigate adaptive batch size strategies derived from adaptive sampling
methods, traditionally applied only in stochastic gradient descent. Given the
significant interplay between learning rates and batch sizes, and considering
the prevalence of adaptive gradient methods in deep learning, we emphasize the
need for adaptive batch size strategies in these contexts. We introduce
AdAdaGrad and its scalar variant AdAdaGradNorm, which incrementally increase
batch sizes during training, while model updates are performed using AdaGrad
and AdaGradNorm. We prove that AdaGradNorm converges with high probability at a
rate of $\mathscr{O}(1/K)$ for finding a first-order stationary point of smooth
nonconvex functions within $K$ iterations. AdaGrad also demonstrates similar
convergence properties when integrated with a novel coordinate-wise variant of
our adaptive batch size strategies. Our theoretical claims are supported by
numerical experiments on various image classification tasks, highlighting the
enhanced adaptability of progressive batching protocols in deep learning and
the potential of such adaptive batch size strategies with adaptive gradient
optimizers in large-scale model training.

通过使用自适应批次大小策略，引入了 AdAdaGrad 和 AdAdaGradNorm，在深度学习中展示了逐步批处理协议的提升适应性以及与自适应梯度优化器结合使用的自适应批次大小策略的潜力。