In distributed deep learning with data parallelism, synchronizing gradients
at each training step can cause a huge communication overhead, especially when
many nodes work together to train large models. Local gradient methods, such as
Local SGD, address this issue by allowing workers to compute locally for $H$
steps without synchronizing with others, hence reducing communication
frequency. While $H$ has been viewed as a hyperparameter to trade optimization
efficiency for communication cost, recent research indicates that setting a
proper $H$ value can lead to generalization improvement. Yet, selecting a
proper $H$ is elusive. This work proposes a theory-grounded method for
determining $H$, named the Quadratic Synchronization Rule (QSR), which
recommends dynamically setting $H$ in proportion to $\frac{1}{\eta^2}$ as the
learning rate $\eta$ decays over time. Extensive ImageNet experiments on ResNet
and ViT show that local gradient methods with QSR consistently improve the test
accuracy over other synchronization strategies. Compared with the standard data
parallel training, QSR enables Local AdamW on ViT-B to cut the training time on
16 or 64 GPUs down from 26.7 to 20.2 hours or from 8.6 to 5.5 hours and, at the
same time, achieves $1.16\%$ or $0.84\%$ higher top-1 validation accuracy.

使用分布式深度学习和数据并行 ism 方法，通过提出基于理论的使用二次同步规则（QSR）来动态确定本地梯度方法中的一个超参数 H 的值，以提高泛化性能。在 ImageNet 上的实验结果表明，使用 QSR 能够持续改善测试准确性，并且相比标准数据并行训练，可以显著减少训练时间并提高验证准确性。