大步长同步分布式 SGD 的通信权衡

Apr, 2019

Communication trade-offs for synchronized distributed SGD with large step size

Kumar Kshitij Patel, Aymeric Dieuleveut

TL;DR本文提出了一种名为local-SGD的算法，通过逐步同步而非每一步都进行通信提高了通信效率，同时在大步长情况下提供了自适应下限比较。

Abstract

Synchronous mini-batch sgd is state-of-the-art for large-scale distributed machine learning. However, in practice, its convergence is bott