In distributed training, deep neural networks (DNNs) are launched over
multiple workers concurrently and aggregate their local updates on each step in
bulk-synchronous parallel (BSP) training. However, BSP does not linearly
scale-out due to high communication cost of aggregation. To mitigate this
overhead, alternatives like Federated Averaging (FedAvg) and Stale-Synchronous
Parallel (SSP) either reduce synchronization frequency or eliminate it
altogether, usually at the cost of lower final accuracy. In this paper, we
present \texttt{SelSync}, a practical, low-overhead method for DNN training
that dynamically chooses to incur or avoid communication at each step either by
calling the aggregation op or applying local updates based on their
significance. We propose various optimizations as part of \texttt{SelSync} to
improve convergence in the context of \textit{semi-synchronous} training. Our
system converges to the same or better accuracy than BSP while reducing
training time by up to 14$\times$.

本文提出了一种名为 SelSync 的低开销方法，用于 DNN 训练，该方法具有动态选择通信与否的功能，可提高收敛性，比 BSP 技术减少训练时间高达 14 倍。