A rich body of prior work has highlighted the existence of communication
bottlenecks in synchronous data-parallel training. To alleviate these
bottlenecks, a long line of recent work proposes gradient and model compression
methods. In this work, we evaluate the efficacy of gradient compression methods
and compare their scalability with optimized implementations of synchronous
data-parallel SGD across more than 200 different setups. Surprisingly, we
observe that only in 6 cases out of more than 200, gradient compression methods
provide speedup over optimized synchronous data-parallel training in the
typical data-center setting. We conduct an extensive investigation to identify
the root causes of this phenomenon, and offer a performance model that can be
used to identify the benefits of gradient compression for a variety of system
setups. Based on our analysis, we propose a list of desirable properties that
gradient compression methods should satisfy, in order for them to provide a
meaningful end-to-end speedup.

为了解决同步数据并行训练中的通信瓶颈，本研究评估了梯度压缩方法的有效性，并将它们的可扩展性与优化后的同步数据并行 SGD 实现进行了比较。分析发现，仅有 6 种情况下，梯度压缩方法提供了速度优势。本研究提出了一种性能模型，用于确定在各种系统设置中梯度压缩的益处，并提出了一些理想的梯度压缩方法的性质列表，以获得有意义的综合优势。