Distributed model training suffers from communication bottlenecks due to
frequent model updates transmitted across compute nodes. To alleviate these
bottlenecks, practitioners use gradient compression techniques like
sparsification, quantization, or low-rank updates. The techniques usually
require choosing a static compression ratio, often requiring users to balance
the trade-off between model accuracy and per-iteration speedup. In this work,
we show that such performance degradation due to choosing a high compression
ratio is not fundamental. An adaptive compression strategy can reduce
communication while maintaining final test accuracy. Inspired by recent
findings on critical learning regimes, in which small gradient errors can have
irrecoverable impact on model performance, we propose Accordion a simple yet
effective adaptive compression algorithm. While Accordion maintains a high
enough compression rate on average, it avoids over-compressing gradients
whenever in critical learning regimes, detected by a simple gradient-norm based
criterion. Our extensive experimental study over a number of machine learning
tasks in distributed environments indicates that Accordion, maintains similar
model accuracy to uncompressed training, yet achieves up to 5.5x better
compression and up to 4.1x end-to-end speedup over static approaches. We show
that Accordion also works for adjusting the batch size, another popular
strategy for alleviating communication bottlenecks.

本文提出一种名为 Accordion 的自适应压缩算法，用于解决分布式模型训练中由于频繁的模型更新导致的通信瓶颈问题，该算法能够在保持最终测试准确度的情况下减少通信，实验结果表明 Accordion 相比于静态压缩算法可获得高达 5.5 倍的压缩率和 4.1 倍的端到端加速。