A significant bottleneck in federated learning is the network communication cost of sending model updates from client devices to the central server. We propose a method to reduce this cost. Our method encodes quantized updates with an appropriate universal code, taking into account their empirical distribution. Because quantization introduces error, we select quantization levels by optimizing for the desired trade-off in average total bitrate and gradient distortion. We demonstrate empirically that in spite of the non-i.i.d. nature of federated learning, the rate-distortion frontier is consistent across datasets, optimizers, clients and training rounds, and within each setting, distortion reliably predicts model performance. This allows for a remarkably simple compression scheme that is near-optimal in many use cases, and outperforms Top-K, DRIVE, 3LC and QSGD on the Stack Overflow next-word prediction benchmark.

研究了联邦学习中模型更新的统计学特性及各种压缩技术的作用和优劣，提出了基于失真作为可靠的模型精度代理的降低通信成本的新方法，并在Stack Overflow下一个真实具有挑战性的FL基准测试中实现了接近最优的通信成本降低，在多个数据集、优化器、客户端和训练轮次中，失真-失真前沿保持一致。

使用速率失真理论在联邦学习中优化通信-准确性权衡