Traditional deep learning models are trained at a centralized server using
labeled data samples collected from end devices or users. Such data samples
often include private information, which the users may not be willing to share.
Federated learning (FL) is an emerging approach to train such learning models
without requiring the users to share their possibly private labeled data. In
FL, each user trains its copy of the learning model locally. The server then
collects the individual updates and aggregates them into a global model. A
major challenge that arises in this method is the need of each user to
efficiently transmit its learned model over the throughput limited uplink
channel. In this work, we tackle this challenge using tools from quantization
theory. In particular, we identify the unique characteristics associated with
conveying trained models over rate-constrained channels, and propose a suitable
quantization scheme for such settings, referred to as universal vector
quantization for FL (UVeQFed). We show that combining universal vector
quantization methods with FL yields a decentralized training system in which
the compression of the trained models induces only a minimum distortion. We
then theoretically analyze the distortion, showing that it vanishes as the
number of users grows. We also characterize the convergence of models trained
with the traditional federated averaging method combined with UVeQFed to the
model which minimizes the loss function. Our numerical results demonstrate the
gains of UVeQFed over previously proposed methods in terms of both distortion
induced in quantization and accuracy of the resulting aggregated model.

本研究旨在解决联合学习中模型传输的瓶颈问题，提出一种名为 UVeQFed 的通用向量量化方案，通过量化方法的优化，实现了一种去中心化的训练系统，在保证模型传输最小化失真的同时提高了聚合模型的准确性。

UVeQFed：联邦学习的通用向量量化

UVeQFed: Universal Vector Quantization for Federated Learning

In this paper, we investigate lossy compression of deep neural networks
(DNNs) by weight quantization and lossless source coding for memory-efficient
deployment. Whereas the previous work addressed non-universal scalar
quantization and entropy coding of DNN weights, we for the first time introduce
universal DNN compression by universal vector quantization and universal source
coding. In particular, we examine universal randomized lattice quantization of
DNNs, which randomizes DNN weights by uniform random dithering before lattice
quantization and can perform near-optimally on any source without relying on
knowledge of its probability distribution. Moreover, we present a method of
fine-tuning vector quantized DNNs to recover the performance loss after
quantization. Our experimental results show that the proposed universal DNN
compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the
AlexNet (trained on ImageNet) with compression ratios of $47.1$ and $42.5$,
respectively.

本文研究深度神经网络的权重量化和无损源编码的有损压缩以实现内存有效部署，通过引入通用向量量化和通用源编码，实现了通用的深度神经网络压缩，并尝试运用通用随机格量化方法来随机化神经网络权重，证明该方法在压缩 32 层的 ResNet 和 AlexNet 时具有较高的压缩比和较低的失真率。