Low-Rank Adaptation (LoRA) methods have gained popularity in efficient
parameter fine-tuning of models containing hundreds of billions of parameters.
In this work, instead, we demonstrate the application of LoRA methods to train
small-vision models in Federated Learning (FL) from scratch. We first propose
an aggregation-agnostic method to integrate LoRA within FL, named FLoCoRA,
showing that the method is capable of reducing communication costs by 4.8
times, while having less than 1% accuracy degradation, for a CIFAR-10
classification task with a ResNet-8. Next, we show that the same method can be
extended with an affine quantization scheme, dividing the communication cost by
18.6 times, while comparing it with the standard method, with still less than
1% of accuracy loss, tested with on a ResNet-18 model. Our formulation
represents a strong baseline for message size reduction, even when compared to
conventional model compression works, while also reducing the training memory
requirements due to the low-rank adaptation.

在此研究中，我们展示了 Low-Rank Adaptation 方法在从头开始训练小尺寸视觉模型的联邦学习中的应用，通过提出一种聚合不可知方法 FLoCoRA，我们证明该方法能够减少通信成本 4.8 倍，并且对于具有 ResNet-8 模型的 CIFAR-10 分类任务的准确度损失小于 1%。同时，我们还展示了通过引入仿射量化方案扩展该方法，将通信成本缩减 18.6 倍，与标准方法相比，对于 ResNet-18 模型的测试仍然保持不到 1% 的准确度损失。我们的研究在消息大小减小方面提供了一个强有力的基准，甚至与传统的模型压缩方法相比，同时还减少了由于低秩适应所需的训练内存。