In recent years, Large Language Models (LLMs) through Transformer structures have dominated many machine learning tasks, especially text processing. However, these models require massive amounts of data for training and induce high resource requirements, particularly in terms of the large number of Floating Point Operations (FLOPs) and the high amounts of memory needed. To fine-tune such a model in a parameter-efficient way, techniques like Adapter or LoRA have been developed. However, we observe that the application of LoRA, when used in federated learning (FL), while still being parameter-efficient, is memory and FLOP inefficient. Based on that observation, we develop a novel layer finetuning scheme that allows devices in cross-device FL to make use of pretrained neural networks (NNs) while adhering to given resource constraints. We show that our presented scheme outperforms the current state of the art when dealing with homogeneous or heterogeneous computation and memory constraints and is on par with LoRA regarding limited communication, thereby achieving significantly higher accuracies in FL training.

本研究解决了在联邦学习中微调大型语言模型时内存和计算资源效率低的问题。提出了一种新颖的层微调方案，使得跨设备的联邦学习能够在保持资源约束的同时利用预训练神经网络。研究结果表明，该方案在面对均匀或异构的计算和内存约束时，性能优于现有方法，并在有限通信条件下的准确度与LoRA相当，显著提升了联邦学习训练的准确性。

资源受限设备高效联邦微调小型变压器