Pre-trained large language models (LLMs) require fine-tuning to improve their
responsiveness to natural language instructions. Federated learning (FL) offers
a way to perform fine-tuning using the abundant data on end devices without
compromising data privacy. Most existing federated fine-tuning methods for LLMs
rely on parameter-efficient fine-tuning techniques, which may not reach the
performance heights possible with full-parameter tuning. However, the
communication overhead associated with full-parameter tuning is prohibitively
high for both servers and clients. This work introduces FedKSeed, a novel
approach that employs zeroth-order optimization (ZOO) with a set of random
seeds. It enables federated full-parameter tuning of billion-sized LLMs
directly on devices. Our method significantly reduces transmission requirements
between the server and clients to just a few scalar gradients and random seeds,
amounting to only a few thousand bytes. Building on this, we develop a strategy
to assess the significance of ZOO perturbations for FL, allowing for
probability-differentiated seed sampling. This prioritizes perturbations that
have a greater impact on model accuracy. Experiments across six scenarios with
different LLMs, datasets and data partitions demonstrate that our approach
outperforms existing federated LLM fine-tuning methods in terms of both
communication efficiency and new task generalization.

使用零阶优化技术以及一组随机种子，FedKSeed 提出了一种新颖的方法来在设备上直接对十亿级别的预训练大型语言模型进行联邦全参数调整，从而显著减少了服务器和客户端之间的传输需求，并通过策略评估 ZOO 扰动的显著性，提升模型准确度，我们的实验证明该方法在通信效率和新任务泛化方面优于已有的联邦预训练大型语言模型调整方法。