Federated learning (FL) allows clients in an Internet of Things (IoT) system
to collaboratively train a global model without sharing their local data with a
server. However, clients' contributions to the server can still leak sensitive
information. Differential privacy (DP) addresses such leakage by providing
formal privacy guarantees, with mechanisms that add randomness to the clients'
contributions. The randomness makes it infeasible to train large
transformer-based models, common in modern IoT systems. In this work, we
empirically evaluate the practicality of fine-tuning large scale on-device
transformer-based models with differential privacy in a federated learning
system. We conduct comprehensive experiments on various system properties for
tasks spanning a multitude of domains: speech recognition, computer vision (CV)
and natural language understanding (NLU). Our results show that full
fine-tuning under differentially private federated learning (DP-FL) generally
leads to huge performance degradation which can be alleviated by reducing the
dimensionality of contributions through parameter-efficient fine-tuning (PEFT).
Our benchmarks of existing DP-PEFT methods show that DP-Low-Rank Adaptation
(DP-LoRA) consistently outperforms other methods. An even more promising
approach, DyLoRA, which makes the low rank variable, when naively combined with
FL would straightforwardly break differential privacy. We therefore propose an
adaptation method that can be combined with differential privacy and call it
DP-DyLoRA. Finally, we are able to reduce the accuracy degradation and word
error rate (WER) increase due to DP to less than 2% and 7% respectively with 1
million clients and a stringent privacy budget of {\epsilon}=2.

该研究评估了差分隐私对于联邦学习系统中使用大规模本地化 Transformer 模型进行精调的实用性，发现全精调普遍会导致性能严重下降，但通过参数高效的精调方法能够缓解这一问题，而 DP-DyLoRA 方法在现有的 DP-PEFT 方法中表现出色。