Combining Domain-adaptive Pre-training (DAPT) with Federated Learning (FL)
can enhance model adaptation by leveraging more sensitive and distributed data
while preserving data privacy. However, few studies have focused on this
method. Therefore, we conduct the first comprehensive empirical study to
evaluate the performance of Federated Domain-adaptive Pre-training (FDAPT). We
demonstrate that FDAPT can maintain competitive downstream task performance to
the centralized baseline in both IID and non-IID situations. Furthermore, we
propose a novel algorithm, Frozen Federated Domain-adaptive Pre-training
(FFDAPT). FFDAPT improves the computational efficiency by 12.1% on average and
exhibits similar downstream task performance to standard FDAPT, with general
performance fluctuations remaining less than 1%. Finally, through a critical
evaluation of our work, we identify promising future research directions for
this new research area.

本文首次全面实证研究，将域自适应预训练 (DAPT) 与联邦学习 (FL) 结合，可以在保护数据隐私的同时，通过利用更敏感和分布式的数据来增强模型适应性。我们提出一种新的算法，Frozen Federated Domain-adaptive Pre-training (FFDAPT)，平均节省 12.1% 的计算资源，并在 IID 和非 - IID 情况下均能保持与中心化模型相当的表现。