Federated learning (FL) addresses data privacy concerns by enabling
collaborative training of AI models across distributed data owners. Wide
adoption of FL faces the fundamental challenges of data heterogeneity and the
large scale of data owners involved. In this paper, we investigate the prospect
of Transformer-based FL models for achieving generalization and personalization
in this setting. We conduct extensive comparative experiments involving FL with
Transformers, ResNet, and personalized ResNet-based FL approaches under various
scenarios. These experiments consider varying numbers of data owners to
demonstrate Transformers' advantages over deep neural networks in large-scale
heterogeneous FL tasks. In addition, we analyze the superior performance of
Transformers by comparing the Centered Kernel Alignment (CKA) representation
similarity across different layers and FL models to gain insight into the
reasons behind their promising capabilities.

分布式数据合作的联邦学习旨在解决数据隐私问题，该研究通过比较实验证明基于 Transformer 的联邦学习模型在大规模异构任务中具有泛化和个性化方面的优势。