The high resource consumption of large-scale models discourages
resource-constrained users from developing their customized transformers. To
this end, this paper considers a federated framework named Fed-Grow for
multiple participants to cooperatively scale a transformer from their
pre-trained small models. Under the Fed-Grow, a Dual-LiGO (Dual Linear Growth
Operator) architecture is designed to help participants expand their
pre-trained small models to a transformer. In Dual-LiGO, the Local-LiGO part is
used to address the heterogeneity problem caused by the various pre-trained
models, and the Global-LiGO part is shared to exchange the implicit knowledge
from the pre-trained models, local data, and training process of participants.
Instead of model sharing, only sharing the Global-LiGO strengthens the privacy
of our approach. Compared with several state-of-the-art methods in simulation,
our approach has higher accuracy, better precision, and lower resource
consumption on computations and communications. To the best of our knowledge,
most of the previous model-scaling works are centralized, and our work is the
first one that cooperatively grows a transformer from multiple pre-trained
heterogeneous models with the user privacy protected in terms of local data and
models. We hope that our approach can extend the transformers to the broadly
distributed scenarios and encourage more resource-constrained users to enjoy
the bonus taken by the large-scale transformers.

使用名为 Fed-Grow 的联邦框架，设计了一个名为 Dual-LiGO 的架构来帮助多个参与者从其预训练的小模型扩展到一个 Transformer，以提高模型的准确性和资源利用率，并保护用户的隐私。