Decentralized training of large language models has emerged as an effective
way to democratize this technology. However, the potential threats associated
with this approach have not been carefully discussed, which would hinder the
development of decentralized training infrastructures. This paper aims to
initiate discussion towards this end by exploring the robustness of
decentralized training from three main perspectives. First, we demonstrate the
vulnerabilities inherent in decentralized training frameworks in terms of
hardware, data, and models. Second, we highlight the fundamental difference
between decentralized foundation model training and vanilla federated learning,
where the security techniques employed in federated learning cannot be applied
directly. Third, we discuss the essential components required for a robust and
efficient decentralized training framework and present a case study by modeling
a concrete threat model. Our objective in this vision paper is to emphasize the
importance of addressing security concerns in the context of decentralized
training for large language models.

通过探索去中心化训练的强壮性、去中心化基础模型训练与联邦学习的根本区别以及确保强壮且高效的去中心化训练框架所需的基本组件，本文旨在强调在大型语言模型的去中心化训练上解决安全问题的重要性。