This position paper investigates the integration of Differential Privacy (DP)
in the training of Mixture of Experts (MoE) models within the field of natural
language processing. As Large Language Models (LLMs) scale to billions of
parameters, leveraging expansive datasets, they exhibit enhanced linguistic
capabilities and emergent abilities. However, this growth raises significant
computational and privacy concerns. Our study addresses these issues by
exploring the potential of MoE models, known for their computational
efficiency, and the application of DP, a standard for privacy preservation. We
present the first known attempt to train MoE models under the constraints of
DP, addressing the unique challenges posed by their architecture and the
complexities of DP integration. Our initial experimental studies demonstrate
that MoE models can be effectively trained with DP, achieving performance that
is competitive with their non-private counterparts. This initial study aims to
provide valuable insights and ignite further research in the domain of
privacy-preserving MoE models, softly laying the groundwork for prospective
developments in this evolving field.

本研究调查了差分隐私 (Differential Privacy) 在自然语言处理领域中混合专家模型 (Mixture of Experts models) 训练中的整合。通过对大规模语言模型 (Large Language Models) 在数十亿参数上进行训练，利用海量数据集，使其具备了强大的语言能力和新兴的能力，但这一增长带来了重大的计算和隐私问题。我们的研究通过探索具备计算效率的 MoE 模型及 DP 的应用来解决这些问题。我们首次尝试在 DP 的约束下训练 MoE 模型，解决了其架构以及 DP 整合的复杂性所带来的独特挑战。我们的初步实验研究表明，MoE 模型可以通过 DP 有效训练，并与非隐私对照组有竞争力的性能。本次研究旨在为隐私保护的 MoE 模型领域提供有价值的洞察力，为该领域进一步的研究奠定基础，并为未来的发展提供引导。