Large language models have consistently demonstrated remarkable performance
across a wide spectrum of applications. Nonetheless, the deployment of these
models can inadvertently expose user privacy to potential risks. The
substantial memory demands of these models during training represent a
significant resource consumption challenge. The sheer size of these models
imposes a considerable burden on memory resources, which is a matter of
significant concern in practice. In this paper, we present an innovative
training framework MemDPT that not only reduces the memory cost of large
language models but also places a strong emphasis on safeguarding user data
privacy. MemDPT provides edge network and reverse network designs to
accommodate various differential privacy memory-efficient fine-tuning schemes.
Our approach not only achieves $2 \sim 3 \times$ memory optimization but also
provides robust privacy protection, ensuring that user data remains secure and
confidential. Extensive experiments have demonstrated that MemDPT can
effectively provide differential privacy efficient fine-tuning across various
task scenarios.

通过提出的创新训练框架 MemDPT，本文在减少大型语言模型的内存成本的同时，强调保护用户数据的隐私，通过边缘网络和反向网络设计实现了差分隐私内存高效微调方案，达到了 2 至 3 倍的内存优化，并提供了可靠的隐私保护，确保用户数据的安全和保密性。广泛的实验证明了 MemDPT 能够有效地在各种任务场景下提供差分隐私高效微调。