The widespread practice of fine-tuning pretrained large language models
(LLMs) on domain-specific data faces two major challenges in memory and
privacy. First, as the size of LLMs continue to grow, encompassing billions of
parameters, the memory demands of gradient-based training methods via
backpropagation become prohibitively high. Second, given the tendency of LLMs
to memorize and disclose sensitive training data, the privacy of fine-tuning
data must be respected. To this end, we explore the potential of zeroth-order
methods in differentially private optimization for fine-tuning LLMs.
Zeroth-order methods, which rely solely on forward passes, substantially reduce
memory consumption during training. However, directly combining them with
standard differential privacy mechanism poses dimension-dependent complexity.
To bridge the gap, we introduce DPZero, a novel differentially private
zeroth-order algorithm with nearly dimension-independent rates. Our theoretical
analysis reveals that its complexity hinges primarily on the problem's
intrinsic dimension and exhibits only a logarithmic dependence on the ambient
dimension. This renders DPZero a highly practical option for real-world LLMs
deployments.

通过零阶方法进行微调预训练语言模型，以解决精调大型语言模型在内存和隐私方面面临的挑战。DPZero 是一个新的具有几乎维度无关率的差分隐私零阶算法，对于实际中的语言模型部署具有高度实用性。