Language models are capable of memorizing detailed patterns and information, leading to a double-edged effect: they achieve impressive modeling performance on downstream tasks with the stored knowledge but also raise significant privacy concerns. Traditional differential privacy based training approaches offer robust safeguards by employing a uniform noise distribution across all parameters. However, this overlooks the distinct sensitivities and contributions of individual parameters in privacy protection and often results in suboptimal models. To address these limitations, we propose ANADP, a novel algorithm that adaptively allocates additive noise based on the importance of model parameters. We demonstrate that ANADP narrows the performance gap between regular fine-tuning and traditional DP fine-tuning on a series of datasets while maintaining the required privacy constraints.

本研究解决了语言模型在微调过程中面临的隐私保护不足问题。提出的ANADP算法通过根据模型参数的重要性自适应分配加性噪声，优化了差分隐私的效果。实验结果表明，ANADP在满足隐私要求的同时，缩小了常规微调与传统差分隐私微调之间的性能差距。

通过自适应噪声分配实现差分隐私的语言模型微调