Differentially Private Stochastic Gradients Descent (DP-SGD) is a prominent
paradigm for preserving privacy in deep learning. It ensures privacy by
perturbing gradients with random noise calibrated to their entire norm at each
training step. However, this perturbation suffers from a sub-optimal
performance: it repeatedly wastes privacy budget on the general converging
direction shared among gradients from different batches, which we refer as
common knowledge, yet yields little information gain. Motivated by this, we
propose a differentially private training framework with early gradient
decomposition and reconstruction (DPDR), which enables more efficient use of
the privacy budget. In essence, it boosts model utility by focusing on
incremental information protection and recycling the privatized common
knowledge learned from previous gradients at early training steps. Concretely,
DPDR incorporates three steps. First, it disentangles common knowledge and
incremental information in current gradients by decomposing them based on
previous noisy gradients. Second, most privacy budget is spent on protecting
incremental information for higher information gain. Third, the model is
updated with the gradient reconstructed from recycled common knowledge and
noisy incremental information. Theoretical analysis and extensive experiments
show that DPDR outperforms state-of-the-art baselines on both convergence rate
and accuracy.

通过提出 DPDR（差分隐私训练框架）实现隐私保护，通过提前梯度分解和重构、保护增量信息以及更新模型来提高隐私预算的利用效率，理论分析和广泛实验表明 DPDR 在收敛速度和准确性上优于现有基线方法。