Recent advances have substantially improved the accuracy, memory cost, and
training speed of differentially private (DP) deep learning, especially on
large vision and language models with millions to billions of parameters. In
this work, we thoroughly study the per-sample gradient clipping style, a key
component in DP optimization. We show that different clipping styles have the
same time complexity but instantiate an accuracy-memory trade-off: while the
all-layer clipping (of coarse granularity) is the most prevalent and usually
gives the best accuracy, it incurs heavier memory cost compared to other
group-wise clipping, such as the layer-wise clipping (of finer granularity). We
formalize this trade-off through our convergence theory and complexity
analysis. Importantly, we demonstrate that the accuracy gap between group-wise
clipping and all-layer clipping becomes smaller for larger models, while the
memory advantage of the group-wise clipping remains. Consequently, the
group-wise clipping allows DP optimization of large models to achieve high
accuracy and low peak memory simultaneously.

在本研究中，我们深入研究了差分隐私优化中关键组成部分之一的逐样本梯度剪裁方式，发现不同的剪裁方式具有相同的时间复杂度，但存在准确性 - 内存消耗的权衡关系：粗粒度全部层剪裁通常提供最佳准确性，但相比于细粒度的分组剪裁，会带来更高的内存开销。我们通过收敛性理论和复杂性分析形式化表达了这种权衡关系。重要的是，我们证明了在更大的模型中，分组剪裁与全部层剪裁之间的准确性差距越来越小，而分组剪裁的内存优势仍然存在。因此，分组剪裁允许对大型模型进行差分隐私优化，以同时实现高准确性和低内存峰值。