Differentially private (DP) optimization is the standard paradigm to learn
large neural networks that are accurate and privacy-preserving. The
computational cost for DP deep learning, however, is notoriously heavy due to
the per-sample gradient clipping. Existing DP implementations are
$2-1000\times$ more costly in time and space complexity than the standard
(non-private) training. In this work, we develop a novel Book-Keeping (BK)
technique that implements existing DP optimizers (thus achieving the same
accuracy), with a substantial improvement on the computational cost.
Specifically, BK enables DP training on large models and high dimensional data
to be roughly as efficient as the standard training, whereas previous DP
algorithms can be inefficient or incapable of training due to memory error. The
computational advantage of BK is supported by the complexity analysis as well
as extensive experiments on vision and language tasks. Our implementation
achieves state-of-the-art (SOTA) accuracy with very small extra cost: on GPT2
and at the same memory cost, BK has 1.0$\times$ the time complexity of the
standard training (0.75$\times$ training speed in practice), and 0.6$\times$
the time complexity of the most efficient DP implementation (1.24$\times$
training speed in practice). We will open-source the codebase for the BK
algorithm.

本文提出了一种新的 Book-Keeping（BK）技术，用于实现现有的 DP 优化器，以降低计算成本，并在视觉和语言任务方面进行广泛实验，取得了 SOTA 精度。