We study the effect of the batch size to the total gradient variance in
differentially private stochastic gradient descent (DP-SGD), seeking a
theoretical explanation for the usefulness of large batch sizes. As DP-SGD is
the basis of modern DP deep learning, its properties have been widely studied,
and recent works have empirically found large batch sizes to be beneficial.
However, theoretical explanations of this benefit are currently heuristic at
best. We first observe that the total gradient variance in DP-SGD can be
decomposed into subsampling-induced and noise-induced variances. We then prove
that in the limit of an infinite number of iterations, the effective
noise-induced variance is invariant to the batch size. The remaining
subsampling-induced variance decreases with larger batch sizes, so large
batches reduce the effective total gradient variance. We confirm numerically
that the asymptotic regime is relevant in practical settings when the batch
size is not small, and find that outside the asymptotic regime, the total
gradient variance decreases even more with large batch sizes. We also find a
sufficient condition that implies that large batch sizes similarly reduce
effective DP noise variance for one iteration of DP-SGD.

研究了批量大小对区分隐私随机梯度下降（DP-SGD）中的总梯度方差的影响，旨在理论上解释大批量大小的有用性。发现 DP-SGD 的总梯度方差可以分解为子抽样引起的方差和噪声引起的方差，并证明在迭代次数无限的极限情况下，噪声引起的方差与批量大小无关，而子抽样引起的方差随着批量大小的增加而减小，因此大批量可以减小有效总梯度方差。在实际设置中验证了渐近区域的相关性，并发现在渐近区域之外，总梯度方差随着大批量的增加而进一步减小。还找到了一个足够的条件，表明大批量同样可以减小一个 DP-SGD 迭代的有效随机噪声方差。