We analytically investigate how over-parameterization of models in randomized
machine learning algorithms impacts the information leakage about their
training data. Specifically, we prove a privacy bound for the KL divergence
between model distributions on worst-case neighboring datasets, and explore its
dependence on the initialization, width, and depth of fully connected neural
networks. We find that this KL privacy bound is largely determined by the
expected squared gradient norm relative to model parameters during training.
Notably, for the special setting of linearized network, our analysis indicates
that the squared gradient norm (and therefore the escalation of privacy loss)
is tied directly to the per-layer variance of the initialization distribution.
By using this analysis, we demonstrate that privacy bound improves with
increasing depth under certain initializations (LeCun and Xavier), while
degrades with increasing depth under other initializations (He and NTK). Our
work reveals a complex interplay between privacy and depth that depends on the
chosen initialization distribution. We further prove excess empirical risk
bounds under a fixed KL privacy budget, and show that the interplay between
privacy utility trade-off and depth is similarly affected by the
initialization.

过度参数化和随机化机器学习算法对其训练数据的信息泄露有何影响，通过解析分析我们得出了模型分布间的 KL 散度的隐私界限，并研究其对全连接神经网络的初始化、宽度和深度的依赖性。我们发现，这个 KL 隐私界限很大程度上由训练过程中的模型参数相对于期望梯度范数决定。我们的分析表明，对于线性化网络的特殊设定，梯度范数（从而隐私损失的升级）直接与初始化分布的每层方差相关联。利用这一分析，我们证明了在特定初始化（LeCun 和 Xavier）下随着深度的增加，隐私界限提高，而在其他初始化（He 和 NTK）下随深度的增加，隐私界限降低。我们的工作揭示了隐私与深度之间的复杂相互作用，这取决于所选的初始化分布。我们进一步证明了在固定 KL 隐私预算下的过度经验风险界限，并展示了隐私效用权衡与深度之间的相互作用同样受到初始化方式的影响。