We present a practical method to audit the differential privacy (DP)
guarantees of a machine learning model using a small hold-out dataset that is
not exposed to the model during the training. Having a score function such as
the loss function employed during the training, our method estimates the total
variation (TV) distance between scores obtained with a subset of the training
data and the hold-out dataset. With some meta information about the underlying
DP training algorithm, these TV distance values can be converted to
$(\varepsilon,\delta)$-guarantees for any $\delta$. We show that these score
distributions asymptotically give lower bounds for the DP guarantees of the
underlying training algorithm, however, we perform a one-shot estimation for
practicality reasons. We specify conditions that lead to lower bounds for the
DP guarantees with high probability. To estimate the TV distance between the
score distributions, we use a simple density estimation method based on
histograms. We show that the TV distance gives a very close to optimally robust
estimator and has an error rate $\mathcal{O}(k^{-1/3})$, where $k$ is the total
number of samples. Numerical experiments on benchmark datasets illustrate the
effectiveness of our approach and show improvements over baseline methods for
black-box auditing.

我们提出了一种实用的方法，使用一个未在训练中暴露给模型的小规模保留数据集来审计差分隐私（DP）保证。我们的方法利用得分函数（如训练期间使用的损失函数）估计使用训练数据子集和保留数据集获得的得分之间的总变差（TV）距离，在了解底层 DP 训练算法的一些元信息的情况下，这些 TV 距离值可以转换为任意 δ 的（𝜖, δ）保证。我们表明，这些得分分布渐近地为底层训练算法的 DP 保证提供下界，但基于实用性的原因，我们对其进行了一次估计。我们指定导致高概率下界 DP 保证的条件。为了估计得分分布之间的 TV 距离，我们使用了基于直方图的简单密度估计方法。我们表明，TV 距离提供了一个非常接近最优鲁棒估计器，并具有一个误差率𝒪(k^(−1/3))，其中 k 是总样本数。基准数据集上的数值实验说明了我们方法的有效性，并展示了对于黑盒审计的改进。