For many differentially private algorithms, such as the prominent noisy stochastic gradient descent (DP-SGD), the analysis needed to bound the privacy leakage of a single training run is well understood. However, few studies have reasoned about the privacy leakage resulting from the multiple training runs needed to fine tune the value of the training algorithm's hyperparameters. In this work, we first illustrate how simply setting hyperparameters based on non-private training runs can leak private information. Motivated by this observation, we then provide privacy guarantees for hyperparameter search procedures within the framework of Renyi Differential Privacy. Our results improve and extend the work of Liu and Talwar (STOC 2019). Our analysis supports our previous observation that tuning hyperparameters does indeed leak private information, but we prove that, under certain assumptions, this leakage is modest, as long as each candidate training run needed to select hyperparameters is itself differentially private.

本文研究差分隐私算法包括DP-SGD等在进行多次训练来微调算法超参数时产生的隐私泄漏问题，并提出了基于Renyi差分隐私的超参数搜索方法，结果表明虽然调整超参数的确会带来隐私泄漏，但只要每次候选训练运行本身是差分隐私的，那么泄漏就是适度的。

应用Renyi差分隐私方法进行超参数调优