As collecting huge amounts of personal data from individuals has been established as a standard nowadays, it is really important to use these data in a conscientious way. For example, when performing inference using these data, one has to make sure individuals' identities or the privacy of the data are not compromised. Differential privacy is a powerful framework that introduces stochasticity into the computation to guarantee that it is difficult to breach the privacy using the output of the computation. Differentially private versions of many important machine learning methods have been proposed, but still there is a long way to pave towards an efficient unified approach applicable to handle many models. In this paper, we propose a differentially private variational inference method with a very wide applicability. The variational inference is based on stochastic gradient ascent and can handle non-conjugate models as well as conjugate ones. Differential privacy is achieved by perturbing the gradients. We explore ways to make the algorithm more efficient through privacy amplification from subsampling and through clipping the gradients to limit the amount of information they leak. We explore the effect of different parameter combinations in logistic regression problems where the method can reach an accuracy close to non-private level under reasonably strong privacy guarantees.

本文提出一种差分隐私变分推理方法，该方法基于双重随机变分推理，并通过剪辑和扰动梯度来加入差分隐私，并通过子采样来增加隐私保护的效率。该方法在强隐私保证下的准确性接近于非私密级别，是以前基于抽样方法的替代品的明显改进。

非共轭模型的差分隐私变分推断