How much does a machine learning algorithm leak about its training data, and
why? Membership inference attacks are used as an auditing tool to quantify this
leakage. In this paper, we present a comprehensive \textit{hypothesis testing
framework} that enables us not only to formally express the prior work in a
consistent way, but also to design new membership inference attacks that use
reference models to achieve a significantly higher power (true positive rate)
for any (false positive rate) error. More importantly, we explain \textit{why}
different attacks perform differently. We present a template for
indistinguishability games, and provide an interpretation of attack success
rate across different instances of the game. We discuss various uncertainties
of attackers that arise from the formulation of the problem, and show how our
approach tries to minimize the attack uncertainty to the one bit secret about
the presence or absence of a data point in the training set. We perform a
\textit{differential analysis} between all types of attacks, explain the gap
between them, and show what causes data points to be vulnerable to an attack
(as the reasons vary due to different granularities of memorization, from
overfitting to conditional memorization). Our auditing framework is openly
accessible as part of the \textit{Privacy Meter} software tool.

通过成员推理攻击作为审核工具，我们提出了一个综合的假设检验框架，不仅能够以一致的方式正式表达先前的工作，还可以设计新的成员推理攻击，使用参考模型来实现任意误报率的显著更高的功率（真正的阳性率），并解释为什么不同的攻击效果不同，并最小化攻击不确定性到数据点的存在或缺失的一个比特秘密。