The high cost of model training makes it increasingly desirable to develop
techniques for unlearning. These techniques seek to remove the influence of a
training example without having to retrain the model from scratch. Intuitively,
once a model has unlearned, an adversary that interacts with the model should
no longer be able to tell whether the unlearned example was included in the
model's training set or not. In the privacy literature, this is known as
membership inference. In this work, we discuss adaptations of Membership
Inference Attacks (MIAs) to the setting of unlearning (leading to their
``U-MIA'' counterparts). We propose a categorization of existing U-MIAs into
``population U-MIAs'', where the same attacker is instantiated for all
examples, and ``per-example U-MIAs'', where a dedicated attacker is
instantiated for each example. We show that the latter category, wherein the
attacker tailors its membership prediction to each example under attack, is
significantly stronger. Indeed, our results show that the commonly used U-MIAs
in the unlearning literature overestimate the privacy protection afforded by
existing unlearning techniques on both vision and language models. Our
investigation reveals a large variance in the vulnerability of different
examples to per-example U-MIAs. In fact, several unlearning algorithms lead to
a reduced vulnerability for some, but not all, examples that we wish to
unlearn, at the expense of increasing it for other examples. Notably, we find
that the privacy protection for the remaining training examples may worsen as a
consequence of unlearning. We also discuss the fundamental difficulty of
equally protecting all examples using existing unlearning schemes, due to the
different rates at which examples are unlearned. We demonstrate that naive
attempts at tailoring unlearning stopping criteria to different examples fail
to alleviate these issues.

开发用于消除模型训练影响的非学习技术，研究了基于反向学习设置的成员推断攻击 (U-MIA)，发现不同类别的 U-MIA 强度与基于视觉和语言模型的现有非学习技术所提供的隐私保护高度相关，同时揭示了几种非学习算法对不同训练示例的潜在漏洞，以及保护所有示例的困难和试图适应停止条件的失败。