Recent advancements in Machine Unlearning (MU) have introduced solutions to
selectively remove certain training samples, such as those with outdated or
sensitive information, from trained models. Despite these advancements,
evaluation of MU methods have been inconsistent, employing different trained
models and architectures, and sample removal strategies, which hampers accurate
comparison. In addition, prior MU approaches have mainly focused on singular
tasks or modalities, which is not comprehensive. To address these limitations,
we develop MU-Bench, the first comprehensive benchmark for MU that (i) unifies
the sets of deleted samples and trained models, and (ii) provides broad
coverage of tasks and data modalities, including previously unexplored domains
such as speech and video classification. Our evaluation show that RandLabel and
SalUn are the most effective general MU approaches on MU-Bench, and BadT and
SCRUB are capable of achieving random performance on the deletion set. We
analyze several under-investigated aspects of unlearning, including
scalability, the impacts of parameter-efficient fine-tuning and curriculum
learning, and susceptibility to dataset biases. MU-Bench provides an
easy-to-use package that includes dataset splits, models, and implementations,
together with a leader board to enable unified and scalable MU research.

为了解决目前机器遗忘研究领域的不一致性和局限性，研究者们开发了 MU-Bench，这是第一个综合性的机器遗忘基准测试，它统一了删除样本和训练模型的集合，并广泛涵盖了任务和数据模态，包括语音和视频分类， evaluation 结果表明，RandLabel 和 SalUn 是 MU-Bench 上最有效的一般机器遗忘方法，而 BadT 和 SCRUB 能够在删除集合上达到随机性能，此外，研究还分析了一些尚未完全研究的机器遗忘方面，包括可扩展性，参数高效微调和课程学习的影响，以及对数据集偏差的敏感性，MU-Bench 提供了易于使用的工具包，包含了数据集划分，模型和实现，以及一个排行榜，以促进统一和可扩展的机器遗忘研究。