In this paper, we study a defense against poisoned encoders in SSL called
distillation, which is a defense used in supervised learning originally.
Distillation aims to distill knowledge from a given model (a.k.a the teacher
net) and transfer it to another (a.k.a the student net). Now, we use it to
distill benign knowledge from poisoned pre-trained encoders and transfer it to
a new encoder, resulting in a clean pre-trained encoder. In particular, we
conduct an empirical study on the effectiveness and performance of distillation
against poisoned encoders. Using two state-of-the-art backdoor attacks against
pre-trained image encoders and four commonly used image classification
datasets, our experimental results show that distillation can reduce attack
success rate from 80.87% to 27.51% while suffering a 6.35% loss in accuracy.
Moreover, we investigate the impact of three core components of distillation on
performance: teacher net, student net, and distillation loss. By comparing 4
different teacher nets, 3 student nets, and 6 distillation losses, we find that
fine-tuned teacher nets, warm-up-training-based student nets, and
attention-based distillation loss perform best, respectively.

本研究探索了在 SSL 中针对恶意编码器的一种叫作蒸馏的防御方法，蒸馏最初用于监督学习，其目的是从给定模型（即教师网络）中提取知识并传递给另一个模型（即学生网络），现在我们使用它来从被恶意植入的预训练编码器中提取良性知识并传递给一个新的编码器，以获得一个干净的预训练编码器。通过对两种顶级的预训练图像编码器和四种常用的图像分类数据集进行实验研究，我们的实验结果表明，蒸馏可以将攻击成功率从 80.87% 降低至 27.51%，但精度损失为 6.35%。此外，我们研究了蒸馏的三个核心组成部分：教师网络、学生网络和蒸馏损失对性能的影响，通过比较 4 个不同的教师网络、3 个不同的学生网络和 6 种不同的蒸馏损失，我们发现，精调过的教师网络、基于预训练训练的学生网络和基于注意力的蒸馏损失效果最佳。