In this paper, we review adversarial pretraining of self-supervised deep
networks including both convolutional neural networks and vision transformers.
Unlike the adversarial training with access to labeled examples, adversarial
pretraining is complicated as it only has access to unlabeled examples. To
incorporate adversaries into pretraining models on either input or feature
level, we find that existing approaches are largely categorized into two
groups: memory-free instance-wise attacks imposing worst-case perturbations on
individual examples, and memory-based adversaries shared across examples over
iterations. In particular, we review several representative adversarial
pretraining models based on Contrastive Learning (CL) and Masked Image Modeling
(MIM), respectively, two popular self-supervised pretraining methods in
literature. We also review miscellaneous issues about computing overheads,
input-/feature-level adversaries, as well as other adversarial pretraining
approaches beyond the above two groups. Finally, we discuss emerging trends and
future directions about the relations between adversarial and cooperative
pretraining, unifying adversarial CL and MIM pretraining, and the trade-off
between accuracy and robustness in adversarial pretraining.

本文综述了对抗性预训练的自监督深度网络，包括卷积神经网络和视觉变换器。我们发现现有方法主要分为两组：针对单个样例进行的无内存实例攻击和在迭代中跨样例共享的基于内存的对手。特别地，我们回顾了基于对比学习（CL）和掩码图像建模（MIM）的几个典型的对抗性预训练模型，讨论了计算开销、输入 / 特征级对手以及其他对抗性预训练方法等杂项问题，最后，我们讨论了关于对抗性和协作式预训练之间关系、统一对比度和 MIM 预训练的未来趋势以及在对抗性预训练中准确性和鲁棒性之间的权衡。