Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We find these robust pre-trained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3.83% on robust accuracy and 1.3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline. Moreover, we find that different self-supervised pre-trained models have a diverse adversarial vulnerability. It inspires us to ensemble several pretraining tasks, which boosts robustness more. Our ensemble strategy contributes to a further improvement of 3.59% on robust accuracy, while maintaining a slightly higher standard accuracy on CIFAR-10. Our codes are available at https://github.com/TAMU-VITA/Adv-SS-Pretraining.

本研究将对抗训练引入自监督学习中，提出了一种通用的具有强健性的预训练模型，可以在提高最终模型的强健性和节省计算成本方面，对后续的微调任务有两种作用。我们进行了大量实验，证明了所提出的框架与传统的对抗训练基线相比，在CIFAR-10数据集上可以获得大幅的性能提升。同时，我们发现不同的自监督预训练模型具有不同的对抗性漏洞，并提出了一种集成策略来提高其强健性。

对抗鲁棒性：从自监督预训练到微调