Pre-training serves as a broadly adopted starting point for transfer learning
on various downstream tasks. Recent investigations of lottery tickets
hypothesis (LTH) demonstrate such enormous pre-trained models can be replaced
by extremely sparse subnetworks (a.k.a. matching subnetworks) without
sacrificing transferability. However, practical security-crucial applications
usually pose more challenging requirements beyond standard transfer, which also
demand these subnetworks to overcome adversarial vulnerability. In this paper,
we formulate a more rigorous concept, Double-Win Lottery Tickets, in which a
located subnetwork from a pre-trained model can be independently transferred on
diverse downstream tasks, to reach BOTH the same standard and robust
generalization, under BOTH standard and adversarial training regimes, as the
full pre-trained model can do. We comprehensively examine various pre-training
mechanisms and find that robust pre-training tends to craft sparser double-win
lottery tickets with superior performance over the standard counterparts. For
example, on downstream CIFAR-10/100 datasets, we identify double-win matching
subnetworks with the standard, fast adversarial, and adversarial pre-training
from ImageNet, at 89.26%/73.79%, 89.26%/79.03%, and 91.41%/83.22% sparsity,
respectively. Furthermore, we observe the obtained double-win lottery tickets
can be more data-efficient to transfer, under practical data-limited (e.g., 1%
and 10%) downstream schemes. Our results show that the benefits from robust
pre-training are amplified by the lottery ticket scheme, as well as the
data-limited transfer setting. Codes are available at
this https URL.

本研究提出了一种 “双赢彩票模型”，在同时具有标准训练和对抗训练的下游任务中， 可独立转移预训练模型的定位子网络，以达到与完整预训练模型相同的标准和强健的识别泛化效果。然后，本研究探讨了各种预先训练机制，发现强健的预先训练倾向于制作更稀疏的双赢彩票模型，并且相较于标准机制，在实际数据有限的情况下会更加高效。