Fine-tuning through knowledge transfer from a pre-trained model on a
large-scale dataset is a widely spread approach to effectively build models on
small-scale datasets. In this work, we show that a recent adversarial attack
designed for transfer learning via re-training the last linear layer can
successfully deceive models trained with transfer learning via end-to-end
fine-tuning. This raises security concerns for many industrial applications. In
contrast, models trained with random initialization without transfer are much
more robust to such attacks, although these models often exhibit much lower
accuracy. To this end, we propose noisy feature distillation, a new transfer
learning method that trains a network from random initialization while
achieving clean-data performance competitive with fine-tuning. Code available
at this https URL

本文研究表明，用预先训练的大规模数据集上的模型进行知识转移进行微调，可以帮助构建在小规模数据集上有效的模型。通过针对最后一个线性层重新训练的转移学习的最新对抗性攻击可以成功欺骗通过端到端微调进行转移学习的模型，这引起了许多工业应用的安全问题。相比之下，没有转移训练的随机初始化的模型通常表现出更强的鲁棒性，虽然这些模型的准确性往往较低。为此，我们提出了一种称为嘈杂特征蒸馏的新的转移学习方法，可以从随机初始化的网络进行训练，同时可以达到与微调相竞争的清洁数据性能。