Transfer learning provides an effective solution for feasibly and fast
customize accurate \textit{Student} models, by transferring the learned
knowledge of pre-trained \textit{Teacher} models over large datasets via
fine-tuning. Many pre-trained Teacher models used in transfer learning are
publicly available and maintained by public platforms, increasing their
vulnerability to backdoor attacks. In this paper, we demonstrate a backdoor
threat to transfer learning tasks on both image and time-series data leveraging
the knowledge of publicly accessible Teacher models, aimed at defeating three
commonly-adopted defenses: \textit{pruning-based}, \textit{retraining-based}
and \textit{input pre-processing-based defenses}. Specifically, (A)
ranking-based selection mechanism to speed up the backdoor trigger generation
and perturbation process while defeating \textit{pruning-based} and/or
\textit{retraining-based defenses}. (B) autoencoder-powered trigger generation
is proposed to produce a robust trigger that can defeat the \textit{input
pre-processing-based defense}, while guaranteeing that selected neuron(s) can
be significantly activated. (C) defense-aware retraining to generate the
manipulated model using reverse-engineered model inputs.
We launch effective misclassification attacks on Student models over
real-world images, brain Magnetic Resonance Imaging (MRI) data and
Electrocardiography (ECG) learning systems. The experiments reveal that our
enhanced attack can maintain the $98.4\%$ and $97.2\%$ classification accuracy
as the genuine model on clean image and time series inputs respectively while
improving $27.9\%-100\%$ and $27.1\%-56.1\%$ attack success rate on trojaned
image and time series inputs respectively in the presence of pruning-based
and/or retraining-based defenses.

本文提出了一种利用公共可访问 Teacher Models 的知识展开的转移学习任务中的后门威胁，用于攻击真实世界图像和时间序列，提出了三种通常采用的防御类型：基于修剪的防御，基于重训练的防御和基于输入预处理的防御。

针对使用预训练深度学习模型的迁移学习的后门攻击

Backdoor Attacks against Transfer Learning with Pre-trained Deep  Learning Models

We present a novel framework of knowledge distillation that is capable of
learning powerful and efficient student models from ensemble teacher networks.
Our approach addresses the inherent model capacity issue between teacher and
student and aims to maximize benefit from teacher models during distillation by
reducing their capacity gap. Specifically, we employ a neural architecture
search technique to augment useful structures and operations, where the
searched network is appropriate for knowledge distillation towards student
models and free from sacrificing its performance by fixing the network
capacity. We also introduce an oracle knowledge distillation loss to facilitate
model search and distillation using an ensemble-based teacher model, where a
student network is learned to imitate oracle performance of the teacher. We
perform extensive experiments on the image classification datasets---CIFAR-100
and TinyImageNet---using various networks. We also show that searching for a
new student model is effective in both accuracy and memory size and that the
searched models often outperform their teacher models thanks to neural
architecture search with oracle knowledge distillation.

提出了一种新的知识蒸馏框架，利用神经架构搜索技术和神经网络来学习深度神经网络教师模型以获得高效的学生模型。该方法通过减少教师和学生之间的模型容量差距，从而最大化从教师模型进行蒸馏的收益，我们在基于 CIFAR-100 和 TinyImageNet 的图像分类数据集上进行了广泛的实验，并通过神经架构搜索技术发现新的学生模型，其在准确性和内存使用上都比教师模型表现更好。