Adversarial attacks pose a significant threat to the security and safety of
deep neural networks being applied to modern applications. More specifically,
in computer vision-based tasks, experts can use the knowledge of model
architecture to create adversarial samples imperceptible to the human eye.
These attacks can lead to security problems in popular applications such as
self-driving cars, face recognition, etc. Hence, building networks which are
robust to such attacks is highly desirable and essential. Among the various
methods present in literature, defensive distillation has shown promise in
recent years. Using knowledge distillation, researchers have been able to
create models robust against some of those attacks. However, more attacks have
been developed exposing weakness in defensive distillation. In this project, we
derive inspiration from teacher assistant knowledge distillation and propose
that introducing an assistant network can improve the robustness of the
distilled model. Through a series of experiments, we evaluate the distilled
models for different distillation temperatures in terms of accuracy,
sensitivity, and robustness. Our experiments demonstrate that the proposed
hypothesis can improve robustness in most cases. Additionally, we show that
multi-step distillation can further improve robustness with very little impact
on model accuracy.

该项目研究深度神经网络中的对抗攻击并探讨防御性蒸馏的方法，提出通过引入辅助网络来改善蒸馏模型的鲁棒性，并通过实验证明该方法的有效性及其对模型准确性的影响。

利用助教提高防御性蒸馏

Improving Defensive Distillation using Teacher Assistant

Despite the fact that deep neural networks are powerful models and achieve
appealing results on many tasks, they are too large to be deployed on edge
devices like smartphones or embedded sensor nodes. There have been efforts to
compress these networks, and a popular method is knowledge distillation, where
a large (teacher) pre-trained network is used to train a smaller (student)
network. However, in this paper, we show that the student network performance
degrades when the gap between student and teacher is large. Given a fixed
student network, one cannot employ an arbitrarily large teacher, or in other
words, a teacher can effectively transfer its knowledge to students up to a
certain size, not smaller. To alleviate this shortcoming, we introduce
multi-step knowledge distillation, which employs an intermediate-sized network
(teacher assistant) to bridge the gap between the student and the teacher.
Moreover, we study the effect of teacher assistant size and extend the
framework to multi-step distillation. Theoretical analysis and extensive
experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet
architectures substantiate the effectiveness of our proposed approach.

本论文针对深度神经网络过于庞大以至于不能部署在移动设备等边缘设备的问题，提出了一种基于知识蒸馏的网络压缩方法。然而，该论文表明，如果学生网络与教师网络之间的差距太大，那么知识蒸馏的性能会下降；为此，该论文提出了一种多步知识蒸馏的方法，通过使用一个中等大小的网络（即教师助手）来弥补学生与教师之间的差距，并通过对 CIFAR-10,100 和 ImageNet 数据集进行广泛的实验证明了该方法的有效性。