Knowledge Distillation (KD) facilitates the transfer of discriminative
capabilities from an advanced teacher model to a simpler student model,
ensuring performance enhancement without compromising accuracy. It is also
exploited for model stealing attacks, where adversaries use KD to mimic the
functionality of a teacher model. Recent developments in this domain have been
influenced by the Stingy Teacher model, which provided empirical analysis
showing that sparse outputs can significantly degrade the performance of
student models. Addressing the risk of intellectual property leakage, our work
introduces an approach to train a teacher model that inherently protects its
logits, influenced by the Nasty Teacher concept. Differing from existing
methods, we incorporate sparse outputs of adversarial examples with standard
training data to strengthen the teacher's defense against student distillation.
Our approach carefully reduces the relative entropy between the original and
adversarially perturbed outputs, allowing the model to produce adversarial
logits with minimal impact on overall performance. The source codes will be
made publicly available soon.

通过在训练数据中结合稀疏输出与对抗样本，以增强教师模型对学生模型的防御，我们的研究引入了一种保护其输出的教师模型的方法，从而减少了对整体性能的影响。

对抗稀疏教师：利用对抗样本防御基于蒸馏的模型窃取攻击

Adversarial Sparse Teacher: Defense Against Distillation-Based Model  Stealing Attacks Using Adversarial Examples

This work aims to address the long-established problem of learning
diversified representations. To this end, we combine information-theoretic
arguments with stochastic competition-based activations, namely Stochastic
Local Winner-Takes-All (LWTA) units. In this context, we ditch the conventional
deep architectures commonly used in Representation Learning, that rely on
non-linear activations; instead, we replace them with sets of locally and
stochastically competing linear units. In this setting, each network layer
yields sparse outputs, determined by the outcome of the competition between
units that are organized into blocks of competitors. We adopt stochastic
arguments for the competition mechanism, which perform posterior sampling to
determine the winner of each block. We further endow the considered networks
with the ability to infer the sub-part of the network that is essential for
modeling the data at hand; we impose appropriate stick-breaking priors to this
end. To further enrich the information of the emerging representations, we
resort to information-theoretic principles, namely the Information Competing
Process (ICP). Then, all the components are tied together under the stochastic
Variational Bayes framework for inference. We perform a thorough experimental
investigation for our approach using benchmark datasets on image
classification. As we experimentally show, the resulting networks yield
significant discriminative representation learning abilities. In addition, the
introduced paradigm allows for a principled investigation mechanism of the
emerging intermediate network representations.

通过将信息论和随机竞争激活结合起来，本文提出一种去除常规深度学习架构中非线性激活的方法，使用局部和随机竞争的线性单元替代，从而实现学习多样化表征，同时在图像分类基准数据集上进行了实验验证，结果表明该方法具有显著的表征学习能力和推断机制。