Any continuous function $f^*$ can be approximated arbitrarily well by a
neural network with sufficiently many neurons $k$. We consider the case when
$f^*$ itself is a neural network with one hidden layer and $k$ neurons.
Approximating $f^*$ with a neural network with $n< k$ neurons can thus be seen
as fitting an under-parameterized "student" network with $n$ neurons to a
"teacher" network with $k$ neurons. As the student has fewer neurons than the
teacher, it is unclear, whether each of the $n$ student neurons should copy one
of the teacher neurons or rather average a group of teacher neurons. For
shallow neural networks with erf activation function and for the standard
Gaussian input distribution, we prove that "copy-average" configurations are
critical points if the teacher's incoming vectors are orthonormal and its
outgoing weights are unitary. Moreover, the optimum among such configurations
is reached when $n-1$ student neurons each copy one teacher neuron and the
$n$-th student neuron averages the remaining $k-n+1$ teacher neurons. For the
student network with $n=1$ neuron, we provide additionally a closed-form
solution of the non-trivial critical point(s) for commonly used activation
functions through solving an equivalent constrained optimization problem.
Empirically, we find for the erf activation function that gradient flow
converges either to the optimal copy-average critical point or to another point
where each student neuron approximately copies a different teacher neuron.
Finally, we find similar results for the ReLU activation function, suggesting
that the optimal solution of underparameterized networks has a universal
structure.

通过神经网络可以对任何连续函数进行无限接近的逼近，本文讨论了当函数本身是一个具有一层隐藏神经元和 k 个神经元的神经网络时，使用具有 n 个神经元的神经网络对 f * 进行逼近可以看作是将一个欠参数化的 “学生” 网络与一个具有 k 个神经元的 “教师” 网络进行拟合，提供了对传统激活函数的非平凡临界点的闭式解，并发现对于 erf 激活函数，梯度流要么收敛到最优的拷贝 - 平均临界点，要么收敛到每个学生神经元大致拷贝不同的教师神经元的另一个临界点，最后发现 ReLU 激活函数也有类似的结果，表明欠参数化网络的最优解具有普遍结构。

欠参数化的学生网络应该复制还是平均教师权重？

Should Under-parameterized Student Networks Copy or Average Teacher  Weights?

Although deep neural networks have enjoyed remarkable success across a wide
variety of tasks, their ever-increasing size also imposes significant overhead
on deployment. To compress these models, knowledge distillation was proposed to
transfer knowledge from a cumbersome (teacher) network into a lightweight
(student) network. However, guidance from a teacher does not always improve the
generalization of students, especially when the size gap between student and
teacher is large. Previous works argued that it was due to the high certainty
of the teacher, resulting in harder labels that were difficult to fit. To
soften these labels, we present a pruning method termed Prediction Uncertainty
Enlargement (PrUE) to simplify the teacher. Specifically, our method aims to
decrease the teacher's certainty about data, thereby generating soft
predictions for students. We empirically investigate the effectiveness of the
proposed method with experiments on CIFAR-10/100, Tiny-ImageNet, and ImageNet.
Results indicate that student networks trained with sparse teachers achieve
better performance. Besides, our method allows researchers to distill knowledge
from deeper networks to improve students further. Our code is made public at:
https://github.com/wangshaopu/prue.

提出一种称为预测不确定性增大（PrUE）的剪枝方法，通过减弱教师网络的确定性，生成学生网络所需的软性预测，从而提高学生网络性能。在 CIFAR-10/100，Tiny-ImageNet 和 ImageNet 上进行的实验证明，使用稀疏的教师网络训练学生网络可以达到更好的性能，并且该方法允许研究者从更深层的网络中提取知识以进一步提高学生性能。

PrUE: 从稀疏教师网络中提炼知识

PrUE: Distilling Knowledge from Sparse Teacher Networks

This paper addresses the problem of model compression via knowledge
distillation. To this end, we propose a new knowledge distillation method based
on transferring feature statistics, specifically the channel-wise mean and
variance, from the teacher to the student. Our method goes beyond the standard
way of enforcing the mean and variance of the student to be similar to those of
the teacher through an $L_2$ loss, which we found it to be of limited
effectiveness. Specifically, we propose a new loss based on adaptive instance
normalization to effectively transfer the feature statistics. The main idea is
to transfer the learned statistics back to the teacher via adaptive instance
normalization (conditioned on the student) and let the teacher network
"evaluate" via a loss whether the statistics learned by the student are
reliably transferred. We show that our distillation method outperforms other
state-of-the-art distillation methods over a large set of experimental settings
including different (a) network architectures, (b) teacher-student capacities,
(c) datasets, and (d) domains.

本文提出了一种新的知识蒸馏方法，基于转移来自教师到学生的通道均值和方差等特征统计信息，以及新的适应性实例归一化损失，以提高模型压缩效果。