In theoretical ML, the teacher-student paradigm is often employed as an
effective metaphor for real-life tuition. The above scheme proves particularly
relevant when the student network is overparameterized as compared to the
teacher network. Under these operating conditions, it is tempting to speculate
that the student ability to handle the given task could be eventually stored in
a sub-portion of the whole network. This latter should be to some extent
reminiscent of the frozen teacher structure, according to suitable metrics,
while being approximately invariant across different architectures of the
student candidate network. Unfortunately, state-of-the-art conventional
learning techniques could not help in identifying the existence of such an
invariant subnetwork, due to the inherent degree of non-convexity that
characterizes the examined problem. In this work, we take a leap forward by
proposing a radically different optimization scheme which builds on a spectral
representation of the linear transfer of information between layers. The
gradient is hence calculated with respect to both eigenvalues and eigenvectors
with negligible increase in terms of computational and complexity load, as
compared to standard training algorithms. Working in this framework, we could
isolate a stable student substructure, that mirrors the true complexity of the
teacher in terms of computing neurons, path distribution and topological
attributes. When pruning unimportant nodes of the trained student, as follows a
ranking that reflects the optimized eigenvalues, no degradation in the recorded
performance is seen above a threshold that corresponds to the effective teacher
size. The observed behavior can be pictured as a genuine second-order phase
transition that bears universality traits.

在本研究中，我们通过基于线性信息传递的谱表示的全新优化方案，成功地分离出一种稳定的学生子结构，其在计算神经元、路径分布和拓扑属性方面反映了教师的真实复杂性；当按照优化后的特征值排名修剪训练后的学生的不重要节点时，在达到有效教师规模的阈值以上，观察到性能没有下降，这种行为可以被看作是具有普遍性特征的真正的二阶相变。