To solve deep neural network (DNN)'s huge training dataset and its high
computation issue, so-called teacher-student (T-S) DNN which transfers the
knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN
has limited range of use, and the knowledge of T-DNN is insufficiently
transferred to S-DNN. To improve the quality of the transferred knowledge from
T-DNN, we propose a new knowledge distillation using singular value
decomposition (SVD). In addition, we define a knowledge transfer as a
self-supervised task and suggest a way to continuously receive information from
T-DNN. Simulation results show that a S-DNN with a computational cost of 1/5 of
the T-DNN can be up to 1.1\% better than the T-DNN in terms of classification
accuracy. Also assuming the same computational cost, our S-DNN outperforms the
S-DNN driven by the state-of-the-art distillation with a performance advantage
of 1.79\%. code is available on this https URL

本文提出了一种使用奇异值分解来改善知识转移的新方法，将 T-DNN 的知识传递给 S-DNN 以提高计算效率，并通过自我监督的方式实现知识转移，模拟结果表明，在与 T-DNN 相同的计算成本下，我们的 S-DNN 比最先进的蒸馏技术驱动的 S-DNN 性能高出 1.79 个百分点，并且在分类准确性方面可比 T-DNN 好 1.1 个百分点。