Tiny, causal models are crucial for embedded audio machine learning
applications. Model compression can be achieved via distilling knowledge from a
large teacher into a smaller student model. In this work, we propose a novel
two-step approach for tiny speech enhancement model distillation. In contrast
to the standard approach of a weighted mixture of distillation and supervised
losses, we firstly pre-train the student using only the knowledge distillation
(KD) objective, after which we switch to a fully supervised training regime. We
also propose a novel fine-grained similarity-preserving KD loss, which aims to
match the student's intra-activation Gram matrices to that of the teacher. Our
method demonstrates broad improvements, but particularly shines in adverse
conditions including high compression and low signal to noise ratios (SNR),
yielding signal to distortion ratio gains of 0.9 dB and 1.1 dB, respectively,
at -5 dB input SNR and 63x compression compared to baseline.

借助知识蒸馏技术，我们提出了一种新的两步法来压缩 Tiny 语音增强模型，其中包括先用知识蒸馏目标进行预训练，然后切换到完全监督训练模式，同时提出了一种新的细粒度相似性保持 KD 损失函数，该方法在高压缩和低信噪比条件下表现出突出的改进，相对于基线，在输入信噪比为 - 5dB 和 63 倍压缩的情况下，信噪失真比分别提高了 0.9dB 和 1.1dB。