Self-supervised learning (SSL) has achieved remarkable success across various
speech-processing tasks. To enhance its efficiency, previous works often
leverage the use of compression techniques. A notable recent attempt is
DPHuBERT, which applies joint knowledge distillation (KD) and structured
pruning to learn a significantly smaller SSL model. In this paper, we
contribute to this research domain by introducing SKILL, a novel method that
conducts distillation across groups of layers instead of distilling individual
arbitrarily selected layers within the teacher network. The identification of
the layers to distill is achieved through a hierarchical clustering procedure
applied to layer similarity measures. Extensive experiments demonstrate that
our distilled version of WavLM Base+ not only outperforms DPHuBERT but also
achieves state-of-the-art results in the 30M parameters model class across
several SUPERB tasks.

通过引入 SKILL 方法，本文在自我监督学习研究领域提出了一种新的方法，通过层分组而非个别层进行知识蒸馏，实验结果表明，我们的 WavLM Base + 的蒸馏版本不仅优于 DPHuBERT，还在 30M 参数模型类的多个任务中达到了最先进的结果。

技能：相似度感知的语音自我监督学习知识蒸馏

SKILL: Similarity-aware Knowledge distILLation for Speech  Self-Supervised Learning

Self-supervised learning (SSL) has achieved notable success in many speech
processing tasks, but the large model size and heavy computational cost hinder
the deployment. Knowledge distillation trains a small student model to mimic
the behavior of a large teacher model. However, the student architecture
usually needs to be manually designed and will remain fixed during training,
which requires prior knowledge and can lead to suboptimal performance. Inspired
by recent success of task-specific structured pruning, we propose DPHuBERT, a
novel task-agnostic compression method for speech SSL based on joint
distillation and pruning. Experiments on SUPERB show that DPHuBERT outperforms
pure distillation methods in almost all tasks. Moreover, DPHuBERT requires
little training time and performs well with limited training data, making it
suitable for resource-constrained applications. Our method can also be applied
to various speech SSL models. Our code and models will be publicly available.

DPHuBERT 是一种基于联合蒸馏和修剪的任务无关压缩方法，适用于资源受限的应用，可应用于各种语音自监督学习模型，并优于纯蒸馏方法。