Distilled self-supervised models have shown competitive performance and
efficiency in recent years. However, there is a lack of experience in jointly
distilling multiple self-supervised speech models. In our work, we performed
Ensemble Knowledge Distillation (EKD) on various self-supervised speech models
such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation
techniques, layerwise-average and layerwise-concatenation, to the
representations of different teacher models and found that the former was more
effective. On top of that, we proposed a multiple prediction head method for
student models to predict different layer outputs of multiple teacher models
simultaneously. The experimental results show that our method improves the
performance of the distilled models on four downstream speech processing tasks,
Phoneme Recognition, Speaker Identification, Emotion Recognition, and Automatic
Speech Recognition in the hidden-set track of the SUPERB benchmark.

本文提出了一种多自监督语音模型集成的方法，通过使用层次平均聚合不同教师模型的表示，并为学生模型提出了多重预测头方法，从而大大提高了四个下游语音处理任务的性能，并在 SUPERB 基准的隐藏集轨道中取得了显著的成果。