Distilled self-supervised models have shown competitive performance and
efficiency in recent years. However, there is a lack of experience in jointly
distilling multiple self-supervised speech models. In our work, we performed
Ensemble Knowledge Distillation (EKD) on various self-supervised speech models
such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation
techniques, layerwise-average and layerwise-concatenation, to the
representations of different teacher models and found that the former was more
effective. On top of that, we proposed a multiple prediction head method for
student models to predict different layer outputs of multiple teacher models
simultaneously. The experimental results show that our method improves the
performance of the distilled models on four downstream speech processing tasks,
Phoneme Recognition, Speaker Identification, Emotion Recognition, and Automatic
Speech Recognition in the hidden-set track of the SUPERB benchmark.

本文提出了一种多自监督语音模型集成的方法，通过使用层次平均聚合不同教师模型的表示，并为学生模型提出了多重预测头方法，从而大大提高了四个下游语音处理任务的性能，并在 SUPERB 基准的隐藏集轨道中取得了显著的成果。

自监督语音模型的集成知识蒸馏

Ensemble knowledge distillation of self-supervised speech models

Ensemble knowledge distillation can extract knowledge from multiple teacher
models and encode it into a single student model. Many existing methods learn
and distill the student model on labeled data only. However, the teacher models
are usually learned on the same labeled data, and their predictions have high
correlations with groudtruth labels. Thus, they cannot provide sufficient
knowledge complementary to task labels for student teaching. Distilling on
unseen unlabeled data has the potential to enhance the knowledge transfer from
the teachers to the student. In this paper, we propose a unified and effective
ensemble knowledge distillation method that distills a single student model
from an ensemble of teacher models on both labeled and unlabeled data. Since
different teachers may have diverse prediction correctness on the same sample,
on labeled data we weight the predictions of different teachers according to
their correctness. In addition, we weight the distillation loss based on the
overall prediction correctness of the teacher ensemble to distill high-quality
knowledge. On unlabeled data, there is no groundtruth to evaluate prediction
correctness. Fortunately, the disagreement among teachers is an indication of
sample hardness, and thereby we weight the distillation loss based on teachers'
disagreement to emphasize knowledge distillation on important samples.
Extensive experiments on four datasets show the effectiveness of our proposed
ensemble distillation method.

本文提出了一种有效的集成知识蒸馏方法，该方法能够从多个教师模型学习未标记数据的知识，并据此训练单个学生模型。研究表明，通过考虑不同教师之间的预测差异以及样本难度，可以进一步提高蒸馏的效果。