Self-supervised speech representation learning aims to extract meaningful
factors from the speech signal that can later be used across different
downstream tasks, such as speech and/or emotion recognition. Existing models,
such as HuBERT, however, can be fairly large thus may not be suitable for edge
speech applications. Moreover, realistic applications typically involve speech
corrupted by noise and room reverberation, hence models need to provide
representations that are robust to such environmental factors. In this study,
we build on the so-called DistilHuBERT model, which distils HuBERT to a
fraction of its original size, with three modifications, namely: (i) augment
the training data with noise and reverberation, while the student model needs
to distill the clean representations from the teacher model; (ii) introduce a
curriculum learning approach where increasing levels of noise are introduced as
the model trains, thus helping with convergence and with the creation of more
robust representations; and (iii) introduce a multi-task learning approach
where the model also reconstructs the clean waveform jointly with the
distillation task, thus also acting as an enhancement step to ensure additional
environment robustness to the representation. Experiments on three SUPERB tasks
show the advantages of the proposed method not only relative to the original
DistilHuBERT, but also to the original HuBERT, thus showing the advantages of
the proposed method for ``in the wild'' edge speech applications.

本研究提出了一种改进的 DistilHuBERT 模型，该模型引入了噪声和混响以及多任务学习的训练方法，能够有效提高在真实环境下的边缘语音应用中的鲁棒性。