Learning on a massive amount of speech corpus leads to the recent success of
many self-supervised speech models. With knowledge distillation, these models
may also benefit from the knowledge encoded by language models that are
pre-trained on rich sources of texts. The distillation process, however, is
challenging due to the modal disparity between textual and speech embedding
spaces. This paper studies metric-based distillation to align the embedding
space of text and speech with only a small amount of data without modifying the
model structure. Since the semantic and granularity gap between text and speech
has been omitted in literature, which impairs the distillation, we propose the
Prior-informed Adaptive knowledge Distillation (PAD) that adaptively leverages
text/speech units of variable granularity and prior distributions to achieve
better global and local alignments between text and speech pre-trained models.
We evaluate on three spoken language understanding benchmarks to show that PAD
is more effective in transferring linguistic knowledge than other metric-based
distillation approaches.

本文介绍了一种通过度量学进行知识蒸馏以改善文本和音频的嵌入向量的对齐，提出了 Prior-informed Adaptive knowledge Distillation (PAD) 方法，该方法具有更好的文本语音模型之间的传输能力，我们在三个口语理解基准测试中进行了评估。