Recent work on speech representation models jointly pre-trained with text has
demonstrated the potential of improving speech representations by encoding
speech and text in a shared space. In this paper, we leverage such shared
representations to address the persistent challenge of limited data
availability in spoken language understanding tasks. By employing a pre-trained
speech-text model, we find that models fine-tuned on text can be effectively
transferred to speech testing data. With as little as 1 hour of labeled speech
data, our proposed approach achieves comparable performance on spoken language
understanding tasks (specifically, sentiment analysis and named entity
recognition) when compared to previous methods using speech-only pre-trained
models fine-tuned on 10 times more data. Beyond the proof-of-concept study, we
also analyze the latent representations. We find that the bottom layers of
speech-text models are largely task-agnostic and align speech and text
representations into a shared space, while the top layers are more
task-specific.

通过使用预训练的语音 - 文本模型，本研究发现只需 1 小时标注的语音数据，即可与仅使用 10 倍数据的仅语音预训练模型在口语理解任务（情感分析和命名实体识别）上取得可比较的性能；同时发现底层的语音 - 文本模型作为任务自主层面，在共享空间中对齐语音和文本表示，而顶层则更加任务特定。