Recent advances in self-supervised speech models have shown significant
improvement in many downstream tasks. However, these models predominantly
centered on frame-level training objectives, which can fall short in spoken
language understanding tasks that require semantic comprehension. Existing
works often rely on additional speech-text data as intermediate targets, which
is costly in the real-world setting. To address this challenge, we propose
Pseudo-Word HuBERT (PW-HuBERT), a framework that integrates pseudo word-level
targets into the training process, where the targets are derived from a
visually-ground speech model, notably eliminating the need for speech-text
paired data. Our experimental results on four spoken language understanding
(SLU) benchmarks suggest the superiority of our model in capturing semantic
information.

我们提出了 Pseudo-Word HuBERT （PW-HuBERT）框架，该框架将伪词级目标集成到训练过程中，这些目标是从视觉引导的语音模型中获取的，明显消除了对语音 - 文本配对数据的需求，在四个口语理解基准测试中，我们的模型在捕捉语义信息方面表现出了优越性。