Pre-trained speech encoders have been central to pushing state-of-the-art
results across various speech understanding and generation tasks. Nonetheless,
the capabilities of these encoders in low-resource settings are yet to be
thoroughly explored. To address this, we conduct a comprehensive set of
experiments using a representative set of 3 state-of-the-art encoders
(Wav2vec2, WavLM, Whisper) in the low-resource setting across 7 speech
understanding and generation tasks. We provide various quantitative and
qualitative analyses on task performance, convergence speed, and
representational properties of the encoders. We observe a connection between
the pre-training protocols of these encoders and the way in which they capture
information in their internal layers. In particular, we observe the Whisper
encoder exhibits the greatest low-resource capabilities on content-driven tasks
in terms of performance and convergence speed.

使用三种最先进的语音编码器，Wav2vec2、WavLM 和 Whisper，对于 7 个语音理解和生成任务在低资源环境下进行了全面的实验，研究结果表明，Whisper 编码器在执行与内容相关的任务时具有最好的性能和收敛速度。