The Spoken Language Understanding Evaluation (SLUE) suite of benchmark tasks
was recently introduced to address the need for open resources and benchmarking
of complex spoken language understanding (SLU) tasks, including both
classification and sequence generation tasks, on natural speech. The benchmark
has demonstrated preliminary success in using pre-trained speech foundation
models (SFM) for these SLU tasks. However, the community still lacks a
fine-grained understanding of the comparative utility of different SFMs.
Inspired by this, we ask: which SFMs offer the most benefits for these complex
SLU tasks, and what is the most effective approach for incorporating these
SFMs? To answer this, we perform an extensive evaluation of multiple supervised
and self-supervised SFMs using several evaluation protocols: (i) frozen SFMs
with a lightweight prediction head, (ii) frozen SFMs with a complex prediction
head, and (iii) fine-tuned SFMs with a lightweight prediction head. Although
the supervised SFMs are pre-trained on much more speech recognition data (with
labels), they do not always outperform self-supervised SFMs; the latter tend to
perform at least as well as, and sometimes better than, supervised SFMs,
especially on the sequence generation tasks in SLUE. While there is no
universally optimal way of incorporating SFMs, the complex prediction head
gives the best performance for most tasks, although it increases the inference
time. We also introduce an open-source toolkit and performance leaderboard,
SLUE-PERB, for these tasks and modeling strategies.

通过对多个受监督和自监督 SFMs 进行广泛评估，我们发现，尽管受监督的 SFMs 在语音识别数据上已经进行了预训练（带有标签），但并不总是表现优于自监督的 SFMs；后者在 SLUE 的序列生成任务上往往至少与受监督的 SFMs 表现一样出色，有时甚至更好。对于大多数任务来说，复杂预测头的性能最佳，尽管会增加推理时间。我们还引入了一个开源工具包和性能排行榜 SLUE-PERB，用于这些任务和建模策略。