Collecting sufficient labeled data for spoken language understanding (SLU) is
expensive and time-consuming. Recent studies achieved promising results by
using pre-trained models in low-resource scenarios. Inspired by this, we aim to
ask: which (if any) pre-training strategies can improve performance across SLU
benchmarks? To answer this question, we employ four types of pre-trained models
and their combinations for SLU. We leverage self-supervised speech and language
models (LM) pre-trained on large quantities of unpaired data to extract strong
speech and text representations. We also explore using supervised models
pre-trained on larger external automatic speech recognition (ASR) or SLU
corpora. We conduct extensive experiments on the SLU Evaluation (SLUE)
benchmark and observe self-supervised pre-trained models to be more powerful,
with pre-trained LM and speech models being most beneficial for the Sentiment
Analysis and Named Entity Recognition task, respectively.

本研究旨在探索使用预训练策略增强口语理解任务性能，通过自监督的语音和语言模型在 SLU 测评基准上进行实验，发现预训练模型可于低资源环境中提升结果，其中以预训练的语言和语音模型分别在情感分析和命名实体识别任务上效果最突出。