This paper explores the instruction fine-tuning technique for speech semantic
understanding by introducing a unified end-to-end (E2E) framework that
generates semantic labels conditioned on a task-related prompt for audio data.
We pre-train the model using large and diverse data, where instruction-speech
pairs are constructed via a text-to-speech (TTS) system. Extensive experiments
demonstrate that our proposed model significantly outperforms state-of-the-art
(SOTA) models after fine-tuning downstream tasks. Furthermore, the proposed
model achieves competitive performance in zero-shot and few-shot scenarios. To
facilitate future work on instruction fine-tuning for speech-to-semantic tasks,
we release our instruction dataset and code.

该论文通过引入统一的端到端（E2E）框架，对语音语义理解任务进行了指令微调技术的探索，该框架在语音数据上生成与任务相关提示条件下的语义标签。实验证明，我们提出的模型在微调下游任务后显著优于最先进的模型，并且在零样本和少样本情况下取得了有竞争力的性能。为了促进未来对语音到语义任务的指令微调工作，我们发布了我们的指令数据集和代码。