Recent speech language models (SLMs) typically incorporate pre-trained speech
models to extend the capabilities from large language models (LLMs). In this
paper, we propose a Descriptive Speech-Text Alignment approach that leverages
speech captioning to bridge the gap between speech and text modalities,
enabling SLMs to interpret and generate comprehensive natural language
descriptions, thereby facilitating the capability to understand both linguistic
and non-linguistic features in speech. Enhanced with the proposed approach, our
model demonstrates superior performance on the Dynamic-SUPERB benchmark,
particularly in generalizing to unseen tasks. Moreover, we discover that the
aligned model exhibits a zero-shot instruction-following capability without
explicit speech instruction tuning. These findings highlight the potential to
reshape instruction-following SLMs by incorporating rich, descriptive speech
captions.

通过 Descriptive Speech-Text Alignment 方法，本研究提出了一种新型的语音语言模型，可以解释和生成全面的自然语言描述，从而促进对语音的语言和非语言特征的理解，并在 Dynamic-SUPERB 基准测试中展现出卓越性能，尤其是在未见过的任务上的泛化能力，同时发现对齐模型具有零样本指令跟随能力，无需显式调整语音指令，这些发现突显了通过加入丰富的描述性语音字幕来改变指令跟随语音语言模型的潜力。