The integration of pre-trained text-based large language models (LLM) with
speech input has enabled instruction-following capabilities for diverse speech
tasks. This integration requires the use of a speech encoder, a speech adapter,
and an LLM, trained on diverse tasks. We propose the use of discrete speech
units (DSU), rather than continuous-valued speech encoder outputs, that are
converted to the LLM token embedding space using the speech adapter. We
generate DSU using a self-supervised speech encoder followed by k-means
clustering. The proposed model shows robust performance on speech inputs from
seen/unseen domains and instruction-following capability in spoken question
answering. We also explore various types of DSU extracted from different layers
of the self-supervised speech encoder, as well as Mel frequency Cepstral
Coefficients (MFCC). Our findings suggest that the ASR task and datasets are
not crucial in instruction-tuning for spoken question answering tasks.

通过将预训练的基于文本的大型语言模型（LLM）与语音输入集成，我们提出使用离散语音单元（DSU）代替连续值语音编码器输出，通过语音适配器将其转换为 LLM 的标记嵌入空间。我们使用自监督语音编码器后跟 k-means 聚类生成 DSU。所提出的模型在来自可见 / 不可见领域的语音输入上表现出强大的性能，并具有口语问答指令遵循能力。我们还探索了来自自监督语音编码器的不同层以及 Mel 频率倒谱系数（MFCC）提取的各种类型的 DSU。我们的发现表明，在口语问答任务的指令调优中，ASR 任务和数据集并不关键。