The integration of pre-trained text-based large language models (LLM) with
speech input has enabled instruction-following capabilities for diverse speech
tasks. This integration requires the use of a speech encoder, a speech adapter,
and an LLM, trained on diverse tasks. We propose the use of discrete speech
units (DSU), rather than continuous-valued speech encoder outputs, that are
converted to the LLM token embedding space using the speech adapter. We
generate DSU using a self-supervised speech encoder followed by k-means
clustering. The proposed model shows robust performance on speech inputs from
seen/unseen domains and instruction-following capability in spoken question
answering. We also explore various types of DSU extracted from different layers
of the self-supervised speech encoder, as well as Mel frequency Cepstral
Coefficients (MFCC). Our findings suggest that the ASR task and datasets are
not crucial in instruction-tuning for spoken question answering tasks.

通过将预训练的基于文本的大型语言模型（LLM）与语音输入集成，我们提出使用离散语音单元（DSU）代替连续值语音编码器输出，通过语音适配器将其转换为 LLM 的标记嵌入空间。我们使用自监督语音编码器后跟 k-means 聚类生成 DSU。所提出的模型在来自可见 / 不可见领域的语音输入上表现出强大的性能，并具有口语问答指令遵循能力。我们还探索了来自自监督语音编码器的不同层以及 Mel 频率倒谱系数（MFCC）提取的各种类型的 DSU。我们的发现表明，在口语问答任务的指令调优中，ASR 任务和数据集并不关键。

离散 SLU：自我監督的离散语音单元用于口语理解的大型语言模型

DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech  Units for Spoken Language Understanding

Recent studies find existing self-supervised speech encoders contain
primarily acoustic rather than semantic information. As a result, pipelined
supervised automatic speech recognition (ASR) to large language model (LLM)
systems achieve state-of-the-art results on semantic spoken language tasks by
utilizing rich semantic representations from the LLM. These systems come at the
cost of labeled audio transcriptions, which is expensive and time-consuming to
obtain. We propose a task-agnostic unsupervised way of incorporating semantic
information from LLMs into self-supervised speech encoders without labeled
audio transcriptions. By introducing semantics, we improve existing speech
encoder spoken language understanding performance by over 10\% on intent
classification, with modest gains in named entity resolution and slot filling,
and spoken question answering FF1 score by over 2\%. Our unsupervised approach
achieves similar performance as supervised methods trained on over 100 hours of
labeled audio transcripts, demonstrating the feasibility of unsupervised
semantic augmentations to existing speech encoders.

本文提出了一种无监督的任务无关方法，将大型语言模型中的语义信息融合进入自监督语音编码器，提高了其在意图分类，名称实体识别及槽填充等方面的性能，使其达到了有监督方法同等的效果，为已有的语音编码器的无监督实现提供了可行性证明。