The integration of pre-trained text-based large language models (LLM) with
speech input has enabled instruction-following capabilities for diverse speech
tasks. This integration requires the use of a speech encoder, a speech adapter,
and an LLM, trained on diverse tasks. We propose the use of discrete speech
units (DSU), rather than continuous-valued speech encoder outputs, that are
converted to the LLM token embedding space using the speech adapter. We
generate DSU using a self-supervised speech encoder followed by k-means
clustering. The proposed model shows robust performance on speech inputs from
seen/unseen domains and instruction-following capability in spoken question
answering. We also explore various types of DSU extracted from different layers
of the self-supervised speech encoder, as well as Mel frequency Cepstral
Coefficients (MFCC). Our findings suggest that the ASR task and datasets are
not crucial in instruction-tuning for spoken question answering tasks.

通过将预训练的基于文本的大型语言模型（LLM）与语音输入集成，我们提出使用离散语音单元（DSU）代替连续值语音编码器输出，通过语音适配器将其转换为 LLM 的标记嵌入空间。我们使用自监督语音编码器后跟 k-means 聚类生成 DSU。所提出的模型在来自可见 / 不可见领域的语音输入上表现出强大的性能，并具有口语问答指令遵循能力。我们还探索了来自自监督语音编码器的不同层以及 Mel 频率倒谱系数（MFCC）提取的各种类型的 DSU。我们的发现表明，在口语问答任务的指令调优中，ASR 任务和数据集并不关键。

离散 SLU：自我監督的离散语音单元用于口语理解的大型语言模型

DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech  Units for Spoken Language Understanding

ASR remains unsatisfactory in scenarios where the speaking style diverges
from that used to train ASR systems, resulting in erroneous transcripts. To
address this, ASR Error Correction (AEC), a post-ASR processing approach, is
required. In this work, we tackle an understudied issue: the Low-Resource
Out-of-Domain (LROOD) problem, by investigating crossmodal AEC on very limited
downstream data with 1-best hypothesis transcription. We explore pre-training
and fine-tuning strategies and uncover an ASR domain discrepancy phenomenon,
shedding light on appropriate training schemes for LROOD data. Moreover, we
propose the incorporation of discrete speech units to align with and enhance
the word embeddings for improving AEC quality. Results from multiple corpora
and several evaluation metrics demonstrate the feasibility and efficacy of our
proposed AEC approach on LROOD data, as well as its generalizability and
superiority on large-scale data. Finally, a study on speech emotion recognition
confirms that our model produces ASR error-robust transcripts suitable for
downstream applications.

ASR Error Correction (AEC) is investigated to tackle the Low-Resource Out-of-Domain (LROOD) problem, covering pre-training, fine-tuning, and incorporating discrete speech units to improve AEC quality. The proposed AEC approach demonstrates feasibility, efficacy, generalizability, superiority, and suitability for downstream applications.

跨模态离散语音单元的 ASR 错误修正

Crossmodal ASR Error Correction with Discrete Speech Units

Using Self-Supervised Learning (SSL) as model initialization is now common to
obtain strong results in Speech Translation (ST). However, they also impose a
large memory footprint, hindering on-device deployment. In this paper, we
leverage the SSL models by pretraining smaller models on their Discrete Speech
Units (DSU). We pretrain encoder-decoder models on 1) Filterbank-to-DSU and 2)
DSU-to-Translation data, and take the encoder from 1) and the decoder from 2)
to initialise a new model, finetuning this on limited speech-translation data.
The final model becomes compact by using the DSU pretraining to distil the
knowledge of the SSL model. Our method has several benefits over using DSU as
model inputs, such as shorter inference pipeline and robustness over (DSU)
tokenization. In contrast to ASR pretraining, it does not require transcripts,
making it applicable to low-resource settings. Evaluation on CoVoST-2 X-En
shows that our method is >$0.5$ BLEU better than a ST model that directly
finetune the SSL model, given only half the model size, and on a par with ASR
pretraining.

使用自监督学习作为模型初始化在语音翻译中取得较好结果已经很常见，但也对设备上的部署造成了大量的内存开销。本文通过在离散语音单元上对自监督学习模型进行预训练，从而在有限的语音翻译数据上微调初始化的新模型，并利用离散语音单元预训练来凝结自监督学习模型的知识，从而使得最终模型更加紧凑。我们的方法相比于将离散语音单元用作模型输入，具有短推理流程和对（离散语音单元）分词具有鲁棒性等多个优点。与自动语音识别的预训练相比，它不需要转录，因此适用于资源有限的环境。在 CoVoST-2 X-En 数据集上的评估结果显示，我们的方法比直接微调自监督学习模型的语音翻译模型具有更高的 BLEU 得分（提升 0.5），且模型大小仅为其一半，并且与自动语音识别的预训练方法相媲美。

通过离散语音单元预训练的紧凑语音翻译模型

Compact Speech Translation Models via Discrete Speech Units Pretraining

Speech signals, typically sampled at rates in the tens of thousands per
second, contain redundancies, evoking inefficiencies in sequence modeling.
High-dimensional speech features such as spectrograms are often used as the
input for the subsequent model. However, they can still be redundant. Recent
investigations proposed the use of discrete speech units derived from
self-supervised learning representations, which significantly compresses the
size of speech data. Applying various methods, such as de-duplication and
subword modeling, can further compress the speech sequence length. Hence,
training time is significantly reduced while retaining notable performance. In
this study, we undertake a comprehensive and systematic exploration into the
application of discrete units within end-to-end speech processing models.
Experiments on 12 automatic speech recognition, 3 speech translation, and 1
spoken language understanding corpora demonstrate that discrete units achieve
reasonably good results in almost all the settings. We intend to release our
configurations and trained models to foster future research efforts.

通过对离散单元在端到端语音处理模型中应用的系统和全面的探索，实验证明离散单元在几乎所有设置中都可以取得相当好的结果。