Large Language Models have demonstrated unparalleled effectiveness in various
NLP tasks, and integrating LLMs with automatic speech recognition is becoming a
mainstream paradigm. Building upon this momentum, our research delves into an
indepth examination of this paradigm on a large opensource Chinese dataset.
Specifically, our research aims to evaluate the impact of various
configurations of speech encoders, LLMs, and projector modules in the context
of the speech foundation encoderLLM ASR paradigm. Furthermore, we introduce a
threestage training approach, expressly developed to enhance the model's
ability to align auditory and textual information. The implementation of this
approach, alongside the strategic integration of ASR components, enabled us to
achieve the SOTA performance on the AISHELL1, TestNet, and TestMeeting test
sets. Our analysis presents an empirical foundation for future research in
LLMbased ASR systems and offers insights into optimizing performance using
Chinese datasets. We will publicly release all scripts used for data
preparation, training, inference, and scoring, as well as pretrained models and
training logs to promote reproducible research.

基于大型语言模型的自动语音识别研究，探索了多种配置下的语音编码器、语言模型和投影模块对 ASR 性能的影响，采用三阶段训练方法实现了在中文数据集上的最佳表现，为未来 LLM 基于 ASR 系统的研究提供了实证基础和性能优化的见解。

揭示基于 LLM 的中文开源数据集上的 ASR 潜力

Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets

The impressive capability and versatility of large language models (LLMs)
have aroused increasing attention in automatic speech recognition (ASR), with
several pioneering studies attempting to build integrated ASR models by
connecting a speech encoder with an LLM. This paper presents a comparative
study of three commonly used structures as connectors, including fully
connected layers, multi-head cross-attention, and Q-Former. Speech encoders
from the Whisper model series as well as LLMs from the Vicuna model series with
different model sizes were studied. Experiments were performed on the commonly
used LibriSpeech, Common Voice, and GigaSpeech datasets, where the LLMs with
Q-Formers demonstrated consistent and considerable word error rate (WER)
reductions over LLMs with other connector structures. Q-Former-based LLMs can
generalise well to out-of-domain datasets, where 12% relative WER reductions
over the Whisper baseline ASR model were achieved on the Eval2000 test set
without using any in-domain training data from Switchboard. Moreover, a novel
segment-level Q-Former is proposed to enable LLMs to recognise speech segments
with a duration exceeding the limitation of the encoders, which results in 17%
relative WER reductions over other connector structures on 90-second-long
speech data.

该论文通过比较研究了三种常用的连接结构，包括全连接层、多头交叉注意力和 Q-Former，并对 Whisper 系列的语音编码器和 Vicuna 系列的大语言模型进行了实验，结果表明基于 Q-Former 的大语言模型相比其他连接结构在 LibriSpeech、Common Voice 和 GigaSpeech 数据集上均取得了一致且显著的词错误率降低。此外，提出了一种新颖的片段级 Q-Former，使大语言模型能够识别超过编码器限制的持续时间的语音片段，在 90 秒长的语音数据上相比其他连接结构取得了 17% 的词错误率降低。

连接语音编码器和大型语言模型用于 ASR

Connecting Speech Encoder and Large Language Model for ASR

This paper reimagines some aspects of speech processing using speech
encoders, specifically about extracting entities directly from speech, with no
intermediate textual representation. In human-computer conversations,
extracting entities such as names, postal addresses and email addresses from
speech is a challenging task. In this paper, we study the impact of fine-tuning
pre-trained speech encoders on extracting spoken entities in human-readable
form directly from speech without the need for text transcription. We
illustrate that such a direct approach optimizes the encoder to transcribe only
the entity relevant portions of speech, ignoring the superfluous portions such
as carrier phrases and spellings of entities. In the context of dialogs from an
enterprise virtual agent, we demonstrate that the 1-step approach outperforms
the typical 2-step cascade of first generating lexical transcriptions followed
by text-based entity extraction for identifying spoken entities.

本文探讨了利用语音编码器从语音中直接提取实体的方法，研究表明这种直接的方法优化了编码器以仅转录语音中与实体相关的部分，忽略不必要的部分，并且在企业虚拟代理的对话中，优于传统的两步法。