We present a novel integration of an instruction-tuned large language model
(LLM) and end-to-end automatic speech recognition (ASR). Modern LLMs can
perform a wide range of linguistic tasks within zero-shot learning when
provided with a precise instruction or a prompt to guide the text generation
process towards the desired task. We explore using this zero-shot capability of
LLMs to extract linguistic information that can contribute to improving ASR
performance. Specifically, we direct an LLM to correct grammatical errors in an
ASR hypothesis and harness the embedded linguistic knowledge to conduct
end-to-end ASR. The proposed model is built on the hybrid connectionist
temporal classification (CTC) and attention architecture, where an
instruction-tuned LLM (i.e., Llama2) is employed as a front-end of the decoder.
An ASR hypothesis, subject to correction, is obtained from the encoder via CTC
decoding, which is then fed into the LLM along with an instruction. The decoder
subsequently takes as input the LLM embeddings to perform sequence generation,
incorporating acoustic information from the encoder output. Experimental
results and analyses demonstrate that the proposed integration yields promising
performance improvements, and our approach largely benefits from LLM-based
rescoring.

我们提出了一种将指导训练的大型语言模型 (LLM) 与端到端自动语音识别 (ASR) 相结合的新方法。该方法利用 LLMs 的零 - shot 能力来提取语言信息，从而改善 ASR 性能，其中包括纠正 ASR 假设中的语法错误，并利用嵌入的语言知识进行端到端 ASR。实验证明，所提出的集成方法能够取得良好的性能改进，并且我们的方法在 LLM 基于重排序时得到了很大的好处。

利用调教的大型语言模型的零射击能力进行端到端语音识别

Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model  in End-to-End Speech Recognition

Automatic short answer grading (ASAG), which autonomously score student
answers according to reference answers, provides a cost-effective and
consistent approach to teaching professionals and can reduce their monotonous
and tedious grading workloads. However, ASAG is a very challenging task due to
two reasons: (1) student answers are made up of free text which requires a deep
semantic understanding; and (2) the questions are usually open-ended and across
many domains in K-12 scenarios. In this paper, we propose a generalized
end-to-end ASAG learning framework which aims to (1) autonomously extract
linguistic information from both student and reference answers; and (2)
accurately model the semantic relations between free-text student and reference
answers in open-ended domain. The proposed ASAG model is evaluated on a large
real-world K-12 dataset and can outperform the state-of-the-art baselines in
terms of various evaluation metrics.

本文提出了一种自动短答案评分学习框架，该框架针对 K-12 教育中的自由文本问题，旨在自动提取语言信息并在自由文本学生答案与参考答案之间准确建模语义关系，实验结果表明在各项评估指标上均高于最新算法模型。