Given the great success of large language models (LLMs) across various tasks,
in this paper, we introduce LLM-ST, a novel and effective speech translation
model constructed upon a pre-trained LLM. By integrating the large language
model (LLM) with a speech encoder and employing multi-task instruction tuning,
LLM-ST can produce accurate timestamped transcriptions and translations, even
from long audio inputs. Furthermore, our findings indicate that the
implementation of Chain-of-Thought (CoT) prompting can yield advantages in the
context of LLM-ST. Through rigorous experimentation on English and Chinese
datasets, we showcase the exceptional performance of LLM-ST, establishing a new
benchmark in the field of speech translation. Demo:
this https URL

我们介绍了一种新颖有效的语音翻译模型 LLM-ST，它基于预训练的大型语言模型（LLM），通过将 LLM 与语音编码器集成并使用多任务指导调优，LLM-ST 能够准确地生成带时间戳的转录和翻译，甚至可以处理长时间的音频输入。此外，我们的研究结果表明，在 LLM-ST 的环境下使用 CoT 提示可以带来优势。通过对英语和中文数据集的严格实验，我们展示了 LLM-ST 的卓越性能，在语音翻译领域树立了新的基准。演示：this https URL

大语言模型的语音翻译：一个工业实践

Speech Translation with Large Language Models: An Industrial Practice

A cascaded speech translation model relies on discrete and non-differentiable
transcription, which provides a supervision signal from the source side and
helps the transformation between source speech and target text. Such modeling
suffers from error propagation between ASR and MT models. Direct speech
translation is an alternative method to avoid error propagation; however, its
performance is often behind the cascade system. To use an intermediate
representation and preserve the end-to-end trainability, previous studies have
proposed using two-stage models by passing the hidden vectors of the recognizer
into the decoder of the MT model and ignoring the MT encoder. This work
explores the feasibility of collapsing the entire cascade components into a
single end-to-end trainable model by optimizing all parameters of ASR and MT
models jointly without ignoring any learned parameters. It is a tightly
integrated method that passes renormalized source word posterior distributions
as a soft decision instead of one-hot vectors and enables backpropagation.
Therefore, it provides both transcriptions and translations and achieves strong
consistency between them. Our experiments on four tasks with different data
scenarios show that the model outperforms cascade models up to 1.8% in BLEU and
2.0% in TER and is superior compared to direct models.

本文提出了一种端到端的可训练语音翻译模型，通过优化所有 ASR 和 MT 模型的参数而避免级联模型中的误差传播，并且通过使用人工神经网络实现 backpropagation 传递误差以实现同时提供转录和翻译，对于四个不同数据场景的实验表明，该模型在 BLEU 和 TER 方面均优于传统级联模型和直接模型。