With the emergence of large language models (LLMs), multimodal models based
on LLMs have demonstrated significant potential. Models such as LLaSM, X-LLM,
and SpeechGPT exhibit an impressive ability to comprehend and generate human
instructions. However, their performance often falters when faced with complex
tasks like end-to-end speech translation (E2E-ST), a cross-language and
cross-modal translation task. In comparison to single-modal models, multimodal
models lag behind in these scenarios. This paper introduces LST, a Large
multimodal model designed to excel at the E2E-ST task. LST consists of a speech
frontend, an adapter, and a LLM backend. The training of LST consists of two
stages: (1) Modality adjustment, where the adapter is tuned to align speech
representation with text embedding space, and (2) Downstream task fine-tuning,
where both the adapter and LLM model are trained to optimize performance on the
E2EST task. Experimental results on the MuST-C speech translation benchmark
demonstrate that LST-13B achieves BLEU scores of 30.39/41.55/35.33 on
En-De/En-Fr/En-Es language pairs, surpassing previous models and establishing a
new state-of-the-art. Additionally, we conduct an in-depth analysis of
single-modal model selection and the impact of training strategies, which lays
the foundation for future research. We will open up our code and models after
review.

本研究提出了 LST，一个大型多模型，旨在优化端到端语音翻译任务的性能并创造新的技术水平。在 MuST-C 语音翻译基准测试中，LST-13B 取得了 30.39/41.55/35.33 的 BLEU 分数，超过了之前的模型并树立了新的最新技术水平。此外，还进行了单模型选择和训练策略对结果的影响的深入分析，为未来的研究奠定了基础。