This paper investigates a novel approach to end-to-end speech translation (ST) based on aligning frozen pre-trained automatic speech recognition (ASR) and machine translation (MT) models via a small connector module (Q-Former, our Subsampler-Transformer Encoder). This connector bridges the gap between the speech and text modalities, transforming ASR encoder embeddings into the latent representation space of the MT encoder while being the only part of the system optimized during training. Experiments are conducted on the How2 English-Portuguese dataset as we investigate the alignment approach in a small-scale scenario focusing on ST. While keeping the size of the connector module constant and small in comparison ( < 5% of the size of the larger aligned models), increasing the size and capability of the foundation ASR and MT models universally improves translation results. We also find that the connectors can serve as domain adapters for the foundation MT models, significantly improving translation performance in the aligned ST setting. We conclude that this approach represents a viable and scalable approach to training end-to-end ST systems.

本研究解决了口语翻译（ST）中的一个关键问题，即如何有效对齐预训练的自动语音识别（ASR）和机器翻译（MT）模型。作者提出了一种小型连接模块（Q-Former）来实现这一对齐，并通过实验表明，增强的基础ASR和MT模型显著提升了翻译效果，且这一方法具有可扩展性和实用性。

对齐预训练模型用于口语翻译