End-to-end speech-to-text translation (E2E-ST) is becoming increasingly
popular due to the potential of its less error propagation, lower latency, and
fewer parameters. Given the triplet training corpus $\langle speech,
transcription, translation\rangle$, the conventional high-quality E2E-ST system
leverages the $\langle speech, transcription\rangle$ pair to pre-train the
model and then utilizes the $\langle speech, translation\rangle$ pair to
optimize it further. However, this process only involves two-tuple data at each
stage, and this loose coupling fails to fully exploit the association between
triplet data. In this paper, we attempt to model the joint probability of
transcription and translation based on the speech input to directly leverage
such triplet data. Based on that, we propose a novel regularization method for
model training to improve the agreement of dual-path decomposition within
triplet data, which should be equal in theory. To achieve this goal, we
introduce two Kullback-Leibler divergence regularization terms into the model
training objective to reduce the mismatch between output probabilities of
dual-path. Then the well-trained model can be naturally transformed as the
E2E-ST models by the pre-defined early stop tag. Experiments on the MuST-C
benchmark demonstrate that our proposed approach significantly outperforms
state-of-the-art E2E-ST baselines on all 8 language pairs, while achieving
better performance in the automatic speech recognition task. Our code is
open-sourced at this https URL

本文介绍了一种新颖的正则化方法用于 E2E-ST 系统的训练，并在 MuST-C 基准测试中显著优于现有技术，该正则化方法通过对 KL 散度进行约束，以更好地利用音频文字翻译三元组数据。