Significant improvements in end-to-end speech translation (ST) have been
achieved through the application of multi-task learning. However, the extent to
which auxiliary tasks are highly consistent with the ST task, and how much this
approach truly helps, have not been thoroughly studied. In this paper, we
investigate the consistency between different tasks, considering different
times and modules. We find that the textual encoder primarily facilitates
cross-modal conversion, but the presence of noise in speech impedes the
consistency between text and speech representations. Furthermore, we propose an
improved multi-task learning (IMTL) approach for the ST task, which bridges the
modal gap by mitigating the difference in length and representation. We conduct
experiments on the MuST-C dataset. The results demonstrate that our method
attains state-of-the-art results. Moreover, when additional data is used, we
achieve the new SOTA result on MuST-C English to Spanish task with 20.8% of the
training time required by the current SOTA method.

通过应用多任务学习，已经在端到端语音翻译中取得了显著的改进。本文研究了不同任务之间的一致性，并提出了一种改进的多任务学习方法，通过缓解长度和表征的差异来弥合模态间的差距。实验证明我们的方法达到了最先进的结果。此外，当使用额外的数据时，我们在 MuST-C 英语到西班牙语任务上以当前最先进方法所需的 20.8% 的训练时间取得了新的最先进结果。