In this paper, we employ Singular Value Canonical Correlation Analysis
(SVCCA) to analyze representations learnt in a multilingual end-to-end speech
translation model trained over 22 languages. SVCCA enables us to estimate
representational similarity across languages and layers, enhancing our
understanding of the functionality of multilingual speech translation and its
potential connection to multilingual neural machine translation. The
multilingual speech translation model is trained on the CoVoST 2 dataset in all
possible directions, and we utilize LASER to extract parallel bitext data for
SVCCA analysis. We derive three major findings from our analysis: (I)
Linguistic similarity loses its efficacy in multilingual speech translation
when the training data for a specific language is limited. (II) Enhanced
encoder representations and well-aligned audio-text data significantly improve
translation quality, surpassing the bilingual counterparts when the training
data is not compromised. (III) The encoder representations of multilingual
speech translation demonstrate superior performance in predicting phonetic
features in linguistic typology prediction. With these findings, we propose
that releasing the constraint of limited data for low-resource languages and
subsequently combining them with linguistically related high-resource languages
could offer a more effective approach for multilingual end-to-end speech
translation.

利用奇异值典型相关分析 (SVCCA)，我们分析了一个在 22 种语言上进行训练的多语言端到端语音翻译模型的表示学习。我们通过 SVCCA 估计了不同语言和层之间的表示相似性，增进了我们对多语言语音翻译功能及其与多语言神经机器翻译的潜在联系的理解。通过对分析的结论，我们提出了解除对低资源语言的数据限制，将其与语言相关的高资源语言相结合的方法，以提供更有效的多语言端到端语音翻译。

探索对多语言端到端语音翻译的深入理解

Towards a Deep Understanding of Multilingual End-to-End Speech  Translation

In this paper, we propose a simple yet effective framework for multilingual
end-to-end speech translation (ST), in which speech utterances in source
languages are directly translated to the desired target languages with a
universal sequence-to-sequence architecture. While multilingual models have
shown to be useful for automatic speech recognition (ASR) and machine
translation (MT), this is the first time they are applied to the end-to-end ST
problem. We show the effectiveness of multilingual end-to-end ST in two
scenarios: one-to-many and many-to-many translations with publicly available
data. We experimentally confirm that multilingual end-to-end ST models
significantly outperform bilingual ones in both scenarios. The generalization
of multilingual training is also evaluated in a transfer learning scenario to a
very low-resource language pair. All of our codes and the database are publicly
available to encourage further research in this emergent multilingual ST topic.

本文提出了一种简单且有效的多语言端到端语音翻译框架，并证明了其在自动语音识别、机器翻译、一对多翻译以及多对多翻译中的有效性以及相对于双语端到端语音翻译的优势。