In this paper, we employ Singular Value Canonical Correlation Analysis
(SVCCA) to analyze representations learnt in a multilingual end-to-end speech
translation model trained over 22 languages. SVCCA enables us to estimate
representational similarity across languages and layers, enhancing our
understanding of the functionality of multilingual speech translation and its
potential connection to multilingual neural machine translation. The
multilingual speech translation model is trained on the CoVoST 2 dataset in all
possible directions, and we utilize LASER to extract parallel bitext data for
SVCCA analysis. We derive three major findings from our analysis: (I)
Linguistic similarity loses its efficacy in multilingual speech translation
when the training data for a specific language is limited. (II) Enhanced
encoder representations and well-aligned audio-text data significantly improve
translation quality, surpassing the bilingual counterparts when the training
data is not compromised. (III) The encoder representations of multilingual
speech translation demonstrate superior performance in predicting phonetic
features in linguistic typology prediction. With these findings, we propose
that releasing the constraint of limited data for low-resource languages and
subsequently combining them with linguistically related high-resource languages
could offer a more effective approach for multilingual end-to-end speech
translation.

利用奇异值典型相关分析 (SVCCA)，我们分析了一个在 22 种语言上进行训练的多语言端到端语音翻译模型的表示学习。我们通过 SVCCA 估计了不同语言和层之间的表示相似性，增进了我们对多语言语音翻译功能及其与多语言神经机器翻译的潜在联系的理解。通过对分析的结论，我们提出了解除对低资源语言的数据限制，将其与语言相关的高资源语言相结合的方法，以提供更有效的多语言端到端语音翻译。

探索对多语言端到端语音翻译的深入理解

Towards a Deep Understanding of Multilingual End-to-End Speech  Translation

Multilingual Neural Machine Translation (NMT) models have yielded large
empirical success in transfer learning settings. However, these black-box
representations are poorly understood, and their mode of transfer remains
elusive. In this work, we attempt to understand massively multilingual NMT
representations (with 103 languages) using Singular Value Canonical Correlation
Analysis (SVCCA), a representation similarity framework that allows us to
compare representations across different languages, layers and models. Our
analysis validates several empirical results and long-standing intuitions, and
unveils new observations regarding how representations evolve in a multilingual
translation model. We draw three major conclusions from our analysis, with
implications on cross-lingual transfer learning: (i) Encoder representations of
different languages cluster based on linguistic similarity, (ii)
Representations of a source language learned by the encoder are dependent on
the target language, and vice-versa, and (iii) Representations of high resource
and/or linguistically similar languages are more robust when fine-tuning on an
arbitrary language pair, which is critical to determining how much
cross-lingual transfer can be expected in a zero or few-shot setting. We
further connect our findings with existing empirical observations in
multilingual NMT and transfer learning.

本研究使用 Singular Value Canonical Correlation Analysis（SVCCA）分析了包含 103 种语言的 NMT 模型，发现不同语言的编码器表示会基于语言相似性聚集，源语言和目标语言的表示相互依赖，并且高资源和 / 或语言相似性更强的语言在任意语言对上进行微调时更为稳健，这些结论对于跨语言转移学习非常重要，并进一步联系到现有的实证观察。