Multilingual machine translation has proven immensely useful for low-resource
and zero-shot language pairs. However, language pairs in multilingual models
sometimes see worse performance than in bilingual models, especially when
translating in a one-to-many setting. To understand why, we examine the
geometric differences in the representations from bilingual models versus those
from one-to-many multilingual models. Specifically, we evaluate the isotropy of
the representations, to measure how well they utilize the dimensions in their
underlying vector space. Using the same evaluation data in both models, we find
that multilingual model decoder representations tend to be less isotropic than
bilingual model decoder representations. Additionally, we show that much of the
anisotropy in multilingual decoder representations can be attributed to
modeling language-specific information, therefore limiting remaining
representational capacity.

探讨了多语言翻译中双语模型和一对多多语言模型的在表示上的几何差异，发现多语言解码器的表示比双语解码器的表示具有更低的等向性，限制了剩余的表达能力，并分析了这种差异的原因。

探索多语和双语翻译模型之间的表征差异

Exploring Representational Disparities Between Multilingual and  Bilingual Translation Models

Representation learning of source code is essential for applying machine
learning to software engineering tasks. Learning code representation from a
multilingual source code dataset has been shown to be more effective than
learning from single-language datasets separately, since more training data
from multilingual dataset improves the model's ability to extract
language-agnostic information from source code. However, existing multilingual
training overlooks the language-specific information which is crucial for
modeling source code across different programming languages, while only
focusing on learning a unified model with shared parameters among different
languages for language-agnostic information modeling. To address this problem,
we propose MetaTPTrans, a meta learning approach for multilingual code
representation learning. MetaTPTrans generates different parameters for the
feature extractor according to the specific programming language type of the
input code snippet, enabling the model to learn both language-agnostic and
language-specific information with dynamic parameters in the feature extractor.
We conduct experiments on the code summarization and code completion tasks to
verify the effectiveness of our approach. The results demonstrate the
superiority of our approach with significant improvements on state-of-the-art
baselines.

本研究旨在解决现有的多语言编程数据集忽略编程语言特定信息的问题。为此，作者提出了 MetaTPTrans 方法，该方法可以在特征提取器中为不同编程语言类型的输入代码片段生成动态参数，从而实现对编程语言通用信息和特定信息的同时学习。实验证明该方法在代码概述和代码完成任务中优于现有的基准模型。

MetaTPTrans: 多语言代码表示学习的元学习方法

MetaTPTrans: A Meta Learning Approach for Multilingual Code Representation Learning

Multilingual pretrained language models (MPLMs) exhibit multilinguality and
are well suited for transfer across languages. Most MPLMs are trained in an
unsupervised fashion and the relationship between their objective and
multilinguality is unclear. More specifically, the question whether MPLM
representations are language-agnostic or they simply interleave well with
learned task prediction heads arises. In this work, we locate language-specific
information in MPLMs and identify its dimensionality and the layers where this
information occurs. We show that language-specific information is scattered
across many dimensions, which can be projected into a linear subspace. Our
study contributes to a better understanding of MPLM representations, going
beyond treating them as unanalyzable blobs of information.

本研究旨在了解多语言预训练语言模型（MPLMs）在不受监督学习方式下与其多语性之间的关系，指出 MPLMs 的表示是否是语言无关的或与学习任务预测头相互交错，同时，我们定位了 MPLM 中的语言特定信息，并确定其维度和发生这些信息的层次，以及该信息散布在许多维度中，并可投影为线性子空间。