We present an analysis tool based on joint matrix factorization for comparing
latent representations of multilingual and monolingual models. An alternative
to probing, this tool allows us to analyze multiple sets of representations in
a joint manner. Using this tool, we study to what extent and how
morphosyntactic features are reflected in the representations learned by
multilingual pre-trained models. We conduct a large-scale empirical study of
over 33 languages and 17 morphosyntactic categories. Our findings demonstrate
variations in the encoding of morphosyntactic information across upper and
lower layers, with category-specific differences influenced by language
properties. Hierarchical clustering of the factorization outputs yields a tree
structure that is related to phylogenetic trees manually crafted by linguists.
Moreover, we find the factorization outputs exhibit strong associations with
performance observed across different cross-lingual tasks. We release our code
to facilitate future research.

我们提出了一种基于联合矩阵分解的分析工具，用于比较多语言和单语言模型的潜在表示。我们使用这个工具，研究了多语言预训练模型学习的表示中在何种程度上和如何反映了形态句法特征。通过对 33 种语言和 17 种形态句法类别进行大规模实证研究，我们发现不同的形态句法信息在不同层次的编码中存在差异，这些差异受到语言属性的影响。对分解输出进行层次聚类得到了一棵树状结构，与由语言学家手工制作的系统发生关系的系统发生树。此外，我们发现分解输出与不同的跨语言任务中观察到的性能之间存在强关联。我们发布了我们的代码以促进未来的研究。

多语言表示的联合矩阵因式分析

A Joint Matrix Factorization Analysis of Multilingual Representations

Linguistic analysis of language models is one of the ways to explain and
describe their reasoning, weaknesses, and limitations. In the probing part of
the model interpretability research, studies concern individual languages as
well as individual linguistic structures. The question arises: are the detected
regularities linguistically coherent, or on the contrary, do they dissonate at
the typological scale? Moreover, the majority of studies address the inherent
set of languages and linguistic structures, leaving the actual typological
diversity knowledge out of scope. In this paper, we present and apply the
GUI-assisted framework allowing us to easily probe a massive number of
languages for all the morphosyntactic features present in the Universal
Dependencies data. We show that reflecting the anglo-centric trend in NLP over
the past years, most of the regularities revealed in the mBERT model are
typical for the western-European languages. Our framework can be integrated
with the existing probing toolboxes, model cards, and leaderboards, allowing
practitioners to use and share their standard probing methods to interpret
multilingual models. Thus we propose a toolkit to systematize the multilingual
flaws in multilingual models, providing a reproducible experimental setup for
104 languages and 80 morphosyntactic features.
this https URL

本文提出并应用一种 GUI 辅助框架，使我们能够轻松地探寻普遍依赖数据中所有形态句法特征所出现的大量语言。我们发现，反映了过去几年自然语言处理的西方中心趋势，mBERT 模型中揭示的大多数规律对于西欧语言而言是典型的。因此，我们提出了一个工具包，以系统化解决多语种模型中存在的缺陷，为 104 种语言和 80 种形态句法特征提供可重复的实验设置。