Pre-trained language models (LMs), such as BERT (Devlin et al., 2018) and its
variants, have led to significant improvements on various NLP tasks in past
years. However, a theoretical framework for studying their relationships is
still missing. In this paper, we fill this gap by investigating the linear
dependency between pre-trained LMs. The linear dependency of LMs is defined
analogously to the linear dependency of vectors. We propose Language Model
Decomposition (LMD) to represent a LM using a linear combination of other LMs
as basis, and derive the closed-form solution. A goodness-of-fit metric for LMD
similar to the coefficient of determination is defined and used to measure the
linear dependency of a set of LMs. In experiments, we find that BERT and eleven
(11) BERT-like LMs are 91% linearly dependent. This observation suggests that
current state-of-the-art (SOTA) LMs are highly "correlated". To further advance
SOTA we need more diverse and novel LMs that are less dependent on existing
LMs.

本文提出了一种基于线性依赖性的语言模型分解（LMD）方法，研究了预训练语言模型（如 BERT）之间的线性依赖关系，并且发现现有预训练语言模型之间高度相关，为了进一步推进最先进技术，需要更加多元化和新颖的模型。