Over-parameterized models, typically pretrained language models (LMs), have
shown an appealing expressive power due to their small learning bias. However,
the huge learning capacity of LMs can also lead to large learning variance. In
a pilot study, we find that, when faced with multiple domains, a critical
portion of parameters behave unexpectedly in a domain-specific manner while
others behave in a domain-general one. Motivated by this phenomenon, we for the
first time posit that domain-general parameters can underpin a domain-general
LM that can be derived from the original LM. To uncover the domain-general LM,
we propose to identify domain-general parameters by playing lottery tickets
(dubbed doge tickets). In order to intervene the lottery, we propose a
domain-general score, which depicts how domain-invariant a parameter is by
associating it with the variance. Comprehensive experiments are conducted on
the Amazon, Mnli and OntoNotes datasets. The results show that the doge tickets
obtains an improved out-of-domain generalization in comparison with a range of
competitive baselines. Analysis results further hint the existence of
domain-general parameters and the performance consistency of doge tickets.

本文为了寻求一种领域通用的语言模型，提出从预训练语言模型中鉴别领域无关的参数，实现利用 Doge tickets 方式来提高文本在不同领域上的推广性能优化。通过实验验证表明，将中性参数组成领域通用的语言模型可以得到更好性能。