This work connects language model adaptation with concepts of machine
learning theory. We consider a training setup with a large out-of-domain set
and a small in-domain set. We derive how the benefit of training a model on
either set depends on the size of the sets and the distance between their
underlying distributions. We analyze how out-of-domain pre-training before
in-domain fine-tuning achieves better generalization than either solution
independently. Finally, we present how adaptation techniques based on data
selection, such as importance sampling, intelligent data selection and
influence functions, can be presented in a common framework which highlights
their similarity and also their subtle differences.

探讨了语言模型适应与机器学习理论的关系，研究了大型领域外训练集和小型领域内训练集之间的训练方法的优劣，提出了领域外预训练加上领域内微调比单独应用更为通用，并提出了基于数据选择的适应技术的公共框架。

神经语言模型的域自适应权衡

The Trade-offs of Domain Adaptation for Neural Language Models

With the rapid increase in the volume of dialogue data from daily life, there
is a growing demand for dialogue summarization. Unfortunately, training a large
summarization model is generally infeasible due to the inadequacy of dialogue
data with annotated summaries. Most existing works for low-resource dialogue
summarization directly pretrain models in other domains, e.g., the news domain,
but they generally neglect the huge difference between dialogues and
conventional articles. To bridge the gap between out-of-domain pretraining and
in-domain fine-tuning, in this work, we propose a multi-source pretraining
paradigm to better leverage the external summary data. Specifically, we exploit
large-scale in-domain non-summary data to separately pretrain the dialogue
encoder and the summary decoder. The combined encoder-decoder model is then
pretrained on the out-of-domain summary data using adversarial critics, aiming
to facilitate domain-agnostic summarization. The experimental results on two
public datasets show that with only limited training data, our approach
achieves competitive performance and generalizes well in different dialogue
scenarios.

本文提出了一种多源预训练范式以更好地利用外部摘要数据，通过有限的训练数据实现对话摘要的有竞争力的性能和不同对话场景中的泛化能力。