In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained
language models for downstream tasks. We introduce UDALM, a fine-tuning
procedure, using a mixed classification and Masked Language Model loss, that
can adapt to the target domain distribution in a robust and sample efficient
manner. Our experiments show that performance of models trained with the mixed
loss scales with the amount of available target data and the mixed loss can be
effectively used as a stopping criterion during UDA training. Furthermore, we
discuss the relationship between A-distance and the target error and explore
some limitations of the Domain Adversarial Training approach. Our method is
evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset,
yielding $91.74\%$ accuracy, which is an $1.11\%$ absolute improvement over the
state-of-the-art.

本文中，我们探索了使用预训练的语言模型进行下游任务的无监督领域适应（UDA）。我们提出了 UDALM，一种微调过程，使用混合分类和蒙面语言模型损失，可以以强健和样本高效的方式适应目标域分布。我们的实验表明，使用混合损失训练的模型的性能与可用目标数据的数量成比例，并且可以有效地用作 UDA 训练中的停止准则。此外，我们还讨论了 A 距离和目标误差之间的关系，并探索了域对抗训练方法的一些局限性。我们的方法在 Amazon 评论情感数据集的 12 个领域对上进行评估，得到了 91.74％的准确率，比最先进技术提高了 1.11％的绝对值。