Adapting pre-trained language models (PrLMs) (e.g., BERT) to new domains has
gained much attention recently. Instead of fine-tuning PrLMs as done in most
previous work, we investigate how to adapt the features of PrLMs to new domains
without fine-tuning. We explore unsupervised domain adaptation (UDA) in this
paper. With the features from PrLMs, we adapt the models trained with labeled
data from the source domain to the unlabeled target domain. Self-training is
widely used for UDA which predicts pseudo labels on the target domain data for
training. However, the predicted pseudo labels inevitably include noise, which
will negatively affect training a robust model. To improve the robustness of
self-training, in this paper we present class-aware feature self-distillation
(CFd) to learn discriminative features from PrLMs, in which PrLM features are
self-distilled into a feature adaptation module and the features from the same
class are more tightly clustered. We further extend CFd to a cross-language
setting, in which language discrepancy is studied. Experiments on two
monolingual and multilingual Amazon review datasets show that CFd can
consistently improve the performance of self-training in cross-domain and
cross-language settings.

本研究中，我们探究了将预训练语言模型（PrLMs）特征适应新域的方法，而非像以往一样对其进行精细调整。我们提出了面向无监督域适应 (UDA) 的类感知特征自蒸馏（CFd）算法，能够有效地提高跨领域和跨语言设置下的模型稳健性。