Cross-domain text classification aims to adapt models to a target domain that
lacks labeled data. It leverages or reuses rich labeled data from the different
but related source domain(s) and unlabeled data from the target domain. To this
end, previous work focuses on either extracting domain-invariant features or
task-agnostic features, ignoring domain-aware features that may be present in
the target domain and could be useful for the downstream task. In this paper,
we propose a two-stage framework for cross-domain text classification. In the
first stage, we finetune the model with mask language modeling (MLM) and
labeled data from the source domain. In the second stage, we further fine-tune
the model with self-supervised distillation (SSD) and unlabeled data from the
target domain. We evaluate its performance on a public cross-domain text
classification benchmark and the experiment results show that our method
achieves new state-of-the-art results for both single-source domain adaptations
(94.17% $\uparrow$1.03%) and multi-source domain adaptations (95.09%
$\uparrow$1.34%).

本文提出了一种二阶段跨域文本分类的框架，使用掩蔽语言模型和标记数据从源域进行模型微调，然后使用自监督蒸馏和目标域的无标注数据进一步微调模型，实验结果表明我们的方法在单源域（94.17％ ↑1.03％）和多源域（95.09％ ↑1.34％）适应都达到了最新的最先进水平。

自监督蒸馏的两阶段跨领域文本分类框架

A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain  Text Classification

Pre-trained transformers has declared its success in many NLP tasks. One
thread of work focuses on training bi-encoder models (i.e., dense retrievers)
to effectively encode sentences or passages into single-vector dense vectors
for efficient approximate nearest neighbor (ANN) search. However, recent work
has demonstrated that transformers pre-trained with mask language modeling
(MLM) are not capable of effectively aggregating text information into a single
dense vector due to task-mismatch between pre-training and fine-tuning.
Therefore, computationally expensive techniques have been adopted to train
dense retrievers, such as large batch size, knowledge distillation or post
pre-training. In this work, we present a simple approach to effectively
aggregate textual representation from the pre-trained transformer into a dense
vector. Extensive experiments show that our approach improves the robustness of
the single-vector approach under both in-domain and zero-shot evaluations
without any computationally expensive training techniques. Our work
demonstrates that MLM pre-trained transformers can be used to effectively
encode text information into a single-vector for dense retrieval. Code are
available at: this https URL

本研究采用一种简单的方法，有效地将预训练转换器中的文本表示聚合成稠密向量，从而提高了在不同领域和零样本评估下单向量方法的鲁棒性，进而表明 MLM 预训练变形金刚可以用于将文本信息编码成单个向量以进行密集检索。