Teacher-student knowledge distillation is a popular technique for compressing
today's prevailing large language models into manageable sizes that fit
low-latency downstream applications. Both the teacher and the choice of
transfer set used for distillation are crucial ingredients in creating a high
quality student. Yet, the generic corpora used to pretrain the teacher and the
corpora associated with the downstream target domain are often significantly
different, which raises a natural question: should the student be distilled
over the generic corpora, so as to learn from high-quality teacher predictions,
or over the downstream task corpora to align with finetuning? Our study
investigates this trade-off using Domain Classification (DC) and Intent
Classification/Named Entity Recognition (ICNER) as downstream tasks. We distill
several multilingual students from a larger multilingual LM with varying
proportions of generic and task-specific datasets, and report their performance
after finetuning on DC and ICNER. We observe significant improvements across
tasks and test sets when only task-specific corpora is used. We also report on
how the impact of adding task-specific data to the transfer set correlates with
the similarity between generic and task-specific data. Our results clearly
indicate that, while distillation from a generic LM benefits downstream tasks,
students learn better using target domain data even if it comes at the price of
noisier teacher predictions. In other words, target domain data still trumps
teacher knowledge.

本文研究了采用不同数据集训练的应用领域教师和泛化语言模型辅导学生学习的关系，并发现即使存在较高噪声，使用应用领域数据进行辅导仍然比使用泛化数据表现更好。

知识蒸馏传递集及其对下游 NLU 任务的影响

Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks

Under the domain shift, cross-domain few-shot object detection aims to adapt
object detectors in the target domain with a few annotated target data. There
exists two significant challenges: (1) Highly insufficient target domain data;
(2) Potential over-adaptation and misleading caused by inappropriately
amplified target samples without any restriction. To address these challenges,
we propose an adaptive method consisting of two parts. First, we propose an
adaptive optimization strategy to select augmented data similar to target
samples rather than blindly increasing the amount. Specifically, we filter the
augmented candidates which significantly deviate from the target feature
distribution in the very beginning. Second, to further relieve the data
limitation, we propose the multi-level domain-aware data augmentation to
increase the diversity and rationality of augmented data, which exploits the
cross-image foreground-background mixture. Experiments show that the proposed
method achieves state-of-the-art performance on multiple benchmarks.

本文提出一种新方法以应对跨域少样本目标检测面临的数据不足及误导等问题，该方法由自适应优化策略与多层次领域感知数据增强两部分构成，最终在多个基准测试中取得最优表现。

AcroFOD：一种跨领域少样本对象检测的自适应方法

AcroFOD: An Adaptive Method for Cross-domain Few-shot Object Detection

Self-supervised learning of speech representations has been a very active
research area but most work is focused on a single domain such as read audio
books for which there exist large quantities of labeled and unlabeled data. In
this paper, we explore more general setups where the domain of the unlabeled
data for pre-training data differs from the domain of the labeled data for
fine-tuning, which in turn may differ from the test data domain. Our
experiments show that using target domain data during pre-training leads to
large performance improvements across a variety of setups. On a large-scale
competitive setup, we show that pre-training on unlabeled in-domain data
reduces the gap between models trained on in-domain and out-of-domain labeled
data by 66%-73%. This has obvious practical implications since it is much
easier to obtain unlabeled target domain data than labeled data. Moreover, we
find that pre-training on multiple domains improves generalization performance
on domains not seen during training. Code and models will be made available at
this https URL

本文探讨了语音表示的自监督学习，其中更富含挑战的是那些预训练数据的领域与微调和测试数据的领域不同的情形，试验结果表明在预训练过程中加入目标领域的数据可以显著提高性能。

稳健的 wav2vec 2.0：自监督预训练中的领域漂移分析

Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised  Pre-Training

In sequence labeling, previous domain adaptation methods focus on the
adaptation from the source domain to the entire target domain without
considering the diversity of individual target domain samples, which may lead
to negative transfer results for certain samples. Besides, an important
characteristic of sequence labeling tasks is that different elements within a
given sample may also have diverse domain relevance, which requires further
consideration. To take the multi-level domain relevance discrepancy into
account, in this paper, we propose a fine-grained knowledge fusion model with
the domain relevance modeling scheme to control the balance between learning
from the target domain data and learning from the source domain model.
Experiments on three sequence labeling tasks show that our fine-grained
knowledge fusion model outperforms strong baselines and other state-of-the-art
sequence labeling domain adaptation methods.

本研究提出一种细粒度知识融合模型，通过领域关联建模来控制从目标域数据学习和从源域模型学习之间的平衡，以解决序列标注中多级领域相关性差异的问题，实验结果表明，这种模型在三种序列标注任务中表现优异。