Recently, there has been a growing availability of pre-trained text models on
various model repositories. These models greatly reduce the cost of training
new models from scratch as they can be fine-tuned for specific tasks or trained
on large datasets. However, these datasets may not be publicly accessible due
to the privacy, security, or intellectual property issues. In this paper, we
aim to develop a lightweight student network that can learn from multiple
teacher models without accessing their original training data. Hence, we
investigate Data-Free Knowledge Amalgamation (DFKA), a knowledge-transfer task
that combines insights from multiple pre-trained teacher models and transfers
them effectively to a compact student network. To accomplish this, we propose
STRATANET, a modeling framework comprising: (a) a steerable data generator that
produces text data tailored to each teacher and (b) an amalgamation module that
implements a self-regulative strategy using confidence estimates from the
teachers' different layers to selectively integrate their knowledge and train a
versatile student. We evaluate our method on three benchmark text
classification datasets with varying labels or domains. Empirically, we
demonstrate that the student model learned using our STRATANET outperforms
several baselines significantly under data-driven and data-free constraints.

该论文提出了一种轻量级的学生网络，通过多个预训练模型进行知识传递，以在无法访问原始训练数据的情况下训练一个高效的学生模型。使用提出的 STRATANET 建模框架，该方法在不同的文本分类数据集上得到了显著优于其他基准方法的结果，同时满足数据驱动和无数据约束。