This paper studies the problem of pre-training for small models, which is
essential for many mobile devices. Current state-of-the-art methods on this
problem transfer the representational knowledge of a large network (as a
Teacher) into a smaller model (as a Student) using self-supervised
distillation, improving the performance of the small model on downstream tasks.
However, existing approaches are insufficient in extracting the crucial
knowledge that is useful for discerning categories in downstream tasks during
the distillation process. In this paper, for the first time, we introduce
language guidance to the distillation process and propose a new method named
Language-Guided Distillation (LGD) system, which uses category names of the
target downstream task to help refine the knowledge transferred between the
teacher and student. To this end, we utilize a pre-trained text encoder to
extract semantic embeddings from language and construct a textual semantic
space called Textual Semantics Bank (TSB). Furthermore, we design a
Language-Guided Knowledge Aggregation (LGKA) module to construct the visual
semantic space, also named Visual Semantics Bank (VSB). The task-related
knowledge is transferred by driving a student encoder to mimic the similarity
score distribution inferred by a teacher over TSB and VSB. Compared with other
small models obtained by either ImageNet pre-training or self-supervised
distillation, experiment results show that the distilled lightweight model
using the proposed LGD method presents state-of-the-art performance and is
validated on various downstream tasks, including classification, detection, and
segmentation. We have made the code available at this https URL

本论文研究小型模型的预训练问题，提出了一种名为 Language-Guided Distillation (LGD) 的新方法，利用语言指导来帮助在大型网络和小型模型之间传递知识，实现优于现有方法的性能，验证了在分类、检测和分割等多个下游任务中的卓越表现。