With the rapid development of computer vision, Vision Transformers (ViTs)
offer the tantalizing prospect of unified information processing across visual
and textual domains. But due to the lack of inherent inductive biases in ViTs,
they require enormous amount of data for training. To make their applications
practical, we introduce an innovative ensemble-based distillation approach
distilling inductive bias from complementary lightweight teacher models. Prior
systems relied solely on convolution-based teaching. However, this method
incorporates an ensemble of light teachers with different architectural
tendencies, such as convolution and involution, to instruct the student
transformer jointly. Because of these unique inductive biases, instructors can
accumulate a wide range of knowledge, even from readily identifiable stored
datasets, which leads to enhanced student performance. Our proposed framework
also involves precomputing and storing logits in advance, essentially the
unnormalized predictions of the model. This optimization can accelerate the
distillation process by eliminating the need for repeated forward passes during
knowledge distillation, significantly reducing the computational burden and
enhancing efficiency.

通过集成式蒸馏方法从轻量级教师模型中蒸馏归纳偏差，我们介绍了一种创新的蒸馏方法，以支持视觉转换和文本领域之间的统一信息处理，提高了学生性能，并减轻了计算负担和提高了效率。