To reduce a model size but retain performance, we often rely on knowledge
distillation (KD) which transfers knowledge from a large "teacher" model to a
smaller "student" model. However, KD on multimodal datasets such as
vision-language tasks is relatively unexplored, and digesting multimodal
information is challenging since different modalities present different types
of information. In this paper, we perform a large-scale empirical study to
investigate the importance and effects of each modality in knowledge
distillation. Furthermore, we introduce a multimodal knowledge distillation
framework, modality-specific distillation (MSD), to transfer knowledge from a
teacher on multimodal tasks by learning the teacher's behavior within each
modality. The idea aims at mimicking a teacher's modality-specific predictions
by introducing auxiliary loss terms for each modality. Furthermore, because
each modality has different saliency for predictions, we define saliency scores
for each modality and investigate saliency-based weighting schemes for the
auxiliary losses. We further study a weight learning approach to learn the
optimal weights on these loss terms. In our empirical analysis, we examine the
saliency of each modality in KD, demonstrate the effectiveness of the weighting
scheme in MSD, and show that it achieves better performance than KD on four
multimodal datasets.

研究对知识蒸馏在多模态数据集上的应用，提出了一种多模态知识蒸馏框架 MSD，其中引入辅助损失项和基于显著度得分的加权方案以及基于权重学习的方法，实验结果表明 MSD 比 KD 在四个多模态数据集上表现更好。