Crossmodal knowledge distillation (KD) extends traditional knowledge
distillation to the area of multimodal learning and demonstrates great success
in various applications. To achieve knowledge transfer across modalities, a
pretrained network from one modality is adopted as the teacher to provide
supervision signals to a student network learning from another modality. In
contrast to the empirical success reported in prior works, the working
mechanism of crossmodal KD remains a mystery. In this paper, we present a
thorough understanding of crossmodal KD. We begin with two case studies and
demonstrate that KD is not a universal cure in crossmodal knowledge transfer.
We then present the modality Venn diagram to understand modality relationships
and the modality focusing hypothesis revealing the decisive factor in the
efficacy of crossmodal KD. Experimental results on 6 multimodal datasets help
justify our hypothesis, diagnose failure cases, and point directions to improve
crossmodal knowledge transfer in the future.

本文主要探讨了跨模态知识迁移中知识蒸馏的机制，并提出使用模态 Venn 图和模态聚焦假设来提高跨模态知识迁移的有效性。通过对 6 个多模态数据集的实验结果，证明了作者的假设，并指向未来的改进方向。

感知模态聚焦假设：跨模态知识蒸馏的理解

The Modality Focusing Hypothesis: Towards Understanding Crossmodal  Knowledge Distillation

As an important and challenging problem in computer vision, zero-shot
learning (ZSL) aims at automatically recognizing the instances from unseen
object classes without training data. To address this problem, ZSL is usually
carried out in the following two aspects: 1) capturing the domain distribution
connections between seen classes data and unseen classes data; and 2) modeling
the semantic interactions between the image feature space and the label
embedding space. Motivated by these observations, we propose a bidirectional
mapping based semantic relationship modeling scheme that seeks for crossmodal
knowledge transfer by simultaneously projecting the image features and label
embeddings into a common latent space. Namely, we have a bidirectional
connection relationship that takes place from the image feature space to the
latent space as well as from the label embedding space to the latent space. To
deal with the domain shift problem, we further present a transductive learning
approach that formulates the class prediction problem in an iterative refining
process, where the object classification capacity is progressively reinforced
through bootstrapping-based model updating over highly reliable instances.
Experimental results on three benchmark datasets (AwA, CUB and SUN) demonstrate
the effectiveness of the proposed approach against the state-of-the-art
approaches.

本文提出一种基于双向映射的语义关系建模方案，实现跨模态的知识转移并解决领域转化问题，通过迭代更新使得分类器不断加强预测能力，实验结果在三个基准数据集上显著优于现有方法。