Multimodal models have gained significant success in recent years. Standard
multimodal approaches often assume unchanged modalities from training stage to
inference stage. In practice, however, many scenarios fail to satisfy such
assumptions with missing modalities during inference, leading to limitations on
where multimodal models can be applied. While existing methods mitigate the
problem through reconstructing the missing modalities, it increases unnecessary
computational cost, which could be just as critical, especially for large,
deployed systems. To solve the problem from both sides, we propose a novel
guidance network that promotes knowledge sharing during training, taking
advantage of the multimodal representations to train better single-modality
models for inference. Real-life experiment in violence detection shows that our
proposed framework trains single-modality models that significantly outperform
its traditionally trained counterparts while maintaining the same inference
cost.

提出了一种通过引导网络在训练阶段促进知识共享，利用多模式表示训练用于推理的更好的单模式模型，以解决存在缺失模式的多模态模型应用受限和过高计算成本的问题。通过真实生活中的暴力检测实验证明，所提出的框架训练的单模式模型明显优于传统训练的模型，并且推理成本相同。