Learning based on multimodal data has attracted increasing interest recently.
While a variety of sensory modalities can be collected for training, not all of
them are always available in development scenarios, which raises the challenge
to infer with incomplete modality. To address this issue, this paper presents a
one-stage modality distillation framework that unifies the privileged knowledge
transfer and modality information fusion into a single optimization procedure
via multi-task learning. Compared with the conventional modality distillation
that performs them independently, this helps to capture the valuable
representation that can assist the final model inference directly.
Specifically, we propose the joint adaptation network for the modality transfer
task to preserve the privileged information. This addresses the representation
heterogeneity caused by input discrepancy via the joint distribution
adaptation. Then, we introduce the cross translation network for the modality
fusion task to aggregate the restored and available modality features. It
leverages the parameters-sharing strategy to capture the cross-modal cues
explicitly. Extensive experiments on RGB-D classification and segmentation
tasks demonstrate the proposed multimodal inheritance framework can overcome
the problem of incomplete modality input in various scenes and achieve
state-of-the-art performance.

基于多模态数据的学习近年来引起了越来越多的关注。尽管可以采集各种感官模态进行训练，但并不总是能在开发场景中获得所有的模态，这给基于不完整模态的推断带来了挑战。为了解决这个问题，本文提出了一个一阶段模态蒸馏框架，通过多任务学习将特权知识传递和模态信息融合统一为一个优化过程。与独立执行它们的传统模态蒸馏相比，这有助于直接捕捉有助于最终模型推断的有价值表征。本文具体介绍了模态转换任务的联合适应网络，以保留特权信息。通过联合分布适应解决由输入差异引起的表示异质性。然后，我们介绍了模态融合任务的交叉翻译网络，以聚合恢复和可用的模态特征。它利用参数共享策略明确捕捉跨模态线索。在 RGB-D 分类和分割任务上的大量实验证明，提出的多模态继承框架可以克服各种场景中不完整模态输入的问题，并实现最先进的性能。