Heterogeneous data modalities can provide complementary cues for several
tasks, usually leading to more robust algorithms and better performance.
However, while training data can be accurately collected to include a variety
of sensory modalities, it is often the case that not all of them are available
in real life (testing) scenarios, where a model has to be deployed. This raises
the challenge of how to extract information from multimodal data in the
training stage, in a form that can be exploited at test time, considering
limitations such as noisy or missing modalities. This paper presents a new
approach in this direction for RGB-D vision tasks, developed within the
adversarial learning and privileged information frameworks. We consider the
practical case of learning representations from depth and RGB videos, while
relying only on RGB data at test time. We propose a new approach to train a
hallucination network that learns to distill depth information via adversarial
learning, resulting in a clean approach without several losses to balance or
hyperparameters. We report state-of-the-art results on object classification on
the NYUD dataset and video action recognition on the largest multimodal dataset
available for this task, the NTU RGB+D, as well as on the Northwestern-UCLA.

本研究针对深度学习中的异构数据模态问题提出了一种新的方法 —— 基于对抗学习与特权信息框架下的 RGB-D 视觉任务的幻象网络训练方法，旨在从深度和 RGB 视频中提取信息，使得在测试时，只需要使用 RGB 数据即可。这一方法在 NYUD 数据集的目标分类和最大多模态视频行为识别数据集（NTU RGB+D）以及美国西北大学加洛斯高等研究院的数据集上均取得了领先的结果。