Heterogeneous data modalities can provide complementary cues for several
tasks, usually leading to more robust algorithms and better performance.
However, while training data can be accurately collected to include a variety
of sensory modalities, it is often the case that not all of them are available
in real life (testing) scenarios, where a model has to be deployed. This raises
the challenge of how to extract information from multimodal data in the
training stage, in a form that can be exploited at test time, considering
limitations such as noisy or missing modalities. This paper presents a new
approach in this direction for RGB-D vision tasks, developed within the
adversarial learning and privileged information frameworks. We consider the
practical case of learning representations from depth and RGB videos, while
relying only on RGB data at test time. We propose a new approach to train a
hallucination network that learns to distill depth information via adversarial
learning, resulting in a clean approach without several losses to balance or
hyperparameters. We report state-of-the-art results on object classification on
the NYUD dataset and video action recognition on the largest multimodal dataset
available for this task, the NTU RGB+D, as well as on the Northwestern-UCLA.

本研究针对深度学习中的异构数据模态问题提出了一种新的方法 —— 基于对抗学习与特权信息框架下的 RGB-D 视觉任务的幻象网络训练方法，旨在从深度和 RGB 视频中提取信息，使得在测试时，只需要使用 RGB 数据即可。这一方法在 NYUD 数据集的目标分类和最大多模态视频行为识别数据集（NTU RGB+D）以及美国西北大学加洛斯高等研究院的数据集上均取得了领先的结果。

利用对抗性判别式模态蒸馏技术学习含特殊信息的数据

Learning with privileged information via adversarial discriminative  modality distillation

Diverse input data modalities can provide complementary cues for several
tasks, usually leading to more robust algorithms and better performance.
However, while a (training) dataset could be accurately designed to include a
variety of sensory inputs, it is often the case that not all modalities could
be available in real life (testing) scenarios, where a model has to be
deployed. This raises the challenge of how to learn robust representations
leveraging multimodal data in the training stage, while considering limitations
at test time, such as noisy or missing modalities.
This paper presents a new approach for multimodal video action recognition,
developed within the unified frameworks of distillation and privileged
information, named generalized distillation. Particularly, we consider the case
of learning representations from depth and RGB videos, while relying on RGB
data only at test time. We propose a new approach to train an hallucination
network that learns to distill depth features through multiplicative
connections of spatiotemporal representations, leveraging soft labels and hard
labels, as well as distance between feature maps. We report state-of-the-art
results on video action classification on the largest multimodal dataset
available for this task, the NTU RGB+D. Code available at
this https URL .

本研究提出了一种新的多模态视频动作识别方法，利用 RGB 数据以外的深度数据进行训练，在测试时利用 RGB 数据进行分类，通过精简网络来进行深度特征的提取，经过实验得到了目前最佳的分类结果。