Cross-modal transfer is helpful to enhance modality-specific discriminative
power for scene recognition. To this end, this paper presents a unified
framework to integrate the tasks of cross-modal translation and
modality-specific recognition, termed as Translate-to-Recognize Network
(TRecgNet). Specifically, both translation and recognition tasks share the same
encoder network, which allows to explicitly regularize the training of
recognition task with the help of translation, and thus improve its final
generalization ability. For translation task, we place a decoder module on top
of the encoder network and it is optimized with a new layer-wise semantic loss,
while for recognition task, we use a linear classifier based on the feature
embedding from encoder and its training is guided by the standard cross-entropy
loss. In addition, our TRecgNet allows to exploit large numbers of unlabeled
RGB-D data to train the translation task and thus improve the representation
power of encoder network. Empirically, we verify that this new semi-supervised
setting is able to further enhance the performance of recognition network. We
perform experiments on two RGB-D scene recognition benchmarks: NYU Depth v2 and
SUN RGB-D, demonstrating that TRecgNet achieves superior performance to the
existing state-of-the-art methods, especially for recognition solely based on a
single modality.

该研究提出了一种新的模态间转移识别网络 TRecgNet，利用编码器网络共享任务，采用半监督方式利用未标记的数据进行训练，进一步提高了识别网络的准确性。实验证明，TRecgNet 在 RGB-D 场景识别方面的表现具有优越性。