Current RGB-D scene recognition approaches often train two standalone
backbones for RGB and depth modalities with the same Places or ImageNet
pre-training. However, the pre-trained depth network is still biased by
RGB-based models which may result in a suboptimal solution. In this paper, we
present a single-model self-supervised hybrid pre-training framework for RGB
and depth modalities, termed as CoMAE. Our CoMAE presents a curriculum learning
strategy to unify the two popular self-supervised representation learning
algorithms: contrastive learning and masked image modeling. Specifically, we
first build a patch-level alignment task to pre-train a single encoder shared
by two modalities via cross-modal contrastive learning. Then, the pre-trained
contrastive encoder is passed to a multi-modal masked autoencoder to capture
the finer context features from a generative perspective. In addition, our
single-model design without requirement of fusion module is very flexible and
robust to generalize to unimodal scenario in both training and testing phases.
Extensive experiments on SUN RGB-D and NYUDv2 datasets demonstrate the
effectiveness of our CoMAE for RGB and depth representation learning. In
addition, our experiment results reveal that CoMAE is a data-efficient
representation learner. Although we only use the small-scale and unlabeled
training set for pre-training, our CoMAE pre-trained models are still
competitive to the state-of-the-art methods with extra large-scale and
supervised RGB dataset pre-training. Code will be released at
this https URL

本文提出了一种名为 CoMAE 的单模型自监督混合预训练框架，通过交叉模态对比学习和遮蔽图像建模，采用课程学习策略来统一两种流行的自监督表示学习算法，并设计了一个修补程序级别对齐任务来预训练一种单一的编码器，共享两个模态。CoMAE 在 SUN RGB-D 和 NYUDv2 数据集上的实验表明，在仅使用小规模和无标签的训练集进行预训练的情况下，其预训练模型仍然具有与额外的大规模和监督 RGB 数据集预训练的最新方法相竞争的效果。

CoMAE：基于单模型的小规模 RGB-D 数据混合预训练

CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets

Deep convolutional networks (CNN) can achieve impressive results on RGB scene
recognition thanks to large datasets such as Places. In contrast, RGB-D scene
recognition is still underdeveloped in comparison, due to two limitations of
RGB-D data we address in this paper. The first limitation is the lack of depth
data for training deep learning models. Rather than fine tuning or transferring
RGB-specific features, we address this limitation by proposing an architecture
and a two-step training approach that directly learns effective depth-specific
features using weak supervision via patches. The resulting RGB-D model also
benefits from more complementary multimodal features. Another limitation is the
short range of depth sensors (typically 0.5m to 5.5m), resulting in depth
images not capturing distant objects in the scenes that RGB images can. We show
that this limitation can be addressed by using RGB-D videos, where more
comprehensive depth information is accumulated as the camera travels across the
scene. Focusing on this scenario, we introduce the ISIA RGB-D video dataset to
evaluate RGB-D scene recognition with videos. Our video recognition
architecture combines convolutional and recurrent neural networks (RNNs) that
are trained in three steps with increasingly complex data to learn effective
features (i.e. patches, frames and sequences). Our approach obtains
state-of-the-art performances on RGB-D image (NYUD2 and SUN RGB-D) and video
(ISIA RGB-D) scene recognition.

该研究介绍一种基于深度学习的 RGB-D 场景识别方法，提出了一种深度学习模型训练的两步骤方法和一种新的 RGB-D 视频数据集，实现了在 RGB-D 图像 (NYUD2 和 SUN RGB-D) 和视频 (ISIA RGB-D) 场景识别方面的最优表现。

学习有效的 RGB-D 场景识别表示

Learning Effective RGB-D Representations for Scene Recognition

Scene recognition with RGB images has been extensively studied and has
reached very remarkable recognition levels, thanks to convolutional neural
networks (CNN) and large scene datasets. In contrast, current RGB-D scene data
is much more limited, so often leverages RGB large datasets, by transferring
pretrained RGB CNN models and fine-tuning with the target RGB-D dataset.
However, we show that this approach has the limitation of hardly reaching
bottom layers, which is key to learn modality-specific features. In contrast,
we focus on the bottom layers, and propose an alternative strategy to learn
depth features combining local weakly supervised training from patches followed
by global fine tuning with images. This strategy is capable of learning very
discriminative depth-specific features with limited depth images, without
resorting to Places-CNN. In addition we propose a modified CNN architecture to
further match the complexity of the model and the amount of data available. For
RGB-D scene recognition, depth and RGB features are combined by projecting them
in a common space and further leaning a multilayer classifier, which is jointly
optimized in an end-to-end network. Our framework achieves state-of-the-art
accuracy on NYU2 and SUN RGB-D in both depth only and combined RGB-D data.

本研究利用卷积神经网络和弱监督方法，提出了一种能够在有限深度图像条件下学习具有较强区分性的深度特征，从而实现了 RGB-D 场景识别的最新成果，达到了最好的表现。