Masked Autoencoders learn strong visual representations and achieve
state-of-the-art results in several independent modalities, yet very few works
have addressed their capabilities in multi-modality settings. In this work, we
focus on point cloud and RGB image data, two modalities that are often
presented together in the real world, and explore their meaningful
interactions. To improve upon the cross-modal synergy in existing works, we
propose PiMAE, a self-supervised pre-training framework that promotes 3D and 2D
interaction through three aspects. Specifically, we first notice the importance
of masking strategies between the two sources and utilize a projection module
to complementarily align the mask and visible tokens of the two modalities.
Then, we utilize a well-crafted two-branch MAE pipeline with a novel shared
decoder to promote cross-modality interaction in the mask tokens. Finally, we
design a unique cross-modal reconstruction module to enhance representation
learning for both modalities. Through extensive experiments performed on
large-scale RGB-D scene understanding benchmarks (SUN RGB-D and ScannetV2), we
discover it is nontrivial to interactively learn point-image features, where we
greatly improve multiple 3D detectors, 2D detectors, and few-shot classifiers
by 2.9%, 6.7%, and 2.4%, respectively. Code is available at
this https URL.

本文提出了 PiMAE，一个自我监督的预训练框架，旨在通过三个方面促进 3D 和 2D 交互，以提高现有作品中的交叉模态协同作用。在 SUN RGB-D 和 ScannetV2 上进行广泛的实验后，我们发现交互式学习点 - 图像特征是不平凡的，并通过 2.9％，6.7％和 2.4％显着提高了多个 3D 检测器，2D 检测器和少样本分类器。