In recent years, the field of 3D self-supervised learning has witnessed
significant progress, resulting in the emergence of Multi-Modality Masked
AutoEncoders (MAE) methods that leverage both 2D images and 3D point clouds for
pre-training. However, a notable limitation of these approaches is that they do
not fully utilize the multi-view attributes inherent in 3D point clouds, which
is crucial for a deeper understanding of 3D structures. Building upon this
insight, we introduce a novel approach employing a 3D to multi-view masked
autoencoder to fully harness the multi-modal attributes of 3D point clouds. To
be specific, our method uses the encoded tokens from 3D masked point clouds to
generate original point clouds and multi-view depth images across various
poses. This approach not only enriches the model's comprehension of geometric
structures but also leverages the inherent multi-modal properties of point
clouds. Our experiments illustrate the effectiveness of the proposed method for
different tasks and under different settings. Remarkably, our method
outperforms state-of-the-art counterparts by a large margin in a variety of
downstream tasks, including 3D object classification, few-shot learning, part
segmentation, and 3D object detection. Code will be available at:
this https URL

通过使用 3D 到多视图掩码自编码器，充分利用 3D 点云的多模属性，提出了一种全新的方法，这不仅丰富了模型对几何结构的理解，还利用了点云的固有多模性能，有效地改进了各种任务，包括 3D 对象分类、少样本学习、部分分割和 3D 对象检测。