In contrast to numerous NLP and 2D computer vision foundational models, the
learning of a robust and highly generalized 3D foundational model poses
considerably greater challenges. This is primarily due to the inherent data
variability and the diversity of downstream tasks. In this paper, we introduce
a comprehensive 3D pre-training framework designed to facilitate the
acquisition of efficient 3D representations, thereby establishing a pathway to
3D foundational models. Motivated by the fact that informative 3D features
should be able to encode rich geometry and appearance cues that can be utilized
to render realistic images, we propose a novel universal paradigm to learn
point cloud representations by differentiable neural rendering, serving as a
bridge between 3D and 2D worlds. We train a point cloud encoder within a
devised volumetric neural renderer by comparing the rendered images with the
real images. Notably, our approach demonstrates the seamless integration of the
learned 3D encoder into diverse downstream tasks. These tasks encompass not
only high-level challenges such as 3D detection and segmentation but also
low-level objectives like 3D reconstruction and image synthesis, spanning both
indoor and outdoor scenarios. Besides, we also illustrate the capability of
pre-training a 2D backbone using the proposed universal methodology, surpassing
conventional pre-training methods by a large margin. For the first time,
\sexyname achieves state-of-the-art performance on 11 indoor and outdoor
benchmarks. The consistent improvements in various settings imply the
effectiveness of the proposed method. Code and models will be made available at
this https URL

通过可微分神经渲染，我们提出了一个新的通用方法来学习点云表示，实现了 3D 和 2D 之间的无缝集成，并在室内和室外场景中的多个任务中实现了卓越的性能和持续改进。