Most 3D neural networks are trained from scratch owing to the lack of
large-scale labeled 3D datasets. In this paper, we present a novel 3D
pretraining method by leveraging 2D networks learned from rich 2D datasets. We
propose the contrastive pixel-to-point knowledge transfer to effectively
utilize the 2D information by mapping the pixel-level and point-level features
into the same embedding space. Due to the heterogeneous nature between 2D and
3D networks, we introduce the back-projection function to align the features
between 2D and 3D to make the transfer possible. Additionally, we devise an
upsampling feature projection layer to increase the spatial resolution of
high-level 2D feature maps, which enables learning fine-grained 3D
representations. With a pretrained 2D network, the proposed pretraining process
requires no additional 2D or 3D labeled data, further alleviating the expensive
3D data annotation cost. To the best of our knowledge, we are the first to
exploit existing 2D trained weights to pretrain 3D deep neural networks. Our
intensive experiments show that the 3D models pretrained with 2D knowledge
boost the performances of 3D networks across various real-world 3D downstream
tasks.

本文提出了一种使用 2D 网络预训练 3D 深度神经网络的新方法，包括像素到点知识转移、反向投影函数、上采样特征投影层等，成功提升了各种真实世界 3D 网络下游任务的模型性能。