Neural fields excel in computer vision and robotics due to their ability to
understand the 3D visual world such as inferring semantics, geometry, and
dynamics. Given the capabilities of neural fields in densely representing a 3D
scene from 2D images, we ask the question: Can we scale their self-supervised
pretraining, specifically using masked autoencoders, to generate effective 3D
representations from posed RGB images. Owing to the astounding success of
extending transformers to novel data modalities, we employ standard 3D Vision
Transformers to suit the unique formulation of NeRFs. We leverage NeRF's
volumetric grid as a dense input to the transformer, contrasting it with other
3D representations such as pointclouds where the information density can be
uneven, and the representation is irregular. Due to the difficulty of applying
masked autoencoders to an implicit representation, such as NeRF, we opt for
extracting an explicit representation that canonicalizes scenes across domains
by employing the camera trajectory for sampling. Our goal is made possible by
masking random patches from NeRF's radiance and density grid and employing a
standard 3D Swin Transformer to reconstruct the masked patches. In doing so,
the model can learn the semantic and spatial structure of complete scenes. We
pretrain this representation at scale on our proposed curated posed-RGB data,
totaling over 1.6 million images. Once pretrained, the encoder is used for
effective 3D transfer learning. Our novel self-supervised pretraining for
NeRFs, NeRF-MAE, scales remarkably well and improves performance on various
challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining,
NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF
scene understanding baselines on Front3D and ScanNet datasets with an absolute
performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.

神经场在计算机视觉和机器人技术中表现出色，能够理解三维视觉世界，例如推断语义、几何和动态。我们使用神经场进行自监督预训练，尤其是使用遮盖的自动编码器从 RGB 图像生成有效的三维表示，然后将标准的三维视觉 Transformer 应用于 NeRF 来进行特定形式的训练。我们利用 NeRF 的体积网格作为 Transformer 的密集输入，与像点云等其他三维表示进行对比。通过将相机轨迹用于采样，我们提取了可以规范化不同域中场景的显式表示。我们的目标是通过遮盖 NeRF 的辐射和密度网格的随机补丁并使用标准的三维 Swin Transformer 重建这些补丁，从而使模型能够学习完整场景的语义和空间结构。我们在自己提出的 posed-RGB 数据上规模化地对这个表示进行预训练，总共涉及超过 160 万幅图像。预训练后，编码器可以用于有效的三维迁移学习。我们提出的 NeRF-MAE 自监督预训练方法在各种具有挑战性的三维任务上表现出色，利用未标记的 posed 2D 数据进行预训练，相较于自监督三维预训练和 NeRF 场景理解基线，在 Front3D 和 ScanNet 数据集上的三维物体检测的 AP50 和 AP25 绝对性能提升分别超过 20％和 8％。