Understanding 3D scenes is a crucial challenge in computer vision research
with applications spanning multiple domains. Recent advancements in distilling
2D vision-language foundation models into neural fields, like NeRF and 3DGS,
enables open-vocabulary segmentation of 3D scenes from 2D multi-view images
without the need for precise 3D annotations. While effective, however, the
per-pixel distillation of high-dimensional CLIP features introduces ambiguity
and necessitates complex regularization strategies, adding inefficiencies
during training. This paper presents MaskField, which enables fast and
efficient 3D open-vocabulary segmentation with neural fields under weak
supervision. Unlike previous methods, MaskField distills masks rather than
dense high-dimensional CLIP features. MaskFields employ neural fields as binary
mask generators and supervise them with masks generated by SAM and classified
by coarse CLIP features. MaskField overcomes the ambiguous object boundaries by
naturally introducing SAM segmented object shapes without extra regularization
during training. By circumventing the direct handling of high-dimensional CLIP
features during training, MaskField is particularly compatible with explicit
scene representations like 3DGS. Our extensive experiments show that MaskField
not only surpasses prior state-of-the-art methods but also achieves remarkably
fast convergence, outperforming previous methods with just 5 minutes of
training. We hope that MaskField will inspire further exploration into how
neural fields can be trained to comprehend 3D scenes from 2D models.

通过底层模型的信号转换，快速高效地进行弱监督下的 3D 场景分割。