Emerging neural radiance fields (NeRF) are a promising scene representation
for computer graphics, enabling high-quality 3D reconstruction and novel view
synthesis from image observations. However, editing a scene represented by a
NeRF is challenging, as the underlying connectionist representations such as
MLPs or voxel grids are not object-centric or compositional. In particular, it
has been difficult to selectively edit specific regions or objects. In this
work, we tackle the problem of semantic scene decomposition of NeRFs to enable
query-based local editing of the represented 3D scenes. We propose to distill
the knowledge of off-the-shelf, self-supervised 2D image feature extractors
such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the
radiance field. Given a user-specified query of various modalities such as
text, an image patch, or a point-and-click selection, 3D feature fields
semantically decompose 3D space without the need for re-training and enable us
to semantically select and edit regions in the radiance field. Our experiments
validate that the distilled feature fields (DFFs) can transfer recent progress
in 2D vision and language foundation models to 3D scene representations,
enabling convincing 3D segmentation and selective editing of emerging neural
graphics representations.

本文提出通过蒸馏出自监督的 2D 图像特征提取器的知识，优化一种针对 NeRF 的 3D 特征场，从而实现语义场景分解以及针对局部区域的查询式编辑，从而实现结构编辑的目的。通过实验证明，3D 特征场可以将最近在 2D 视觉和语言基础模型中取得的进展应用于 3D 场景表示，从而实现令人信服的 3D 分割和选择性编辑。