Neural fields (NeRF) have emerged as a promising approach for representing
continuous 3D scenes. Nevertheless, the lack of semantic encoding in NeRFs
poses a significant challenge for scene decomposition. To address this
challenge, we present a single model, Multi-Modal Decomposition NeRF
(${M^2D}$NeRF), that is capable of both text-based and visual patch-based
edits. Specifically, we use multi-modal feature distillation to integrate
teacher features from pretrained visual and language models into 3D semantic
feature volumes, thereby facilitating consistent 3D editing. To enforce
consistency between the visual and language features in our 3D feature volumes,
we introduce a multi-modal similarity constraint. We also introduce a
patch-based joint contrastive loss that helps to encourage object-regions to
coalesce in the 3D feature space, resulting in more precise boundaries.
Experiments on various real-world scenes show superior performance in 3D scene
decomposition tasks compared to prior NeRF-based methods.

多模态分解 NeRF（${M^2D}$NeRF）是一种能够进行文本和视觉补丁编辑的单一模型，通过使用多模态特征蒸馏来整合来自预训练视觉和语言模型的教师特征到三维语义特征体积，从而促进一致的三维编辑。实验证明，在三维场景分解任务中，相较于以前的基于 NeRF 的方法具有更优异的性能。