Obtaining 3D object representations is important for creating photo-realistic
simulations and for collecting AR and VR assets. Neural fields have shown their
effectiveness in learning a continuous volumetric representation of a scene
from 2D images, but acquiring object representations from these models with
weak supervision remains an open challenge. In this paper we introduce LaTeRF,
a method for extracting an object of interest from a scene given 2D images of
the entire scene, known camera poses, a natural language description of the
object, and a set of point-labels of object and non-object points in the input
images. To faithfully extract the object from the scene, LaTeRF extends the
NeRF formulation with an additional `objectness' probability at each 3D point.
Additionally, we leverage the rich latent space of a pre-trained CLIP model
combined with our differentiable object renderer, to inpaint the occluded parts
of the object. We demonstrate high-fidelity object extraction on both synthetic
and real-world datasets and justify our design choices through an extensive
ablation study.

本研究提出了 LaTeRF 方法，通过引入 “对象性” 概率，扩展 NeRF 公式，结合自然语言描述、点标签等信息从场景中提取出感兴趣的物体，并结合预训练的 CLIP 模型和可微分对象渲染器来修复物体的遮挡部分。实验结果表明该方法在合成和真实数据集上均能实现高保真物体提取。