We tackle the task of learning dynamic 3D semantic radiance fields given a
single monocular video as input. Our learned semantic radiance field captures
per-point semantics as well as color and geometric properties for a dynamic 3D
scene, enabling the generation of novel views and their corresponding
semantics. This enables the segmentation and tracking of a diverse set of 3D
semantic entities, specified using a simple and intuitive interface that
includes a user click or a text prompt. To this end, we present DGD, a unified
3D representation for both the appearance and semantics of a dynamic 3D scene,
building upon the recently proposed dynamic 3D Gaussians representation. Our
representation is optimized over time with both color and semantic information.
Key to our method is the joint optimization of the appearance and semantic
attributes, which jointly affect the geometric properties of the scene. We
evaluate our approach in its ability to enable dense semantic 3D object
tracking and demonstrate high-quality results that are fast to render, for a
diverse set of scenes. Our project webpage is available on
this https URL

学习使用单目视频作为输入来生成动态三维语义辐射场，其中学习到的语义辐射场捕捉每个点的语义以及颜色和几何属性，以便生成新的视图和相应的语义。通过联合优化外观和语义属性，使得该方法能够实现密集的语义三维物体跟踪，并展示了渲染快速而质量高的多样化场景结果。