Extensions of Neural Radiance Fields (NeRFs) to model dynamic scenes have
enabled their near photo-realistic, free-viewpoint rendering. Although these
methods have shown some potential in creating immersive experiences, two
drawbacks limit their ubiquity: (i) a significant reduction in reconstruction
quality when the computing budget is limited, and (ii) a lack of semantic
understanding of the underlying scenes. To address these issues, we introduce
Gear-NeRF, which leverages semantic information from powerful image
segmentation models. Our approach presents a principled way for learning a
spatio-temporal (4D) semantic embedding, based on which we introduce the
concept of gears to allow for stratified modeling of dynamic regions of the
scene based on the extent of their motion. Such differentiation allows us to
adjust the spatio-temporal sampling resolution for each region in proportion to
its motion scale, achieving more photo-realistic dynamic novel view synthesis.
At the same time, almost for free, our approach enables free-viewpoint tracking
of objects of interest - a functionality not yet achieved by existing
NeRF-based methods. Empirical studies validate the effectiveness of our method,
where we achieve state-of-the-art rendering and tracking performance on
multiple challenging datasets.

Gear-NeRF 使用语义信息和动态对象追踪技术解决了 NeRF 模型在计算资源受限和场景语义理解方面的局限性，实现了逼真的动态场景渲染和新视角追踪。