Recent breakthroughs in Neural Radiance Fields (NeRFs) have sparked
significant demand for their integration into real-world 3D applications.
However, the varied functionalities required by different 3D applications often
necessitate diverse NeRF models with various pipelines, leading to tedious NeRF
training for each target task and cumbersome trial-and-error experiments.
Drawing inspiration from the generalization capability and adaptability of
emerging foundation models, our work aims to develop one general-purpose NeRF
for handling diverse 3D tasks. We achieve this by proposing a framework called
Omni-Recon, which is capable of (1) generalizable 3D reconstruction and
zero-shot multitask scene understanding, and (2) adaptability to diverse
downstream 3D applications such as real-time rendering and scene editing. Our
key insight is that an image-based rendering pipeline, with accurate geometry
and appearance estimation, can lift 2D image features into their 3D
counterparts, thus extending widely explored 2D tasks to the 3D world in a
generalizable manner. Specifically, our Omni-Recon features a general-purpose
NeRF model using image-based rendering with two decoupled branches: one complex
transformer-based branch that progressively fuses geometry and appearance
features for accurate geometry estimation, and one lightweight branch for
predicting blending weights of source views. This design achieves
state-of-the-art (SOTA) generalizable 3D surface reconstruction quality with
blending weights reusable across diverse tasks for zero-shot multitask scene
understanding. In addition, it can enable real-time rendering after baking the
complex geometry branch into meshes, swift adaptation to achieve SOTA
generalizable 3D understanding performance, and seamless integration with 2D
diffusion models for text-guided 3D editing.

通过提出称为 Omni-Recon 的框架，我们的工作旨在开发一个通用的 NeRF 来处理各种 3D 任务，其中关键见解是基于图像的渲染管道可以将 2D 图像特征提升到 3D 领域，从而在一种通用化的方式下扩展广泛研究的 2D 任务到 3D 世界。